Standard deep reinforcement learning (DRL) aims to maximize expected reward, considering collected experiences equally in formulating a policy. This differs from human decision-making, where gains and losses are valued differently and outlying outcomes are given increased consideration. It also fails to capitalize on opportunities to improve safety and/or performance through the incorporation of distributional context. Several approaches to distributional DRL have been investigated, with one popular strategy being to evaluate the projected distribution of returns for possible actions. We propose a more direct approach whereby risk-sensitive objectives, specified in terms of the cumulative distribution function (CDF) of the distribution of full-episode rewards, are optimized. This approach allows for outcomes to be weighed based on relative quality, can be used for both continuous and discrete action spaces, and may naturally be applied in both constrained and unconstrained settings. We show how to compute an asymptotically consistent estimate of the policy gradient for a broad class of risk-sensitive objectives via sampling, subsequently incorporating variance reduction and regularization measures to facilitate effective on-policy learning. We then demonstrate that the use of moderately "pessimistic" risk profiles, which emphasize scenarios where the agent performs poorly, leads to enhanced exploration and a continual focus on addressing deficiencies. We test the approach using different risk profiles in six OpenAI Safety Gym environments, comparing to state of the art on-policy methods. Without cost constraints, we find that pessimistic risk profiles can be used to reduce cost while improving total reward accumulation. With cost constraints, they are seen to provide higher positive rewards than risk-neutral approaches at the prescribed allowable cost.

Human-aware robot navigation promises a range of applications in which mobile robots bring versatile assistance to people in common human environments. While prior research has mostly focused on modeling pedestrians as independent, intentional individuals, people move in groups; consequently, it is imperative for mobile robots to respect human groups when navigating around people. This paper explores learning group-aware navigation policies based on dynamic group formation using deep reinforcement learning. Through simulation experiments, we show that group-aware policies, compared to baseline policies that neglect human groups, achieve greater robot navigation performance (e.g., fewer collisions), minimize violation of social norms and discomfort, and reduce the robot's movement impact on pedestrians. Our results contribute to the development of social navigation and the integration of mobile robots into human environments.

Humans employ a high degree of redundancy in joint actuation, with different combinations of muscle and tendon action providing the same net joint torque. Both the resolution of these redundancies and the energetics of such systems depend on the dynamic properties of muscles and tendons, particularly their force-length relations. Current walking models that use stock parameters when simulating muscle-tendon dynamics tend to significantly overestimate metabolic consumption, perhaps because they do not adequately consider the role of elasticity. As an alternative, we posit that the muscle-tendon morphology of the human leg has evolved to maximize the metabolic efficiency of walking at self-selected speed. We use a data-driven approach to evaluate this hypothesis, utilizing kinematic, kinetic, electromyographic (EMG), and metabolic data taken from five participants walking at self-selected speed. The kinematic and kinetic data are used to estimate muscle-tendon lengths, muscle moment arms, and joint moments while the EMG data are used to estimate muscle activations. For each subject we perform an optimization using prescribed skeletal kinematics, varying the parameters that govern the force-length curve of each tendon as well as the strength and optimal fiber length of each muscle while seeking to simultaneously minimize metabolic cost and maximize agreement with the estimated joint moments. We find that the metabolic cost of transport (MCOT) values of our participants may be correctly matched (on average 0.36±0.02 predicted, 0.35±0.02 measured) with acceptable joint torque fidelity through application of a single constraint to the muscle metabolic budget. The associated optimal muscle-tendon parameter sets allow us to estimate the forces and states of individual muscles, resolving redundancies in joint actuation and lending insight into the potential roles and control objectives of the muscles of the leg throughout the gait cycle.