Shifting from traditional control strategies to Deep Reinforcement Learning (RL) for legged robots poses inherent challenges, especially when addressing real-world physical constraints during training. While high-fidelity simulations provide significant benefits, they often bypass these essential physical limitations. In this paper, we experiment with the Constrained Markov Decision Process (CMDP) framework instead of the conventional unconstrained RL for robotic applications. We evaluated five constrained policy optimization algorithms for quadrupedal locomotion using three different robot models. Our aim is to evaluate their applicability in real-world scenarios. Our robot experiments demonstrate the critical role of incorporating physical constraints, yielding successful sim-to-real transfers, and reducing operational errors on physical systems. The CMDP formulation streamlines the training process by separately handling constraints from rewards. Our findings underscore the potential of constrained RL for the effective development and deployment of learned controllers in robotics.

Reinforcement learning (RL) has emerged as a powerful approach for locomotion control of highly articulated robotic systems. However, one major challenge is the tedious process of tuning the reward function to achieve the desired motion style. To address this issue, imitation learning approaches such as adversarial motion priors have been proposed, which encourage a pre-defined motion style. In this work, we present an approach to enhance the concept of adversarial motion prior-based RL, allowing for multiple, discretely switchable motion styles. Our approach demonstrates that multiple styles and skills can be learned simultaneously without significant performance differences, even in combination with motion data-free skills. We conducted several real-world experiments using a wheeled-legged robot to validate our approach. The experiments involved learning skills from existing RL controllers and trajectory optimization, such as ducking and walking, as well as novel skills, such as switching between a quadrupedal and humanoid configuration. For the latter skill, the robot was required to stand up, navigate on two wheels, and sit down. Instead of manually tuning the sit-down motion, we found that a reverse playback of the stand-up movement helped the robot discover feasible sit-down behaviors and avoided the need for tedious reward function tuning.

The common approach for local navigation on challenging environments with legged robots requires path planning, path following and locomotion, which usually requires a locomotion control policy that accurately tracks a commanded velocity. However, by breaking down the navigation problem into these sub-tasks, we limit the robot's capabilities since the individual tasks do not consider the full solution space. In this work, we propose to solve the complete problem by training an end-to-end policy with deep reinforcement learning. Instead of continuously tracking a precomputed path, the robot needs to reach a target position within a provided time. The task's success is only evaluated at the end of an episode, meaning that the policy does not need to reach the target as fast as possible. It is free to select its path and the locomotion gait. Training a policy in this way opens up a larger set of possible solutions, which allows the robot to learn more complex behaviors. We compare our approach to velocity tracking and additionally show that the time dependence of the task reward is critical to successfully learn these new behaviors. Finally, we demonstrate the successful deployment of policies on a real quadrupedal robot. The robot is able to cross challenging terrains, which were not possible previously, while using a more energy-efficient gait and achieving a higher success rate. Supplementary videos can be found on the project website: https://sites.google.com/leggedrobotics.com/end-to-end-loco-navigation

Navigating off-road with a fast autonomous vehicle depends on a robust perception system that differentiates traversable from non-traversable terrain. Typically, this depends on a semantic understanding which is based on supervised learning from images annotated by a human expert. This requires a significant investment in human time, assumes correct expert classification, and small details can lead to misclassification. To address these challenges, we propose a method for predicting high- and low-risk terrains from only past vehicle experience in a self-supervised fashion. First, we develop a tool that projects the vehicle trajectory into the front camera image. Second, occlusions in the 3D representation of the terrain are filtered out. Third, an autoencoder trained on masked vehicle trajectory regions identifies low- and high-risk terrains based on the reconstruction error. We evaluated our approach with two models and different bottleneck sizes with two different training and testing sites with a four-wheeled off-road vehicle. Comparison with two independent test sets of semantic labels from similar terrain as training sites demonstrates the ability to separate the ground as low-risk and the vegetation as high-risk with 81.1% and 85.1% accuracy.

We present a model predictive controller (MPC) that automatically discovers collision-free locomotion while simultaneously taking into account the system dynamics, friction constraints, and kinematic limitations. A relaxed barrier function is added to the optimization’s cost function, leading to collision avoidance behavior without increasing the problem’s computational complexity. Our holistic approach does not require any heuristics and enables legged robots to find whole-body motions in the presence of static and dynamic obstacles. We use a dynamically generated euclidean signed distance field for static collision checking. Collision checking for dynamic obstacles is modeled with moving cylinders, increasing the responsiveness to fast-moving agents. Furthermore, we include a Kalman filter motion prediction for moving obstacles into our receding horizon planning, enabling the robot to anticipate possible future collisions. Our experiments1demonstrate collision-free motions on a quadrupedal robot in challenging indoor environments. The robot handles complex scenes like overhanging obstacles and dynamic agents by exploring motions at the robot’s dynamic and kinematic limits.

Our paper proposes a model predictive controller as a single-task formulation that simultaneously optimizes wheel and torso motions. This online joint velocity and ground reaction force optimization integrates a kinodynamic model of a wheeled quadrupedal robot. It defines the single rigid body dynamics along with the robot’s kinematics while treating the wheels as moving ground contacts. With this approach, we can accurately capture the robot’s rolling constraint and dynamics, enabling automatic discovery of hybrid maneuvers without needless motion heuristics. The formulation’s generality through the simultaneous optimization over the robot’s whole-body variables allows for a single set of parameters and makes online gait sequence adaptation possible. Aperiodic gait sequences are automatically found through kinematic leg utilities without the need for predefined contact and lift-off timings, reducing the cost of transport by up to 85 %. Our experiments demonstrate dynamic motions on a quadrupedal robot with non-steerable wheels in challenging indoor and outdoor environments. The paper’s findings contribute to evaluating a decomposed, i.e., sequential optimization of wheel and torso motion, and single-task motion planner with a novel quantity, the prediction error, which describes how well a receding horizon planner can predict the robot’s future state. To this end, we report an improvement of up to 71 % using our proposed single-task approach, making fast locomotion feasible and revealing wheeled-legged robots’ full potential.

The task of robotic mobile manipulation poses several scientific challenges that need to be addressed to execute complex manipulation tasks in unstructured environments, in which collaboration with humans might be required. Therefore, we present ALMA, a motion planning and control framework for a torque-controlled quadrupedal robot equipped with a six degrees of freedom robotic arm capable of performing dynamic locomotion while executing manipulation tasks. The online motion planning framework, together with a whole-body controller based on a hierarchical optimization algorithm, enables the system to walk, trot and pace while executing operational space end-effector control, reactive human-robot collaboration and torso posture optimization to increase the arm's workspace. The torque control of the whole system enables the implementation of compliant behavior, allowing a user to safely interact with the robot. We verify our framework on the real robot by performing tasks such as opening a door and carrying a payload together with a human.

Robots working in natural, urban, and industrial settings need to be able to navigate challenging environments. In this paper, we present a motion planner for the perceptive rough-terrain locomotion with quadrupedal robots. The planner finds safe footholds along with collision-free swing-leg motions by leveraging an acquired terrain map. To this end, we present a novel pose optimization approach that enables the robot to climb over significant obstacles. We experimentally validate our approach with the quadrupedal robot ANYmal by autonomously traversing obstacles such steps, inclines, and stairs. The locomotion planner re-plans the motion at every step to cope with disturbances and dynamic environments. The robot has no prior knowledge of the scene, and all mapping, state estimation, control, and planning is performed in real-time onboard the robot.

Traditional legged robots are capable of traversing challenging terrain, but lack of energy efficiency when compared to wheeled systems operating on flat environments. The combination of both locomotion domains overcomes the trade-off between mobility and efficiency. Therefore, this paper presents a novel motion planner and controller which together enable a legged robot equipped with skates to perform skating maneuvers. These are achieved by an appropriate combination of planned reaction forces and gliding motions. Our novel motion controller formulates a Virtual Model Controller and an optimal contact force distribution which takes into account the nonholonomic constraints introduced by the skates. This approach has been tested on the torque-controllable robot ANY mal equipped with passive wheels and ice skates as end-effectors. We conducted experiments on flat and inclined terrain, whereby we show that skating motions reduces the cost of transport by up to 80 % with respect to traditional walking gaits.

Conventional skid or wheel based helicopter landing gears severely limit off-field landing possibilities, which are crucial when operating in scenarios such as mountain rescue. In this context, slopes beyond 8° and small obstacles can already pose a substantial hazard. An adaptive landing gear is proposed to overcome these limitations. It consists of four legs with one degree of freedom each. The total weight was minimized to demonstrate economic practicability. This was achieved by an innovative actuation, composed of a parallel arrangement of motor and brake, which relieves the motor from large impact loads during hard landings. The loads are alleviated by a spring-damper system acting in series to the actuation. Each leg is individually force controlled for optimal load distribution on compliant ground and to avoid tipping. The operation of the legs is fully autonomous during the landing phase. A prototype was designed and successfully tested on an unmanned helicopter with a maximum take-off weight of 78 kg. Finally, the implementation of the landing gear concept on aircraft of various scales was discussed.

This work introduces a novel hybrid control architecture for a hexapod platform (Weaver), making it capable of autonomously navigating in uneven terrain. The main contribution stems from the use of vision-based exteroceptive terrain perception to adapt the robot's locomotion parameters. Avoiding computationally expensive path planning for the individual foot tips, the adaptation controller enables the robot to reactively adapt to the surface structure it is moving on. The virtual stiffness, which mainly characterizes the behavior of the legs' impedance controller is adapted according to visually perceived terrain properties. To further improve locomotion, the frequency and height of the robot's stride are similarly adapted. Furthermore, novel methods for terrain characterization and a keyframe based visual-inertial odometry algorithm are combined to generate a spatial map of terrain characteristics. Localization via odometry also allows for autonomous missions on variable terrain by incorporating global navigation and terrain adaptation into one control architecture. Autonomous runs on a testbed with variable terrain types illustrate that adaptive stride and impedance behavior decreases the cost of transport by 30 % compared to a non-adaptive approach and simultaneously increases body stability (up to 88 % on even terrain and by 54 % on uneven terrain). Weaver is able to freely explore outdoor environments as it is completely free of external tethers, as shown in the experiments.

To address the goal of locomotion in very complex and difficult terrains, the authors are developing a new class of Ultralight Legged Robots. This paper presents the Multilegged Autonomous eXplorer (MAX), an ultralight, six-legged robot for traversal and exploration of challenging indoor and outdoor environments. The design of MAX emphasizes a low mass/size ratio, high locomotion efficiency, and high payload capability compared to total system mass. MAX is 2.25 m tall at full height and has a mass of approximately 60 kg, which makes it 5 to 20 times lighter than robots of comparable size. MAX is a research vehicle to explore modelling and control of Ultralight Legged Robots subject to flexing, oscillations and swaying; algorithms for gait planning and motion planning under uncertainty; and navigation planning for traversal of complex 3D terrains. This paper presents the design of MAX, provides an overview of the control system developed, summarizes results from indoor and outdoor tests, discusses system performance and outlines the challenges to be addressed next.

Legged robots such as hexapods have the potential to traverse unstructured terrain. This paper introduces a novel hexapod robot (Weaver) using a hierarchical controller, with the ability to efficiently traverse uneven and inclined terrain. The robot has five joints per leg and 30 degrees of freedom overall. The two redundant joints improve the locomotion of the robot by controlling the body pose and the leg orientation with respect to the ground. The impedance controller in Cartesian space reacts to unstructured terrain and thus achieves self-stabilizing behavior without prior profiling of the terrain through exteroceptive sensing. Instead of adding force sensors, the force at the foot tip is calculated by processing the current signals of the actuators. This work experimentally evaluates Weaver with the proposed controller and demonstrates that it can effectively traverse challenging terrains and high gradient slopes, reduce angular movements of the body by more than 55% and reduce the cost of transport (up to 50% on uneven terrain and by 85% on a slope with 20 °). The controller also enables Weaver to walk up inclines of up to 30 °, and remain statically stable on inclines up to 50 °. Furthermore, we present a new metric for legged robot stability performance along with a method for proprioceptive terrain characterization.