Deep Reinforcement Learning (RL) has demonstrated impressive results in solving complex robotic tasks such as quadruped locomotion. Yet, current solvers fail to produce efficient policies respecting hard constraints. In this work, we advocate for integrating constraints into robot learning and present Constraints as Terminations (CaT), a novel constrained RL algorithm. Departing from classical constrained RL formulations, we reformulate constraints through stochastic terminations during policy learning: any violation of a constraint triggers a probability of terminating potential future rewards the RL agent could attain. We propose an algorithmic approach to this formulation, by minimally modifying widely used off-the-shelf RL algorithms in robot learning (such as Proximal Policy Optimization). Our approach leads to excellent constraint adherence without introducing undue complexity and computational overhead, thus mitigating barriers to broader adoption. Through empirical evaluation on the real quadruped robot Solo crossing challenging obstacles, we demonstrate that CaT provides a compelling solution for incorporating constraints into RL frameworks. Videos and code are available at constraints-as-terminations.github.io.

Quadruped robots have proved their robustness to cross complex terrain despite little environment knowledge. Yet advanced locomotion controllers are expected to take advantage of exteroceptive information. This paper presents a complete method to plan and control the locomotion of quadruped robots when 3D information about the surrounding obstacles is available, based on several stages of decision. We first propose a contact planner formulated as a mixed-integer program, optimized on-line at each new robot step. It selects a surface from a set of convex surfaces describing the environment for the next footsteps while ensuring kinematic constraints. We then propose to optimize the exact contact location and the feet trajectories at control frequency to avoid obstacles, thanks to an efficient formulation of quadratic programs optimizing Bezier curves. By relying on the locomotion controller of our quadruped robot Solo, we finally implement the complete method, provided as an open-source package. Its efficiency is asserted by statistical evaluation of the importance of each component in simulation. We have a 100% success rate for our framework, and we show that the deactivation of the contact planning, footstep adaptation and collision avoidance, respectively induced a drop to 70%, 62% and 83% success rate in the worst case, justifying the complete architecture.

As locomotion decisions must be taken by considering the future, most existing quadruped controllers are based on a model predictive controller (MPC) with a reduced model of the dynamics to generate the motion and a whole- body controller to execute it. Yet the simplifying assumptions of the MPC are often chosen ad-hoc or by intuition. In this article, we focus on a set of MPCs and analyze the effect of chosen model reductions on the behavior of the robot. Based on existing formulations, we present additional controllers to better understand the influence of model reductions on the controller capabilities. Finally, we propose a robust predictive controller capable of optimizing the foot placements, gait period, center- of-mass trajectory and ground reaction forces. The behavior of these controllers is statistically evaluated in simulation. This empirical study aims to assess the relative importance of the components of the optimal control problem (variables, costs, dynamics) to be able to take reasoned decisions instead of arbitrarily emphasizing or neglecting some of them. We also provide a qualitative study in simulation and on the real robot Solo-12.

This paper presents a computational framework for the design of high-performance legged robotic systems. The framework relies on the concurrent optimization of hardware parameters and control trajectories to find the best robot design for a given task. In particular, we focus on energy efficiency, presenting novel electro-mechanical models to account for the losses of the actuators due to friction and Joule effects. Thanks to a bi-level optimization scheme, featuring a genetic algorithm in the outer loop, our framework can also optimize for the duration of the motion, the actuators, and the size of the robot. We present a novel approach to scale both the actuators and the robot structure in a way that ensures structural integrity by maintaining constant the normalized deflection of the links. We validated our approach by designing a two-joint monoped robot to execute a jumping task. Our simulation results show that our framework can lead to remarkable energy savings (up to 60%) thanks to the concurrent optimization of robot size, motion duration, and actuators.

State estimation, in particular estimation of the base position, orientation and velocity, plays a big role in the efficiency of legged robot stabilization. The estimation of the base state is particularly important because of its strong correlation with the underactuated dynamics, i.e. the evolution of center of mass and angular momentum. Yet this estimation is typically done in two phases, first estimating the base state, then reconstructing the center of mass from the robot model. The underactuated dynamics is indeed not properly observed, and any bias in the model would not be corrected from the sensors. While it has already been observed that force measurements make such a bias observable, these are often only used for a binary estimation of the contact state. In this paper, we propose to simultaneously estimate the base and the underactuation state by exploiting all measurements simultaneously. To this end, we propose several contributions to implement a complete state estimator using factor graphs. Contact forces altering the underactuated dynamics are pre-integrated using a novel adaptation of the IMU pre-integration method, which constitutes the principal contribution. IMU pre-integration is also used to estimate the positional motion of the base. Encoder measurements then participate to the estimation in two ways: by providing leg odometry displacements which contributes to the observability of IMU biases; and by relating the positional and centroidal states, thus connecting the whole graph and producing a tightly-coupled whole-body estimator. The validity of the approach is demonstrated on real data captured by the Solo12 quadruped robot.

This paper aims at showing the dynamic performance and reliability of the low-cost, open-access quadruped robot Solo-12, which is developed within the framework of Open Dynamic Robot Initiative. It presents the implementation of a state-of-the-art control pipeline, close to the one that was previously implemented on Mini Cheetah, which implements a model predictive controller based on the centroidal dynamics to compute desired contact forces in order to track a reference velocity. Different contributions are proposed to speed up the computation process, notably at the level of the state estimation and the whole body controller. Experimental results demonstrate that the robot closely follow the reference velocity while being highly reactive and able to recover from perturbations.

Self-collision detection and avoidance are essential for reactive control, in particular for dynamics robots equipped with legs or arms. Yet, only few control methods are able to handle such constraints, and it is often necessary to rely on path planning to define a collision-free trajectory that the controller would then track. In this paper, we introduce a combination of two lightweight, conservative and smooth models to generically handle self-collisions in robot control. For pairs of bodies that are far from one another on average (e.g. segments of distinct legs), we rely on a standard forward kinematics approach, using simplified geometries for which we provide analytical derivatives. For bodies that are moving close to one another, we propose to use a data-driven approach, with datasets generated thanks to a standard collision library. We then build a simple torque-based controller that can be implemented on top of any control law to prevent unexpected self-collision. This controller is meant to be implemented as a low-level protection, directly on the robot hardware. We also provide an open-source library to generate ANSI-C code for any robot model, experimented on the real quadruped Solo.