Deep Reinforcement Learning (RL) has demonstrated impressive results in solving complex robotic tasks such as quadruped locomotion. Yet, current solvers fail to produce efficient policies respecting hard constraints. In this work, we advocate for integrating constraints into robot learning and present Constraints as Terminations (CaT), a novel constrained RL algorithm. Departing from classical constrained RL formulations, we reformulate constraints through stochastic terminations during policy learning: any violation of a constraint triggers a probability of terminating potential future rewards the RL agent could attain. We propose an algorithmic approach to this formulation, by minimally modifying widely used off-the-shelf RL algorithms in robot learning (such as Proximal Policy Optimization). Our approach leads to excellent constraint adherence without introducing undue complexity and computational overhead, thus mitigating barriers to broader adoption. Through empirical evaluation on the real quadruped robot Solo crossing challenging obstacles, we demonstrate that CaT provides a compelling solution for incorporating constraints into RL frameworks. Videos and code are available at constraints-as-terminations.github.io.

This paper tackles the use of torque controlled humanoid robot TALOS in the context of industrial manufacturing. It demonstrates that it is possible to use Whole Body Model Predictive Control (WBMPC) to reliably insert a tool in the holes of an aircraft structure with an accuracy of few millimeters. This result is based on the use of Crocoddyl, an optimal control library that exploits differential dynamic programming (DDP) to achieve high numerical efficiency. The focus of this article is put on the procedure that was undertaken to shape the cost function of the optimal controller. Our approach has first been validate in a low performance setting on the humanoid robot TALOS. Then, a strategy to improve the performances by reinjecting information about the posture of the robot from previous experiments is showed in simulation.

The recent successes in deep reinforcement learning largely rely on the capabilities of generating masses of data, which in turn implies the use of a simulator. In particular, current progress in multi body dynamic simulators are under-pinning the implementation of reinforcement learning for end-to-end control of robotic systems. Yet simulators are mostly considered as black boxes while we have the knowledge to make them produce a richer information. In this paper, we are proposing to use the derivatives of the simulator to help with the convergence of the learning. For that, we combine model-based trajectory optimization to produce informative trials using 1st- and 2nd-order simulation derivatives. These locally-optimal runs give fair estimates of the value function and its derivatives, that we use to accelerate the convergence of the critics using Sobolev learning. We empirically demonstrate that the algorithm leads to a faster and more accurate estimation of the value function. The resulting value estimate is used in model-predictive controller as a proxy for shortening the preview horizon. We believe that it is also a first step toward superlinear reinforcement learning algorithm using simulation derivatives, that we need for end-to-end legged locomotion.

This paper presents the first successful experiment implementing whole-body model predictive control with state feedback on a torque-control humanoid robot. We demonstrate that our control scheme is able to do whole-body target tracking, control the balance in front of strong external perturbations and avoid collision with an external object. The key elements for this success are threefold. First, optimal control over a receding horizon is implemented with Crocoddyl, an optimal control library based on differential dynamics programming, providing state-feedback control in less than 10 ms. Second, a warm start strategy based on memory of motion has been implemented to overcome the sensitivity of the optimal control solver to initial conditions. Finally, the optimal trajectories are executed by a low-level torque controller, feedbacking on direct torque measurement at high frequency. This paper provides the details of the method, along with analytical benchmarks with the real humanoid robot Talos.A video of the experiment is available at https://peertube.laas.fr/videos/watch/cbc25927-337c-4635-a1bc-153b9aeb4135

General purpose simulators provide cheap training data to learn complex robotic skills. However, the transition from simulation to reality is often very challenging for the agent. One major issue is the delay on the physical robot that may deteriorate the performance of the deployed agent. Furthermore, once a successfully trained learning-based control policy is available, re-purposing the knowledge acquired by the agent to enable a structurally distinct agent to perform the same task is hazardous if done naively. In this work, we address the above issues with a single method, the DA-UNN (Delay Aware Universal Notice Network), which decomposes the knowledge into robot-specific and task-specific modules for fast transfer. Our framework deals with delays immanent to physical systems in order to improve sim2real transfer. We evaluate the efficiency of our approach using simulated and actual robots on a dynamic manipulation task where delay management is crucial.

In order to fluidly perform complex tasks in collaboration with a human being, such as table handling, a humanoid robot has to recognize and adapt to human movements. To achieve such goals, a realistic model of the human locomotion that is computable on a robot is needed. In this paper, we focus on making a humanoid robot follow a human-like locomotion path. We mainly present two models of human walking which lead to compute an average trajectory of the body center of mass from which a twist in the 2D plane can be deduced. Then the velocities generated by both models are used by a walking pattern generator to drive a real TALOS robot [1]. To determine which of these models is the most realistic for a humanoid robot, we measure human walking paths with motion capture and compare them to the computed trajectories.

Synthesizing legged locomotion requires planning one or several steps ahead (literally): when and where, and with which effector should the next contact(s) be created between the robot and the environment? Validating a contact candidate implies a minima the resolution of a slow, nonlinear optimization problem, to demonstrate that a center of mass (CoM) trajectory, compatible with the contact transition constraints, exists. We propose a conservative reformulation of this trajectory generation problem as a convex 3-D linear program, named convex resolution of centroidal dynamic trajectories (CROC). It results from the observation that if the CoM trajectory is a polynomial with only one free variable coefficient, the nonlinearity of the problem disappears. This has two consequences. On the positive side, in terms of computation times, CROC outperforms the state of the art by at least one order of magnitude, and allows to consider interactive applications (with a planning time roughly equal to the motion time). On the negative side, in our experiments, our approach finds a majority of the feasible trajectories found by a nonlinear solver, but not all of them. Still, we demonstrate that the solution space covered by CROC is large enough to achieve the automated planning of a large variety of locomotion tasks for different robots, demonstrated in simulation and on the real HRP-2 robot, several of which were rarely seen before. Another significant contribution is the introduction of a Bezier curve representation of the problem, which guarantees that the constraints of the CoM trajectory are verified continuously, and not only at discrete points as traditionally done. This formulation is lossless, and results in more robust trajectories. It is not restricted to CROC, but could rather be integrated with any method from the state of the art.

Autonomous robots such as legged robots and mobile manipulators imply new challenges in the design and the control of their actuators. In particular, it is desirable that the actuators are back-drivable, efficient (low friction) and compact. In this paper, we report the complete implementation of an advanced actuator based on screw, nut and cable. This actuator has been chosen for the humanoid robot Romeo. A similar model of the actuator has been used to control the humanoid robot Valkyrie. We expose the design of this actuator and present its Lagrangian model. The actuator being flexible, we propose a two-layer optimal control solver based on Differential Dynamical Programming. The actuator design, model identification and control is validated on a full actuator mounted in a work bench. The results show that this type of actuation is very suitable for legged robots and is a good candidate to replace strain wave gears.

Predictive control is an efficient model-based methodology to control complex dynamical systems. In general, it boils down to the resolution at each control cycle of a large nonlinear optimization problem. A critical issue is then to provide a good guess to initialize the nonlinear solver so as to speed up convergence. This is particularly important when disturbances or changes in the environment prevent the use of the trajectory computed at the previous control cycle as initial guess. In this paper, we introduce an original and very efficient solution to automatically build this initial guess. We propose to rely on off-line computation to build an approximation of the optimal trajectories, that can be used on-line to initialize the predictive controller. To that end, we combined the use of sampling-based planning, policy learning with generic representations (such as neural networks), and direct optimal control. We first propose an algorithm to simultaneously build a kinodynamic probabilistic roadmap (PRM) and approximate value function and control policy. This algorithm quickly converges toward an approximation of the optimal state-control trajectories (along with an optimal PRM). Then, we propose two methods to store the optimal trajectories and use them to initialize the predictive controller. We experimentally show that directly storing the state-control trajectories leads the predictive controller to quickly converges (2 to 5 iterations) toward the (global) optimal solution. The results are validated in simulation with an unmanned aerial vehicle (UAV) and other dynamical systems.

While only continuous motions are possible, the way in which contacts appear and disappear confers to legged locomotion a characteristic discontinuous nature that is traditionally shared by the algorithms used for legged locomotion planning. In this paper, we show that this discontinuous nature can disappear if the notion of collision is well redefined and we efficiently solve two different practical problems of legged locomotion planning with algorithms based on an approach that establishes a bridge between discrete and continuous planning. The first problem consists of reactive footstep planning with a biped robot and the second one consists of nongaited locomotion planning with a hexapod.

This paper presents a generic and efficient approach to generate dynamically consistent motions for under-actuated systems like humanoid or quadruped robots. The main contribution is a walking pattern generator, able to compute a stable trajectory of the center of mass of the robot along with the angular momentum, for any given configuration of contacts (e.g. on uneven, sloppy or slippery terrain, or with closed-gripper). Unlike existing methods, our solver is fast enough to be applied as a model-predictive controller. We then integrate this pattern generator in a complete framework: an acyclic contact planner is first used to automatically compute the contact sequence from a 3D model of the environment and a desired final posture; a stable walking pattern is then computed by the proposed solver; a dynamically-stable whole-body trajectory is finally obtained using a second-order hierarchical inverse kinematics. The implementation of the whole pipeline is fast enough to plan a step while the previous one is executed. The interest of the method is demonstrated by real experiments on the HRP-2 robot, by performing long-step walking and climbing a staircase with handrail support.

Pneumatic actuators have inherent compliance and hence they are very interesting for applications involving interaction with environment or human. But controlling such kind of actuators is not trivial. The paper presents an implementation of iterative Linear Quadratic regulator (iLQR) based optimal control framework to control an anthropomorphic arm with each joint actuated by an agonist-antagonistic pair of Mckibben artificial muscles. The method is applied to positioning tasks and generation of explosive movements by maximizing the link speed. It is then compared to traditional control strategies to justify that optimal control is effective in controlling the position in highly non-linear pneumatic systems. Also the importance of varying compliance is highlighted by repeating the tasks at different compliance level. The algorithm validation is reported here by several simulations and hardware experiments in which the shoulder and elbow flexion are controlled simultaneously.

We introduce a novel notion for lowering the dimensionality of motion planning problems: Irreducibility. Irreducibility of a configuration space trajectory τ means: We cannot find another configuration space trajectory τ', such that the swept volume of τ' is included in the swept volume of τ. The main contribution of our work is twofold: First, we show that motion planning in the space of irreducible trajectories is complete. Second, we show that we can construct reducible subspaces by reasoning about the inherent hierarchical structure of open kinematic chains. Using those theoretical results, we proceed by analytically defining a 7-dimensional irreducible configuration subspace for the humanoid robot HRP-2 under some assumptions. To show its practical importance, we solve a high-dimensional pin-hole problem for HRP-2 from the scratch.

Controlling the robot with a permanently-updated optimal trajectory, also known as model predictive control, is the Holy Grail of whole-body motion generation. Before obtaining it, several challenges should be faced: computation cost, non-linear local minima, algorithm stability, etc. In this paper, we address the problem of applying the updated optimal control in real-time on the physical robot. In particular, we focus on the problems raised by the delays due to computation and by the differences between the real robot and the simulated model. Based on the optimal-control solver MuJoCo, we implemented a complete model-predictive controller and we applied it in real-time on the physical HRP-2 robot. It is the first time that such a whole-body model predictive controller is applied in real-time on a complex dynamic robot. Aside from the technical contributions cited above, the main contribution of this paper is to report the experimental results of this première implementation.

We present a novel approach to introduce visual information in the walking pattern generator for humanoid robots in a more direct way than the current existing methods. We make use of a model predictive control (MPC) visual servoing strategy, which is combined to the walking motion generator. We define two schemes based on that principle: a position-based and an image-based scheme, with a Quadratic Program (QP) formulation in both cases. Finally, we present some simulation results validating our approach.

We presented our works relative to anthropomorphic motions. We performed task recognition, full-dynamic motion generation, motion retargeting and editing in a unified framework: the stack of tasks. Thanks to the genericity of the task function formalism, our works can be further extended. For example, for the recognition, the use of the task function formalism applied to human motion is currently investigated. Also, preliminary results on the real robot for the retargeting and editing method have been obtained.

In this paper we introduce a new bounding box method for footstep planning for humanoid robots. Similar to the classic bounding box method (which uses a single rectangular box to encompass the robot) it is computationally efficient, easy to implement and can be combined with any rigid body motion planning library. However, unlike the classic bounding box method, our method takes into account the stepping over capabilities of the robot, and generates precise leg trajectories to avoid obstacles on the ground. We demonstrate that this method is well suited for footstep planning in cluttered environments.

In this paper, we propose a novel and coherent framework for fast footstep planning for legged robots on a flat ground with 3-D obstacle avoidance. We use swept volume approximations that are computed offline in order to considerably reduce the time spent in collision checking during the online planning phase, in which a rapidly exploring random tree variant is used to find collision-free sequences of half-steps (which are produced by a specific walking pattern generator). Then, an original homotopy is used to smooth the sequences into natural motions, gently avoiding the obstacles. The results are experimentally validated on the robot HRP-2.

Efficient methods to perform motion recognition have been developed using statistical tools. Those methods rely on primitive learning in a suitable space, for example, the latent space of the joint angle and/or adequate task spaces. Learned primitives are often sequential: A motion is segmented according to the time axis. When working with a humanoid robot, a motion can be decomposed into parallel subtasks. For example, in a waiter scenario, the robot has to keep some plates horizontal with one of its arms while placing a plate on the table with its free hand. Recognition can thus not be limited to one task per consecutive segment of time. The method presented in this paper takes advantage of the knowledge of what tasks the robot is able to do and how the motion is generated from this set of known controllers, to perform a reverse engineering of an observed motion. This analysis is intended to recognize parallel tasks that have been used to generate a motion. The method relies on the task-function formalism and the projection operation into the null space of a task to decouple the controllers. The approach is successfully applied on a real robot to disambiguate motion in different scenarios where two motions look similar but have different purposes.

We present a new biped walking pattern generator based on "half-steps". Its key features are a) a 3-dimensional parametrization of the input space, and b) a simple homotopy that efficiently smooths the walking trajectory corresponding to a fixed sequence of steps. We show how these features can be ideally combined in the framework of sampling-based footstep planning. We apply our approach to the robot HRP-2 and are able to quickly produce smooth and dynamically stable trajectories that are solutions to a difficult problem of footstep planning.

In this paper we demonstrate an original equivalence between footstep planning problems, where discrete sequences of steps are searched for, and the more classical problem of motion planning for a 2D rigid shape, where a continuous collision-free path has to be found. This equivalence enables a lot of classical motion planning techniques (such as PRM, RRT, etc.) to be applied almost effortlessly to the specific problem of footstep planning for a humanoid robot.

We present here an original approach to test the feasibility of footsteps for a given walking pattern generator. It is based on a new approximation algorithm intended to cope with this specific problem. The result obtained is used on the robot HRP-2, and enables it to guess a step feasibility 40,000 times faster (in 9μs) than with the normal verification process. As a consequence some advance is made towards fast online motion (re)planning based on a continuous set of possible steps.

This paper introduces a visual servoing scheme for humanoid walking. Though most of the existing approaches follow a perception-decision-action scheme, we close the loop so that the control is robust to model error. Our approach is based on a new reactive pattern generator which modifies, at the control level, the footsteps, the center of mass and the center of pressure trajectories for the center of mass to track a reference velocity. And, in this paper, the reference velocity is directly given by a visual servoing control law. Since, the HRP-2 walk induces a sway motion that disturbs the regulation of the visual control law, we introduce a control law allowing convergence in the image space and taking into account this sway motion.

This article addresses the fast solution of a Quadratic Program underlying a Linear Model Predictive Control scheme that generates walking motions. We introduce an algorithm which is tailored to the particular requirements of this problem, and therefore able to solve it efficiently. Different aspects of the algorithm are examined, its computational complexity is presented, and a numerical comparison with an existing state of the art solver is made. The approach presented here, extends to other general problems in a straightforward way.

A novel approach is presented which aims at building autonomously visual models of unknown objects, using a humanoid robot. Previous methods have been proposed for the specific problem of the next-best-view during the modeling and the recognition process. However our approach differs as it takes advantage of humanoid specificities in terms of embedded vision sensor and redundant motion capabilities. In a previous work, another approach to this specific problem was presented which relies on a derivable formulation of the visual evaluation in order to integrate it with our posture generation method. However to get rid of some limitations we propose a new method, formulated using two steps: (i) an optimization algorithm without derivatives is used to find a camera pose which maximizes the amount of unknown data visible, and (ii) a whole robot posture is generated by using a different optimization method where the computed camera pose is set as a constraint on the robot head.

This paper is the continuation of our previous work in intercontinental, collaborative teleoperation with a humanoid robot. Our new achievement consists in an extension of the former single-arm bilateral teleoperation setting to include bimanual manipulation and walking. A coupling scheme for simultaneous manipulation and locomotion is developed. Furthermore, a task-based control framework, including a force-based control for the arms as well as a walking pattern generation, is presented to realize stable whole-body motions of the highly redundant humanoid robot. Experiments have been performed to assess the proposed control scheme. They bring to light additional scientific challenges that remain in order to reach a smooth and natural telepresent collaboration.

This study proposes a complete solution to make the humanoid robot HRP-2 dynamically step over large obstacles. As compared with previous results using quasistatic stability, where the robot crosses over a 15-cm obstacle in 40 s, our solution allows HRP-2 to step over the same obstacle in 4 s. This approach allows the robot to clear obstacles as high as 21% of the robot's leg length (15 cm) while walking. Simulations show the possibility to step over an obstacle that is 35% of the length (25 cm) with a margin of 3 cm.

This paper proposes a real-time implementation of collision and self-collision avoidance for robots. On the basis of a new proximity distance computation method which ensures having continuous gradient, a new controller in the velocity domain is proposed. The gradient continuity encompasses no jump in the generated command. Included in a stack of tasks architecture, this controller has been implemented on the humanoid platform HRP-2 and experienced in a grasping task while walking and avoiding collisions with the environment and auto-collisions.

In multimodal tele-cooperation as considered in this paper two humans in distant locations jointly perform a task requiring multimodal including haptic feedback. One human operator teleoperates a remotely placed humanoid robot which is collocated with the human cooperator. Time delay in the communication channel as destabilizing factor is one of the multiple challenges associated with such a tele-cooperation setup. In this paper we employ a control architecture with force-position exchange accounting for the admittance type of the haptic input device and the telerobot, which both are position-based admittance controlled. Llewellynpsilas stability criteria are employed for the parameter tuning of the virtual impedances in the presence of time delay. The control strategy is successfully validated in an intercontinental tele-cooperation experiment with the humanoid telerobot HRP-2 located in Japan/Tsukuba and a multimodal human-system-interface located in Germany/Munich, see also the corresponding video submission. The proposed setup gives rise to a large number of exciting new research questions to be addressed in the future.

The video shows an intercontinental cooperative telemanipulation task, whereby the operator site is located in Munich, Germany and the teleoperator site in Tsukuba, Japan. The human operator controls a remotely located teleoperator, which performs a task in the remote environment. Hereby the human operator is assisted by another person located at the remote site. The task consists in jointly grasping an object, moving it to a new position and finally releasing it, see Fig. 1.

In this paper, we apply a general framework for building complex whole-body control for highly redundant robot, and we propose to implement it for visually-guided grasping while walking on a humanoid robot. The key idea is to divide the control into several sensor-based control tasks that are simultaneously executed by a general structure called stack of tasks. This structure enables a very simple access for task sequencing, and can be used for task-level control. This framework was applied for a visual servoing task. The robot walks along a planned path, keeping the specified object in the middle of its field of view and finally, when it is close enough, the robot grasps the object while walking.

This video demonstrates our current investigation in developing autonomous behaviors for humanoid robots. Our main goal is to develop functionalities as much generic as possible in order to realize useful behaviors. More particularly this video demonstrates our current status on extending a popular zero momentum problem (ZMP) preview control based pattern generator, and building some links between walking with vision.

This paper presents an object active visual search behavior in a 3D environment performed by a HRP-2 humanoid robot. The search is formalized as an optimization problem in which the goal is to maximize the target detection probability while minimizing the energy/distance and time to achieve the task. Natural constraints on the camera parameter space based on the characteristics of the recognition system are used to reduce the dimension of the problem and to speed up the optimization process to achieve a real time behavior. We present simulation and real experimental results using an HRP-2 robot.

We present a real-time algorithm which can recover the 3D trajectory of a monocular camera, moving rapidly through a previously unknown scene. Our system, which we dub MonoSLAM, is the first successful application of the SLAM methodology from mobile robotics to the "pure vision" domain of a single uncontrolled camera, achieving real time but drift-free performance inaccessible to Structure from Motion approaches. The core of the approach is the online creation of a sparse but persistent map of natural landmarks within a probabilistic framework. Our key novel contributions include an active approach to mapping and measurement, the use of a general motion model for smooth camera movement, and solutions for monocular feature initialization and feature orientation estimation. Together, these add up to an extremely efficient and robust algorithm which runs at 30 Hz with standard PC and camera hardware. This work extends the range of robotic systems in which SLAM can be usefully applied, but also opens up new areas. We present applications of MonoSLAM to real-time 3D localization and mapping for a high-performance full-size humanoid robot and live augmented reality with a hand-held camera.

This paper addresses the role of toe joints in increasing the walking speed of biped robots. It is worthy that adding a toe joint will increase the step length thanks to the additional degree of freedom. But, the originality of this work is that longer steps are obtained thanks to an under-actuated phase and an appropriate ZMP trajectory. The simulations showed that adding passive toe joints allows smoother and 1.5 faster walking

Humanoid robotics and SLAM (simultaneous localisation and mapping) are certainly two of the most significant themes of the current worldwide robotics research effort, but the two fields have up until now largely run independent parallel paths, despite the obvious benefit to be gained in joining the two. The next major step forward in humanoid robotics will be increased autonomy, and the ability of a robot to create its own world map on the fly will be a significant enabling technology. Meanwhile, SLAM techniques have found most success with robot platforms and sensor configurations which are outside of the humanoid domain. Humanoid robots move with high linear and angular accelerations in full 3D, and normally only vision is available as an outward-looking sensor. Building on recently published work on monocular SLAM using vision, and on pattern generation, we show that real-time SLAM for a humanoid can indeed be achieved. Using HRP-2, we present results in which a sparse 3D map of visual landmarks is acquired on the fly using a single camera and demonstrated loop closing and drift-free 3D motion estimation within a typical cluttered indoor environment. This is achieved by tightly coupling the pattern generator, the robot odometry and inertial sensing to aid visual mapping within a standard EKF framework. To our knowledge this is the first implementation of real-time 3D SLAM for a humanoid robot able to demonstrate loop closing

This paper presents a 3D object recognition method based on spin-images for a humanoid robot having a stereoscopic vision system. Spin-images have been proposed to search CAD models database, and use 3D range informations. In this context, the use of a vision system is taken into account through a multi-resolution approach. A method for quickly computing multi-resolution and interpolating spin-images is proposed. The results on simulation and on real data are given, and show the effectiveness of this method

Considering a stereoscopic visual system, this paper deals with the error involved by a 3D reconstruction process. If image pixels are seen as surfaces instead of points, interval analysis provides bounding boxes in which the reconstructed 3D point lies with certainty. This paper presents a method which refine this bounding box and gives a tighter approximation of the error. Using bisection, and a reprojection test in the image planes, the space in which the 3D reconstructed point may be located is given as an octree. This is achieved through the resolution of a set inversion problem using the SIVIA algorithm. For a lighter manipulation of the result, an englobing ellipsoid is deduced from this approximation. Finally the three models are tested on a recognition process for a humanoid robot.

This papers presents an application of interval analysis to a 3D reconstruction problem. The aim is to build a partial boundaries representation of an object including guaranteed information according to the camera model. Features points coordinates are described using intervals. This representation is used together with a method to search stereo correspondence based on the connectivity of segments.

To operate in a known and dynamic environment, a robot needs two different kinds of a priori knowledge : behavioral and visual. In this paper we propose a general methodology to introduce this a priori knowledge, and we illustrate it by an application : the software developments for the AIBO robots to play soccer. The more difficult task is to extract reliable visual information. It is the reason why a priori knowledge is already introduced at the level of vision processing. Without reliable vision information it is impossible for the robot to switch to the tight behavior, and consequently to play.

The aim of this paper is to present our attempt to create a development platform for complex robotic applications. Such a system needs appropriate tools for handling real-time aspects, distributed architectures, portability through heterogeneous hardware, and code re-usability. We show in this paper that, by constraining the shape of an application, specifying its scheduling and building a separate representation of the hardware, it is possible to realize a system where all those aspects are integrated. The major properties of our system is a strong multithreading architecture, the possibility to handle design patterns, and a powerful model of hardware platforms using a hypergraph.