Event cameras are recent sensors that measure intensity changes in each pixel asynchronously. It is being used due to lower latency and higher temporal resolution compared to traditional frame-based camera. We propose a method of 3D model-based object tracking directly from events captured by event camera. To enable reliable and accurate tracking of objects, we use a new event representation and predict brightness increment images with motion interpolation. Results of object tracking show the new methods significantly improves tracking duration and robustness, both for perspective and fisheye cameras. Our implementation succeeds in tracking objects when the camera speed is reaching 2 m/s.

In a world designed for legs, quadrupeds, bipeds, and humanoids have the opportunity to impact emerging robotics applications from logistics, to agriculture, to home assistance. The goal of this survey is to cover the recent progress toward these applications that have been driven by model-based optimization for the real-time generation and control of movement. The majority of the research community has converged on the idea of generating locomotion control laws by solving an optimal control problem (OCP) in either a model-based or data-driven manner. However, solving the most general of these problems online remains intractable due to complexities from intermittent unidirectional contacts with the environment, and from the many degrees of freedom of legged robots. This survey covers methods that have been pursued to make these OCPs computationally tractable, with a specific focus on how environmental contacts are treated, how the model can be simplified, and how these choices affect the numerical solution methods employed. The survey focuses on model-based optimization while paving its way for broader combination with learning-based formulations to accelerate progress in this growing field.

In this paper we report the safety-oriented framework for controlling the torque in the case of robots with high reduction gears and having no joint torque feedback. This kind of robots suffer from high joint friction and low backdrivability, requiring high gains and integral feedback, which can be dangerous. Our optimization-based framework includes feasibility and safety features borrowed from position control, and we introduce novel ones. We show how we limit the integral terms using a QP-based anti-windup which produces the optimal torque that maintains the best performances under safety limits. We show also a new controller for null-space compliance, providing strong guarantees of convergence in the task-space and ignoring the corresponding null-space where the robot can be moved freely. We validate these features with experiments on one 9 DoF arm of the robot HRP-5P performing a Cartesian task, and then a dual Cartesian / admittance task.

Deformable objects are very common around us in our daily life. Because they have infinitely many degrees of freedom, they present a challenging problem in robotics. Inspired by practical industrial applications, we present in this paper our research on using a humanoid robot to take a long, thin and flexible belt out of a bobbin and pick up the bending part of the belt from the ground. By proposing a novel non-prehensile manipulation strategy "scraping" which utilizes the friction between the gripper and the surface of the belt, efficient manipulation can be achieved. In addition, a 3D shape detection algorithm for deformable objects is used during manipulation process. By integrating the novel "scraping" motion and the shape detection algorithm into our multi-objective QP-based controller, we show experimentally humanoid robots can complete this complex task.

Capturability analysis of the linear inverted pendulum (LIP) model enabled walking with constrained height based on the capture point. In this paper, we generalize this analysis to the variable-height inverted pendulum (VHIP) and show how it enables 3-D walking over uneven terrains based on capture inputs. Thanks to a tailored optimization scheme, we can compute these inputs fast enough for real-time model predictive control. We implement this approach as open-source software and demonstrate it in dynamic simulations.

This paper focuses on a stabilization control for multi-contact motion which enables a humanoid robot to locomote by realizing dynamic contact transitions on non-flat environment. In the stabilization process of the multi-contact motion, the desired Zero-Moment Point (ZMP) is modified by the position of the Divergent Component of Motion (DCM) error with respect to the 3D Center of Mass (CoM) motion generated from the force distribution ratio. The contact wrench of each end-effector is determined by quadratic optimization considering the centroidal dynamics and contact friction constraints so as to satisfy the modified ZMP. Each end-effector is controlled by optimized force reference through a projection of null space by force distribution ratio. We propose a multi-contact stabilization framework which can be designed not only to generate 3D CoM motion but also the CoM position estimation and the optimal force distribution around the reference ZMP in a unified manner from a balance controller, by using the force distribution ratio. The effectiveness of proposed method is validated by a quadruped locomotion leaning against a vertical wall using the joint position controlled humanoid HRP-5P in a dynamic simulator.

We propose a control framework for torque controlled humanoid robots that efficiently minimizes the tracking error in a Quadratic Programming (QP)formulated as multiobjective weighted tasks with constraints. It results in an optimal dynamically-feasible reference that can be tracked robustly, with exponential convergence, without joint torque feedback, in the presence of non modelled torque bias and low-frequency bounded disturbances. This is achieved by introducing integral gains in a Lyapunov-stable torque control, which exploit the passivity properties of the dynamical model of the robot and their effect on the dynamic constraints of the QP solver. The robustness of this framework is demonstrated in simulation by commanding our robot, the HRP-5P, to achieve simultaneously several objectives in the configuration and the Cartesian spaces, in the presence of non-modeled static and kinetic joint friction, as well as an uncertain torque scale.

A motion planning algorithm computes the motion of a robot by computing a path through its configuration space. To improve the runtime of motion planning algorithms, we propose to nest robots in each other, creating a nested quotient-space decomposition of the configuration space. Based on this decomposition we define a new roadmap-based motion planning algorithm called the Quotient-space roadMap Planner (QMP). The algorithm starts growing a graph on the lowest dimensional quotient space, switches to the next quotient space once a valid path has been found, and keeps updating the graphs on each quotient space simultaneously until a valid path in the configuration space has been found. We show that this algorithm is probabilistically complete and outperforms a set of state-of-the-art algorithms implemented in the open motion planning library (OMPL).

We propose a framework to generate static robot configurations satisfying a set of physical and geometrical constraints. This is done by formulating nonlinear constrained optimization problems over non-Euclidean manifolds and solving them. To do so, we present a new sequential quadratic programming solver working natively on general manifolds and propose an interface to easily formulate the problems, with the tedious and error-prone work automated for the user. We also introduce several new types of constraints for having more complex contacts or working on forces/torques. Our approach allows an elegant mathematical description of the constraints and we exemplify it through formulation and computation examples in complex scenarios with humanoid robots.

We study the HRP-2Kai humanoid robot's ability to conduct the precise industrial task of fastening bolts in aircraft production. Our contribution stands mainly in high integration of different modules within the whole-body Quadratic Programing-based controller that has not been yet confronted to tasks demanding high precision in the execution and tuning. This includes the use of a robust visual servoing algorithm which allows the robot to move autonomously to a desired target and the design of specific tasks and estimators: a learning and admittance control that enables the robot to interact smoothly with its environment, and a fast and safe method to autonomously detect correct tool on nut insertion. We then show that the controller indeed enables our humanoid robot to achieve such a high precision task.

In this paper we propose a method to build a smooth and non-singular map from the unit sphere to a Catmull-Clark subdivision surface. We use a tailored ray-casting algorithm to associate a point of the surface to each point of the sphere. This allows us to have smooth approximate representations for a large variety of (possibly non-convex) meshes that we can use to write numerically well-behaved constraints in gradient-based optimization. In particular, we address the writing of contact constraints between a robot and objects of its environment. We first detail our algorithm, taking care of speed and robustness, then show several use-cases for posture generations with a humanoid robot or a hand.

This paper presents a new condition, the fully physical consistency for a set of inertial parameters to determine if they can be generated by a physical rigid body. The proposed condition ensure both the positive definiteness and the triangular inequality of 3D inertia matrices as opposed to existing techniques in which the triangular inequality constraint is ignored. This paper presents also a new parametrization that naturally ensures that the inertial parameters are fully physical consistency. The proposed parametrization is exploited to reformulate the inertial identification problem as a manifold optimization problem, that ensures that the identified parameters can always be generated by a physical body. The proposed optimization problem has been validated with a set of experiments on the iCub humanoid robot.

We propose a many-core GPU implementation of robotic motion planning formulated as a semi-infinite optimization program. Our approach computes the constraints and their gradients in parallel, and feeds the result to a nonlinear optimization solver running on the CPU. To ensure the continuous satisfaction of our constraints, we use polynomial approximations over time intervals. Because each constraint and its gradient can be evaluated independently for each time interval, we end up with a highly parallelizable problem that can take advantage of many-core architectures. Classic robotic computations (geometry, kinematics, and dynamics) can also benefit from parallel processors, and we carefully study their implementation in our context. This results in having a full constraint evaluator running on the GPU. We present several optimization examples with a humanoid robot. They reveal substantial improvements in terms of computation performance compared to a parallel CPU version.

We propose a method for estimation of humanoid and human links' inertial parameters. Our approach formulates the problem as a hierarchical quadratic program by exploiting the linear properties of rigid body dynamics with respect to the inertia parameters. In order to assess our algorithm, we conducted experiments with a humanoid robot and a human subject. We compared ground reaction forces and moments estimated from force measurements with those computed using identified inertia parameters and movement information. Our method is able to accurately reconstruct ground reaction forces and force moments. Moreover, our method is able to estimate correctly masses of the robots links and to accurately detect additional masses placed on the human subject during the experiments.

Humanoid robots being underactuated, they need to interact with their environment to move. When optimizing trajectories, the contact forces must therefore be taken into account. In this paper, we first highlight an issue encountered when using parametrized functions in the presence of equality constraints which need to be satisfied continuously over a time interval. We then propose a parametrization of contact forces and a formulation of the constraints which allow to write a tractable optimization program whose solution verifies at any instant all the constraints where those forces appear. We finish by exemplifying the approach with preliminary results obtained on a HRP-2 humanoid robot.

The mass parameters of robots influence performances of model-based control and validation of the simulation results. The mass parameters provided by CAD data are usually rough approximation of the true parameters. Therefore several methods for estimation of those parameters have been proposed. Their precision depends on the used motion, called optimal exciting trajectories. This paper describes a new approach to determine humanoid robot exciting trajectories for mass parameters identification. The method was inspired by the studies done in the field of human mass parameters identification, and it is based on observation of condition numbers of sub-regressor matrices created from the columns of the regressor matrix. The method has been experimentally applied to identify mass parameters of HRP-2 and HRP-4 humanoid robots. The proposed method is able to reconstruct ground reaction forces and force moments more accurately than parameters obtained from CAD data.

Our work builds largely on Nagasaka's stabilizer in multi-contact motion [1]. Using a sequence of contact stances from an offline multi-contact planner, we use first a Model Predictive Controller to generate a dynamic trajectory of the center of mass, then a whole-body closed-loop model-based controller to track it at best. Relatively to Nagasaka's work, we allow frame changes of the preferred force, provide a heuristic to compute the timing of the transition from purely geometrical features and investigate the synchronization problem between the reduced-model preview control and the whole-body controller. Using our framework, we generate a wide range of 3D motions, while accounting for predictable external forces, which includes transporting objects. Simulation scenarios are presented and obtained results are analyzed and discussed.

In this paper we propose a simple way to formulate geometric contact formation to have an arbitrary intersection shape in a robotic (humanoid) posture generation problem. The contact shape is the outcome of our posture generator that is formulated as a non-linear optimization programming to fulfill a large variety of robot intrinsic limitations (e.g. joint and torque limits) and tasks (e.g. desired contact). Starting by defining convex areas of contact on the robot's body and the environment, that we call contact patches, we can generate contacts with arbitrary intersection of a pair of any of these predefined patches. Our geometric contact modeling writes very simply as additional constraints and variables added to the optimization problem, translating the search for an ellipse inscribed in the intersection of the pair of patches we want in contact. The result of our posture generator is then a configuration where contact patches are not necessarily included in one another. This allows our posture generator to propose contacts of different shapes with a non-predefined number of contact points (used later to compute reaction/contact forces). We illustrate the efficiency of our method in multi-contact posture generation with the HRP-2 and ATLAS humanoid robots with results that can not be generated automatically by existing methods.

We propose a new bounding volume that achieves a tunable strict convexity of a given convex hull. This geometric operator is named sphere-tori-patches bounding volume (STP-BV), which is the acronym for the bounding volume made of patches of spheres and tori. The strict convexity of STP-BV guarantees a unique pair of witness points and at least C1continuity of the distance function resulting from a proximity query with another convex shape. Subsequently, the gradient of the distance function is continuous. This is useful for integrating distance as a constraint in robotic motion planners or controllers using smooth optimization techniques. For the sake of completeness, we compare performance in smooth and nonsmooth optimization with examples of growing complexity when involving distance queries between pairs of convex shapes.

This paper presents a new control framework for virtual humans (VHs) in a physics-based virtual environment. This framework combines multiobjective control with motion capture techniques. Each motion-tracking task is associated with a task wrench. Bounds are imposed on lower priority task wrenches to ensure the controller performance of higher priority tasks. An optimization problem is solved to compute optimal task wrenches that are based on wrench bounds. Finally, joint torques are computed using the optimal task wrenches. The novelty of our wrench-bound method is that it can handle inequality constraints on a higher priority task and maintain passivity as well. This control framework allows an operator to interact with the VH in real time, without the necessity of compromising the VH's balance. It also allows the VH to generate appropriate motions to handle interactions with the virtual environment, rather than to simply emulate captured motions. The effectiveness of our approach is demonstrated by a VH performing reaching and manipulation tasks.

This paper proposes a new framework of online hybrid control for virtual characters, which combines multi-objective control and motion capture techniques. At each time step, the motions of the operator are captured and sent to the character. These captured motions help the character understand what task the operator wants it to perform. Then, the character decides by itself how to execute proper actions while maintaining its balance, in a virtual environment which is different from the real world. Given a manipulation task, the desired virtual task wrench is computed thanks to a force control approach. The control system takes these task wrenches into consideration and solves a constrained quadratic problem for joint torques, which are then used to drive the virtual character. The whole control problem is solved online so that the virtual character can interact in real-time with the virtual environment as well as the operator. Examples using this control framework are presented as results from simulation.

Classically, the inverse kinematics is performed by computing the singular value decomposition of the matrix to invert. This enables a very simple writing of the algorithm. However, the computation cost is high, especially when applied to complex robots and complex sets of constraints (typically around 5ms for 50 degrees of freedom - DOF). In this paper, we propose a dedicated adaptation of quadratic programming that enables fast computations of the hierarchical inverse kinematics (around 0.1ms for 50 DOF). We then extend this algorithm to deal with unilateral constraints, obtaining sufficiently high performances for reactive control.

STP-BV is a bounding volume made of patches of spheres and toruses. These patches are assembled so that a convex polyhedral hull is bulged, in a tunable way, into a strictly convex form. Strict convexity ensures at least C1property of the distance function -and hence, its gradient continuity. STP-BV were introduced in our previous work [1], but proximity distance queries were limited to pairs of STP-BV covered objects. In this work we present an alternative to achieve fast proximity distance queries between a STP-BV object and any other convex shape. This is simply made by proposing a support mapping for STP-BV to be used with GJK algorithm [2] and its EPA extension to penetration cases [3]. Implementation and experiments of the proposed method and its performance are demonstrated with potential applications to robotics and computer graphics.

We present a motion planning algorithm that computes rough trajectories used by a contact-points planner as a guide to grow its search graph. We adapt collision-free motion planning algorithms to plan a path within the guide space, a submanifold of the configuration space included in the free space in which the configurations are subject to static stability constraint. We first discuss the definition of the guide space. Then we detail the different techniques and ideas involved: relevant C-space sampling for humanoid robot, task-driven projection process, static stability test based on polyhedral convex cones theory's double description method. We finally present results from our implementation of the algorithm.

A novel approach is presented which aims at building autonomously visual models of unknown objects, using a humanoid robot. Previous methods have been proposed for the specific problem of the next-best-view during the modeling and the recognition process. However our approach differs as it takes advantage of humanoid specificities in terms of embedded vision sensor and redundant motion capabilities. In a previous work, another approach to this specific problem was presented which relies on a derivable formulation of the visual evaluation in order to integrate it with our posture generation method. However to get rid of some limitations we propose a new method, formulated using two steps: (i) an optimization algorithm without derivatives is used to find a camera pose which maximizes the amount of unknown data visible, and (ii) a whole robot posture is generated by using a different optimization method where the computed camera pose is set as a constraint on the robot head.

This video illustrates our work on contact points planning and its recent enhancement by two new functionalities. First, by taking advantage of the possibilities offered by our initial posture generator, we include additional tasks in the planning that are not related to locomotion. Second, we refine the potential function that guides the planner so as to cope with more challenging scenarios. We then test these novelties on difficult problems with success, and experiment the output of one of the planned scenario on a HRP-2 humanoid robot.

This paper extends our previous work on contact points planning in two ways. First, by taking advantage of the possibilities offered by our initial posture generator, we include additional tasks that are not related to locomotion within the planning. The output motion will be generated so as to cope with these tasks. Second, we refine the potential function that guides the planner by introducing potential fields acting each on a single body. This helps escaping local minima of the original potential field and thus to deal with more challenging scenarios. We then test these novelties on difficult problems with success, and experiment the output of one of the planned scenario on a HRP-2 humanoid robot.

This paper proposes a real-time implementation of collision and self-collision avoidance for robots. On the basis of a new proximity distance computation method which ensures having continuous gradient, a new controller in the velocity domain is proposed. The gradient continuity encompasses no jump in the generated command. Included in a stack of tasks architecture, this controller has been implemented on the humanoid platform HRP-2 and experienced in a grasping task while walking and avoiding collisions with the environment and auto-collisions.

This paper deals with the motion planning of a polyarticulated robotic system for which support contacts are allowed to occur between any part of the body and any part of the environment. Starting with a description of the environment and of a target, it computes a sequence of postures that allow our system to reach its target. We describe a very generic architecture of this planner, highly modular, as well as a first implementation of it. We then present our results, both simulations and real experiments, for a simple grasping task using the HRP-2 humanoid robot