To efficiently achieve complex humanoid loco-manipulation tasks in industrial contexts, we propose a combined vision-based tracker-localization interplay integrated as part of a task-space whole-body optimization control. To achieve good perception complementarity between manipulation and localization, a new fast dense 3D model-based tracking using wide-angle depth image is developed and used in conjunction with a simultaneous localization and mapping software. Our approach allows humanoid robots, targeted for industrial manufacturing, to manipulate and assemble large-scale objects while walking. It is assessed with experiments consisting in rolling and assembling in an unwinder a heavy and wide bobbin using bimanual grasping and bipedal locomotion at a time. This experimental use-case is found in some large-scale manufacturing where bobbins are enrolled with various materials (cables, papers, rubbers, etc.). The same experiments are made using two different humanoid robots of the same family.Note to Practitioners—This paper aims at deploying humanoid robots in large-scale manufacturing industries. We consider non-added value tasks related to transporting large tools or objects such as large bobbins by means of locomanipulation skills, similarly to human workers. We developed a task-space control framework that has been successfully applied in the aircraft industry. In the frame of a current collaboration with other major industrial sectors, we enhanced our control framework to interplay between SLAM and visual tracking to realize robust loco-manipulation tasks. Our approach can be applied and ported to any humanoid robot or bi-manual wheeled mobile robots with minor programming effort as the software is made open. Preliminary experiments with two different humanoids and use-cases suggest that our approach is feasible. In future research, we will address the problem of performance to reach at least human-speed in the execution of locomanipulation tasks in large-scale industry and automation contexts.

In this paper, we propose a trajectory planning framework for a robot that exploits a pre-computed database of end-effector trajectories as the guidance of optimization-based inverse kinematics. We constructed a reachable graph of a robot offline, which represents feasible end-effector paths with corresponding configurations. When performing the online trajectory planning, we applied A* search to the reachable graph to find a feasible path between input start and goal globally in the task space. Its cost function has the separated term dependent on the robot, which comes from the manipulability of configurations preserved in the reachable graph, and that is dependent on the environment. Then, we solve optimization-based inverse kinematics to generate an optimal joint trajectory while utilizing the end-effector trajectory and its corresponding configurations as the guidance to avoid local optimum. We evaluated our framework quantitatively by comparing it with existing methods to confirm that it achieved a high success rate and quality of results while suppressing its computational time. We also qualitatively proved its practicality by applying it to the material handling task in the real-world. This result shows that it improved the performance of the optimization-based inverse kinematics avoiding local optimum and applicability to the different environments of the pre-computed motion database.

It is necessary to use not only foot but also hand, knee and other body parts to support body weight for locomotion in uneven terrain. Such multi-contact motion planning is an important research topic including lots of previous works; however, a problem of computational speed of planning is still remaining. In this paper, we propose a learning-based algorithm to speed up the planning. The algorithm reduces replanning of contact states by learning an evaluation function of key pose to reach goal. We investigated the learning performance by comparing three neural network configurations and two activation function. This research aims at achieving robust robotics system in unknown environments.

Regrasping is one of the most common and important manipulation skills used in our daily life. However, aerial regrasping has not been seriously investigated yet, since most of the aerial manipulator lacks dexterous manipulation abilities except for the basic pick-and-place. In this paper, we focus on pivoting a long box, which is one of the most classical problems among regrasping researches, using a transformable multilink aerial robot. First, we improve our previous controller by compensating for the external wrench. Second, we optimize the joints configuration of our transformable multilink drone for stable grasping form under the constraints of thrust force and joints effort. Third, we sequentially optimize the grasping force in the pivoting process. The optimization goal is to generate continous grasping force whilst maximizing the friction force in case of the downwash, which would influence the grasped object and is difficult to model. Fourth, we develop the impedance controller in joint space and admittance controller in task space. As far as we know, it is the first research to achieve extrinsic contact-aware regrasping task on aerial robots.

The purpose of this study is to make life-sized humanoid robots acquire tool manipulation skills that require complicated force adjustment. The difficulty in acquisition of tool manipulation skills comes from the hardship in physical modeling. Recent research have revealed that deep reinforcement learning (DRL), a model-free approach, performs superior in such tasks. However, DRL in general has a drawback in sample efficiency, and this becomes critical in robot learning especially in life-sized humanoid robots. In this study, we propose an integrated system incorporating DRL method and active learning. Our method also leverages a variety of previous studies on life-sized humanoid robots to overcome the sample efficiency issue. We demonstrated the effectiveness of our proposed system through a hacksaw skill acquisition and a Japanese planer (Kanna) skill acquisition by a life-sized humanoid robot.

We present a novel transformable multilinked aerial robot which consists of link modules with 1 DoF thrust vectoring mechanism. Commonly used UAV is underactuated due to its simplicity and high flight duration, but can not control the position and orientation independently. To overcome this problem, fully actuated multirotor aerial robots have been developed. In our previous work we developed fully actuated multilinked aerial robot which can transform in the air. However, the transformation range was limited because of a singularity problem. In this paper we propose a new design of link module with a tilted rotor and 1 DoF thrust vectoring joint which enables to avoid singularity forms and keep the flight stable during transformation. We describe modeling and control for the fully actuated multilinked multirotor. Then we propose a transformation planning method utilizing the 1 DoF thrust vectoring angle with consideration of guaranteed minimum force/torque. Finally we perform an aerial transformation experiment with a real platform to demonstrate the feasibility of our proposed design and methods.

In this paper, we propose an approach for online whole-body control of position-controlled humanoid robot with 3D multi-contact to cope with contact wrench constraints and joint overload. In our method, robots are controlled under contact wrench constraints with three features: 1) internal wrench control to reduce joint load and prolong the time in which the high-load postures can be maintained 2) feasible utilization of self-collision to reduce joint load by turning off joint servo gains 3) handling degenerated degree of freedom by solving a quadratic optimization problem integrating wrench distribution and inverse kinematics in which internal wrench is controlled only in controllable directions.With our methods, HRP2-JSKNTS could pick up an object under a desk with squatting with the back of the upper leg on the back of the lower leg without sliding at the right arm. We also evaluated the effectiveness of our control to reduce joint load with another experiment.

When a humanoid robot manipulates a heavy object, balance and heavy loads become problems. To solve these problems, we provide a method to generate a key pose sequence of the robot and the object which balance constraints and joint torque limits are kept. As we consider the configurations of both the robot and the object, the key poses of them are optimized in a view of kinematics and statics. Moreover, we generate a smooth key pose sequence by adding an objective function which makes adjacent poses closer. By using the proposed method, we make a humanoid robot RHP4B place a heavy suitcase on a step.

For robots to autonomously achieve manipulation tasks in various scenes, advanced operational skills such as tool use, learning from demonstration, and multi-robot/human-robot cooperation are necessary. In this research, we devise a method for robots to realize such operational skills in a unified manner by evaluating physical consistency (referred to as “physics reasoning”) based on the formulation of the manipulation statics constraints. First, we propose manipulation planning and estimation methods in which the operational feasibility and properties' likelihood are derived by physics reasoning. In addition, we propose a framework to manipulate an object with unknown physical properties by executing planning and estimation both sequentially and in parallel. We demonstrate the effectiveness of the proposed methods by performing experiments in which real humanoid robots achieve various manipulation tasks with advanced operational skills.

In this paper, we propose methods for walking on a steep slope using a rope by a humanoid robot. There are two difficulties for walking on a steep slope without a rope. First, range of motion of ankle joints get limited. Second, feet of a robot slip on a steep slope. For these problems, using a rope is effective solution because the robot can receive enough friction force from the slope and walk on a steep slope by pulling a rope with proper tension. In addition, the robot pulling a rope on a slope can relax limitations of ankle joints. Therefore, we propose methods to determine tension of a grasped rope by solving a linear least-square problem considering deformability of a rope. With these methods, a life-size humanoid robot HRP-2 could walk on a steep slope which angle is 40 degree.

Whole-body manipulation is necessary for a humanoid robot to achieve tasks such as carrying large objects. One difficulty for achieving a whole-body manipulation is that the robot needs to select the appropriate operation from a list of candidates, such as lifting, pushing, and tilting. The appropriate operation depends upon the target object's physical properties, including its mass, center of mass, and friction coefficient, which are difficult to measure directly. In order to select the appropriate manipulation motion online, we propose a method of estimating the object's physical properties and evaluating the feasibility of the object operation. We calculate the likelihood of the object's physical properties from sensor information during manipulation and update these properties' probabifity distribution periodically based on Bayesian methods. The operational feasibility probability is evaluated by physics-based stability determination, allowing the robot to perform manipulation tasks by selecting the appropriate operation. We show the effectiveness of the proposed method by an experiment in which a life-sized humanoid robot carries a large object.

This paper shows a method for achieving multi-contact motion for a humanoid robot on a movable object, such as climbing of a stepladder. Recent research has developed methods for achieving multi-contact motion that considers various constraints, such as joint limits, torques, balance constraints, reachability, and collision avoidance. In addition to these constraints, Motion On a Movable Object (MOMO) has the following features: it has to consider an object's balance during the changing of contact points; and it has to handle scenarios where the mass properties of an object are unknown. In this paper, in order to achieve a humanoid robot having MOMO, we propose balance constraints that consider the constraints imposed by an object as well as an online estimation of object's constraints. First, we use object-environment constraints as the robot's constraints, and then we show a method for estimating them based on information provided by the robot's sensors. Next, we show a method for applying the balance constraints to a humanoid motion planner and for executing planned motion with real-time sensor feedback controller. Finally, we evaluate our proposed method through experiments in which a life-sized humanoid robot climbs stepladders that have unknown mass properties.

Groping behavior based on contact sensors is necessary for manipulation in an unknown environment. For those situations, it is effective for a robot to accumulate contact information as an environment map, and to plan the motions for executing the safe trial motion. We first propose a method of updating the occupancy grid map of the manipulation region from the contact information by introducing the contact sensor model. Using this map, we propose a method of sampling-based motion planning that enables the execution of the safe trial motion based on the criteria of feasibility and safety. To verify the effectiveness, we show the experimentally obtained results, showing that a real robot plans and executes the manipulation with groping behavior in the occluded environment.

Humanoid robot is able to execute various behavior to manipulate objects because of high degree-of-freedom around the whole-body. Although hands contact with objects and exert force in ordinary pushing motion by robot, pushing motion contacting with the object at various regions of whole-body has potential for extending the scope of feasible manipulation. We derive the fundamental formulas of humanoid robot in the situation that the external force is applied to the arbitrary region of whole-body, and then propose the method to generate and execute the pushing motion based on the formulas. The proposed method is generalized for enabling to select a contact point with an object from whole-body regions and control the pushing force applied to the sensorless region. In order to verify the effectiveness, we show the experimental result that a lifesized humanoid carries large and heavy objects by pushing with various regions of whole-body.

Whole-body holding manipulation is effective for carrying the handleless large object. In order to keep the object stability, the dexterous transition motion is necessary. From geometric and physical conditions of object manipulation, we propose the general method of generating the transition graph, which represents the object pose and grasp contact. By searching the path on the graph, the transition motion is planned automatically with considering the object motion and contact switching simultaneously. By generating and modifying the whole-body holding motion, the planned object motion is achieved stably. We show the effectiveness of the proposed method by the experiments, in which robot lifts up a large object with whole-body contact by the planned transition motion.

In case that a robot carries large and heavy objects with unknown physical parameters such as mass automatically, the autonomous decision and execution of the manipulation strategy are necessary. The method to decide the proper strategy from the various candidates depending on the object is a difficult problem and not researched widely. We consider the operation as the mapping from the physical parameter space to the object motion space. Based on the concept of mapping, we define the strategy proving operation (SPO) for determination of strategy feasibility. We introduce two examples of SPO and construct the system for deciding strategy from lifting, pushing, and pivoting. Executing the strategy in the situation that physical parameters are not known is also necessary. We construct the generator and controller for the full-body manipulation, which can be employed regardless of strategy. The controller enables the robot to exert adequate force while keeping balance. We clarify the applicable scope of the proposed method and show that a life-sized humanoid decides the strategy and carries various large and heavy objects autonomously through the experiment.

For humanoid robots to perform whole-body motions, a motion planner should generate feasible motions satisfying various constraints including joint torque limitation, friction, balancing, collision, and so on. Furthermore, for life-size humanoid robots to perform higher-load motions, such as climbing ladders, safely, it is important to generate motions which requirements are not too close to the limitations. In this paper, we propose a humanoid motion planner based on Body Retention Load Vector (BRLV), which is a novel index for representing severity of physical constraints: limitation of joint Torque, contact Force, and contact Moment (TFM limitations). By minimizing the norm of BRLV, we obtain humanoid motions that are farthest from TFM limitations. Finally, we evaluate the proposed motion planner in simulation and confirm the effectiveness of the planner through experiments in which a life-size humanoid robot climbs a ladder and a car.

In the case of object manipulation by a humanoid robot, it is important to deal with contact states between objects, a robot, and an environment both to avoid falling down and to achieve objective manipulations. We propose a method to describe and uniformly execute various object manipulations by a humanoid robot. In description, we focus on the contact states and define manipulation phases according to the contact states. In execution, the humanoid's controller autonomously switches manipulation phases and substantiates the contact-force controller. According to switching of the manipulation phases, the humanoid's manipulation system switches the inputs for the contact-force controller, which includes the estimation of object's information and motion generation. We evaluated our proposed system through experiments in which the HRP-2 robot manipulates four objects without information about the objects' masses and necessary operational forces.