In recent years, multimodal locomotion capabilities have enabled robots to maneuver in both terrestrial and aerial domains. However, most of these robots are designed only for locomotion, and few possess the manipulation capabilities required for practical tasks. By adding a manipulator, ground robots can perform manipulation, and some drones with robotic arms have demonstrated aerial manipulation. Nonetheless, such multirotors cannot be directly used for manipulation on the ground, and this configuration itself is unsuitable for air-ground hybrid locomotion. This is because their thruster-centralized structure makes it difficult to achieve both sufficient degrees of freedom (DoF) for manipulation and stable motion with contact and transformation. Therefore, in this work, we develop a new multilink multirotor with thrusters on each link and capable of contact with the environments. This robot can perform terrestrial rolling locomotion, aerial flight locomotion, and manipulation in multiple environments using joint actuation. First, we introduce a minimal configuration design of the proposed robot. We also describe a kinematic model and propose a design for each component based on this model. Second, we propose a real-time control method based on nonlinear optimization that considers contact and joint motion, which can be applied to various multirotors. Third, we propose motion strategies that include contact constraints specific to air-ground hybrid multilink multirotors, and analyze the limitations of manipulation capabilities based on multi-contact model. Finally, we demonstrate a variety of motions in both domains using the implemented prototype. To the best of our knowledge, this is the first demonstration of air-ground hybrid locomotion and manipulation by a multilink multirotor.

In order to expand the operational range and payload capacity of robots, wire-driven robots that leverage the external environment have been proposed. It can exert forces and operate in spaces far beyond those dictated by its own structural limits. However, for practical use, robots must autonomously attach multiple wires to the environment based on environmental recognition―an operation so difficult that many wire-driven robots remain restricted to specialized, pre-designed environments. Here, in this study, we propose a robot that autonomously connects multiple wires to the environment by employing a multi-small flying anchor system, as well as an RGB-D camera-based control and environmental recognition method. Each flying anchor is a drone with an anchoring mechanism at the wire tip, allowing the robot to attach wires by flying into position. Using the robot’s RGB-D camera to identify suitable attachment points and a flying anchor position, the system can connect wires in environments that are not specially prepared, and can also attach multiple wires simultaneously. Through this approach, a wire-driven robot can autonomously attach its wires to the environment, thereby realizing the benefits of wire-driven operation at any location.

Tendon-driven mechanisms are useful from the perspectives of variable stiffness, redundant actuation, and lightweight design, and they are widely used, particularly in hands, wrists, and waists of robots. The design of these wire arrangements has traditionally been done empirically, but it becomes extremely challenging when dealing with complex structures. Various studies have attempted to optimize wire arrangement, but many of them have oversimplified the problem by imposing conditions such as restricting movements to a 2D plane, keeping the moment arm constant, or neglecting wire crossings. Therefore, this study proposes a three-dimensional wire arrangement optimization that takes wire crossings into account. We explore wire arrangements through a multi-objective black-box optimization method that ensures wires do not cross while providing sufficient joint torque along a defined target trajectory. For a 3D link structure, we optimize the wire arrangement under various conditions, demonstrate its effectiveness, and discuss the obtained design solutions.

For trajectory generation of whole-body jumping motions such as humanoid backflips, it is crucial to simultaneously optimize the takeoff, flight, and landing phases while considering full-body dynamics and kinematics. Although such methods have been proposed for standard jumping motions, they have not been applied to more dynamic actions such as frontflips, backflips, and yaw twist jumps, where strong nonlinearity and high sensitivity to certain parameters (e.g., rotor inertia and torque cost weights) pose significant challenges. To address these challenges, we apply a two-stage optimization strategy to an existing full-body dynamics optimization method that simultaneously optimizes the takeoff, flight, and landing phases. In our approach, the same initialization and reference trajectory generation rules are shared across motions, and the solution from the first optimization is used not only as an initial guess but also as a reference in the second optimization. This strategy improves the convergence of the KKT residuals across various jump types and mitigates sensitivity to parameters such as rotor inertia and torque cost weights. As a result, our method achieves unified trajectory generation for frontflips, backflips, yaw twist jumps, and standard jumps using the same initialization, cost weights, and constraints. We also analyze the sensitivity to rotor inertia and show that exceeding a certain threshold can lead to a sharp deterioration in KKT residual convergence.

Various methods have been proposed to achieve high output torque and a wide output range for fast and high-load robotic motions. However, in robots composed of slender frames, such as humanoid robots, the limited space available for actuators and transmission components restricts the application of conventional methods. In this paper, we propose a Variable Chain Motor (VC Motor), an electric actuator that features both shape variability and speed-torque characteristics variability. Shape variability refers to the ability of the actuator to change its form during operation. This property enhances output torque by enabling a dense motor arrangement even under spatial constraints imposed by the frame structure. For example, the actuator can be placed across adjacent frames and deform according to joint rotation. Speed-torque characteristics variability allows switching output characteristics during operation using a dedicated electrical circuit. This enables an expanded range of output speed and torque without significantly increasing size or weight. We evaluated the performance of the developed VC Motor by measuring output torque and efficiency. Furthermore, by applying the VC Motor to the elbow joint of a humanoid robot, we demonstrated its capability for high-speed and high-load operations.

In recent years, advancements in hardware have enabled quadruped robots to operate with high power and speed, while robust locomotion control using reinforcement learning (RL) has also been realized. As a result, expectations are rising for the automation of tasks such as material transport and exploration in unknown environments. However, autonomous locomotion in rough terrains with significant height variations requires vertical movement, and robots capable of performing such movements stably, along with their control methods, have not yet been fully established. In this study, we developed the quadruped robot KLEIYN, which features a waist joint, and aimed to expand quadruped locomotion by enabling chimney climbing through RL. To facilitate the learning of vertical motion, we introduced Contact-Guided Curriculum Learning (CGCL). As a result, KLEIYN successfully climbed walls ranging from 800 mm to 1000 mm in width at an average speed of 150 mm/s, 50 times faster than conventional robots. Furthermore, we demonstrated that the introduction of a waist joint improves climbing performance, particularly enhancing tracking ability on narrow walls.

Tactile sensors have a wide range of applications, from utilization in robotic grippers to human motion measurement. If tactile sensors could be fabricated and integrated more easily, their applicability would further expand. In this study, we propose a tactile sensor―M3D-skin―that can be easily fabricated with high versatility by leveraging the infill patterns of a multi-material fused deposition modeling (FDM) 3D printer as the sensing principle. This method employs conductive and non-conductive flexible filaments to create a hierarchical structure with a specific infill pattern. The flexible hierarchical structure deforms under pressure, leading to a change in electrical resistance, enabling the acquisition of tactile information. We measure the changes in characteristics of the proposed tactile sensor caused by modifications to the hierarchical structure. Additionally, we demonstrate the fabrication and use of a multi-tile sensor. Furthermore, as applications, we implement motion pattern measurement on the sole of a foot, integration with a robotic hand, and tactile-based robotic operations. Through these experiments, we validate the effectiveness of the proposed tactile sensor.

This study explores interactive scripted performance with a character robot to create an engaging storytelling experience. By involving human participants as performers alongside the robot, we investigated whether this approach could enhance engagement in the storytelling experience. We developed a character robot system capable of executing predefined phrases and movements to foster immersion, thereby enhancing engagement. We conducted interactive scripted performance events at after-school care facilities. The results indicate that interactive scripted performance encouraged participants to engage in the experience actively. Moreover, the findings suggest that the developed character robot enhanced participants’ story immersion and encouraged physical engagement.

Science Fiction Prototyping (SFP) is a method that uses science fiction to imagine future technologies and foster innovation. It is considered effective for exploring human-robot relationships and envisioning better robot designs. However, robot embodiment influences human perception, which plays a crucial role in interaction. Simply imagining future scenarios with robots through SFP may overlook these aspects. We propose an approach called Experiential Science Fiction Prototyping (ESFP), which adds a phase of experiencing the story to the traditional SFP process. To explore the effects of ESFP, we conducted a workshop with Japanese teenagers under the theme of designing a robot that contributes to a sense of “ibasho”—a Japanese concept referring to a space or relationship where one feels accepted and comfortable. ESFP unfolds in three phases: Storytelling, where participants envision future lives with robots and create stories; Experience, where they bring these stories to life through interaction with a physical robot; and Discussion, where they reflect on the story they created and experienced. The results suggested that, through the experiential phase, participants developed new ideas about interaction with robots and expanded their imagination about future relationships. Experiencing the story helped participants connect more closely with the envisioned robot interactions and inspired new reflections and expectations. This study contributes by proposing the ESFP method, detailing its implementation, and discussing its potential through a case study.

Library-based methods are known to be very effective for fast motion planning by adapting an experience retrieved from a precomputed library. This article presents CoverLib, a principled approach for constructing and utilizing such a library. CoverLib iteratively adds an experience-classifier-pair to the library, where each classifier corresponds to an adaptable region of the experience within the problem space. This iterative process is an active procedure, as it selects the next experience based on its ability to effectively cover the uncovered region. During the query phase, these classifiers are utilized to select an experience that is expected to be adaptable for a given problem. Experimental results demonstrate that CoverLib effectively mitigates the tradeoff between plannability and speed observed in global (e.g., sampling-based) and local (e.g., optimization-based) methods. As a result, it achieves both fast planning and high success rates over the problem domain. Moreover, due to its adaptation-algorithm-agnostic nature, CoverLib seamlessly integrates with various adaptation methods, including nonlinear programming-based and sampling-based algorithms.

Various robots have been developed so far; however, we face challenges in modeling the low-rigidity bodies of some robots. In particular, the deflection of the body changes during tool-use due to object grasping, resulting in significant shifts in the tool-tip position and the body’s center of gravity. Moreover, this deflection varies depending on the weight and length of the tool, making these models exceptionally complex. However, there is currently no control or learning method that takes all of these effects into account. In this study, we propose a method for constructing a neural network that describes the mutual relationship among joint angle, visual information, and tactile information from the feet. We aim to train this network using the actual robot data and utilize it for tool-tip control. Additionally, we employ Parametric Bias to capture changes in this mutual relationship caused by variations in the weight and length of tools, enabling us to understand the characteristics of the grasped tool from the current sensor information. We apply this approach to the whole-body tool-use on KXR, a low-rigidity plastic-made humanoid robot, to validate its effectiveness.

In this study, we present an implementation strategy for a robot that performs peg transfer tasks in Fundamentals of Laparoscopic Surgery (FLS) via imitation learning, aimed at the development of an autonomous robot for laparoscopic surgery. Robotic laparoscopic surgery presents two main challenges: (1) the need to manipulate forceps using ports established on the body surface as fulcrums, and (2) difficulty in perceiving depth information when working with a monocular camera that displays its images on a monitor. Especially, regarding issue (2), most prior research has assumed the availability of depth images or models of a target to be operated on. Therefore, in this study, we achieve more accurate imitation learning with only monocular images by extracting motion constraints from one exemplary motion of skilled operators, collecting data based on these constraints, and conducting imitation learning based on the collected data. We implemented an overall system using two Franka Emika Panda Robot Arms and validated its effectiveness.

We propose the method to adapt humanoids the ability to change the body structures that modular robots have by using Attach-Lock-Detachable Magnetic Couplings(ALDMag) to give the ability to detach and attach the robot body with an arm-type robot, and the system to manage the connection state of modularized body elements. Robots and we can use the ALDMag to attach and detach mechanical and electrical connections without actuators. Using xacro for writing the file of the robot model description of each module, we can construct a system that allows the robot to attach and detach modules during task operation. We demonstrated the effectiveness of the proposed method by achieving assembly experiments of a small robot with a life-size arm and experiments with environmental contacts by the small robot.

In this paper, we explain the Morphable StateNet as the StateNet with pseudo-generalized behaviors for robots with various degree-of-freedom arrangements and link lengths. Pseudo-generalization is performed by analytically calculating joint angles that satisfy the desired support conditions, focusing on link lengths and antigravity joints that contribute to motion, with constraints placed on the contact conditions between the environment and the robot body. We apply Morphable StateNet to the standing-up motion of humanoids with variable body structures and conduct evaluation experiments. We have demonstrated the usefulness of the proposed method in environments with low friction coefficients with the environment by conducting evaluations using both a simulator and an actual humanoid.

A robotic hand for social interaction should be capable of comfortable touch with humans. However, it is difficult to mount skin, tactile sensors, and driving mechanism required for human contact, especially holding hands, on a slender finger. In addition, in order to unitize the hand for easy use with any robot and maintainability, the mechanism must be contained within the small space of the fingers and palms. In this paper, we propose a human-sized five-fingered robotic hand named WARABI Hand. It is covered with multi-layored rubber skin to realize human-like soft and pleasant feel. Force sensors on each finger link detect contact with humans and adjust gripping force. We conducted experiments in which a humanoid equipped with WARABI Hand grasped forearm, held hands, and interlocked fingers with a person. The performance for object grasping was also evaluated. We demonstrated that our proposed hand is useful for interaction with humans including receiving and handing over things.

Transferring human motion skills to humanoid robots remains a significant challenge. In this study, we introduce a Wasserstein adversarial imitation learning system, allowing humanoid robots to replicate natural whole-body locomotion patterns and execute seamless transitions by mimicking human motions. First, we present a unified primitive-skeleton motion retargeting to mitigate morphological differences between arbitrary human demonstrators and humanoid robots. An adversarial critic component is integrated with Reinforcement Learning (RL) to guide the control policy to produce behaviors aligned with the data distribution of mixed reference motions. Additionally, we employ a specific Integral Probabilistic Metric (IPM), namely the Wasserstein-1 distance with a novel soft boundary constraint to stabilize the training process and prevent model collapse. Our system is evaluated on a full-sized humanoid JAXON in the simulator. The resulting control policy demonstrates a wide range of locomotion patterns, including standing, push-recovery, squat walking, humanlike straight-leg walking, and dynamic running. Notably, even in the absence of transition motions in the demonstration dataset, the robot showcases an emerging ability to transit naturally between distinct locomotion patterns as desired speed changes.

With the background of research on materials laboratory automation, this study aims to construct an automation system for measuring dielectric property, which is an electrical property of materials. The automation system is composed of the combination of a manipulation by a force-sensing robot and a control system for the measurement instrument. As challenges for the automation system, we worked on stabilizing a polymer film placement during insertion into the measurement instrument, implementing a communication control system between different platforms, and constructing a polymer film transfer environment. In the measurement experiment using the automation system, it was confirmed that the dielectric properties could be measured as well as that of a human.

For humanoid whole body contact motions, it is important to recognize the existence of whole body contacts and the contact forces. The challenges in recognizing the existence of whole body contacts and the contact forces in life-size humanoids are: 1) the measurement part with low mechanical strength must be tolerant of high load and 2) it is difficult to model thick elastic bodies with high impact tolerance and uneven sensor placements when applied to various shapes of the whole body. This paper proposes a method of constructing a load tolerant tactile sensor by separating the loaded part from the measuring part with magnetism and protecting the measuring part inside the frame of the robot. For modeling difficulties, this paper proposes learning the relationship between the change in the detected physical quantity due to deformation of the elastic body and the contact force distribution. This paper shows through experiments that the proposed tactile sensor based on a robot frame is load tolerant enough to support the weight of a life-sized humanoid, and that it can acquire contact force distribution and the robot is able to acclimate to external forces.

A wire-driven parallel robot is a type of robotic system where multiple wires are used to control the movement of a end-effector. The wires are attached to the end-effector and anchored to fixed points on external structures. This configuration allows for the separation of actuators and end-effectors, enabling lightweight and simplified movable parts in the robot. However, its range of motion remains confined within the space formed by the wires, limiting the wire-driven capability to only within the pre-designed operational range. Here, in this study, we develop a wire-driven robot, CubiX, capable of connecting to and utilizing the environment. CubiX connects itself to the environment using up to 8 wires and drives itself by winding these wires. By integrating actuators for winding the wires into CubiX, a portable wire-driven parallel robot is realized without limitations on its workspace. Consequently, the robot can form parallel wire-driven structures by connecting wires to the environment at any operational location.

Robots generally have a structure that combines rotational joints and links in a serial fashion. On the other hand, various joint mechanisms are being utilized in practice, such as prismatic joints, closed links, and wire-driven systems. Previous research have focused on individual mechanisms, proposing methods to design robots capable of achieving given tasks by optimizing the length of links and the arrangement of the joints. In this study, we propose a method for the design optimization of robots that combine different types of joints, specifically rotational and prismatic joints. The objective is to automatically generate a robot that minimizes the number of joints and link lengths while accomplishing a desired task, by utilizing a black-box multi-objective optimization approach. This enables the simultaneous observation of a diverse range of body designs through the obtained Pareto solutions. Our findings confirm the emergence of practical and known combinations of rotational and prismatic joints, as well as the discovery of novel joint combinations.

To give the single robot system the ability to realize many behaviors and to realize tasks with shifting environments and objectives, it is necessary to abstract the robot’s body ability to the extent that they can be detected by sensors in the body so that we can plan as the problem of state transition.In this paper, to abstract the transformer robot system that performs heavy lifting and environmental attachment tasks in an aquatic and terrestrial environment, we extend the graphical representation of the robot’s body to manage joint capability and the body adaptability for the environment. To abstract the body ability, we divide the body into elements and define Connection between them at three different granularities. And using Connection, we propose the Connection Modification Feature(CMF) as the representation for changing body ability. To implement the Connection Modification Feature, we perform the abstract description and extract Connection to construct Body Ability Graph, a graph for the robot to manage its body ability. We show that it is possible to plan to manipulate and use its own Connection Modification Feature through multiple experiments by defining Normal Action that does not change body abilities and Body Ability Modifying Action that manipulate body abilities.

The complex ways in which humans utilize their bodies in sports and martial arts are remarkable, and human motion analysis is one of the most effective tools for robot body design and control. On the other hand, motion analysis is not easy, and it is difficult to measure complex body motions in detail due to the influence of numerous muscles and soft tissues, mainly ligaments. In response, various musculoskeletal simulators have been developed and applied to motion analysis and robotics. However, none of them reproduce the ligaments but only the muscles, nor do they focus on the shoulder complex, including the clavicle and scapula, which is one of the most complex parts of the body. Therefore, in this study, a detailed simulation model of the shoulder complex including ligaments is constructed. The model will mimic not only the skeletal structure and muscle arrangement but also the ligament arrangement and maximum muscle strength. Through model predictive control based on the constructed simulation, we confirmed that the ligaments contribute to joint stabilization in the first movement and that the proper distribution of maximum muscle force contributes to the equalization of the load on each muscle, demonstrating the effectiveness of this simulation.

In order to make robots more useful in a variety of environments, they need to be highly portable so that they can be transported to wherever they are needed, and highly storable so that they can be stored when not in use. We propose "on-site robotics", which uses parts procured at the location where the robot will be active, and propose a new solution to the problem of portability and storability. In this paper, as a proof of concept for on-site robotics, we describe a method for estimating the kinematic model of a robot by using inertial measurement units (IMU) sensor module on rigid links, estimating the relative orientation between modules from angular velocity, and estimating the relative position from the measurement of centrifugal force.At the end of this paper, as an evaluation for this method, we present an experiment in which a robot made up of wooden sticks reaches a target position. In this experiment, even if the combination of the links is changed, the robot is able to reach the target position again immediately after estimation, showing that it can operate even after being reassembled. Our implementation is available on https://github.com/hiroya1224/urdf_estimation_with_imus.

Muscles of the human body are composed of tiny actuators made up of myosin and actin filaments. They can exert force in various shapes such as curved or flat, under contact forces and deformations from the environment. On the other hand, muscles in musculoskeletal robots so far have faced challenges in generating force in such shapes and environments. To address this issue, we propose Patterned Structure Muscle (PSM), artificial muscles for musculoskeletal robots. PSM utilizes patterned structures with anisotropic characteristics, wire-driven mechanisms, and is made of flexible material Thermoplastic Polyurethane (TPU) using FDM 3D printing. This method enables the creation of various shapes of muscles, such as simple 1 degree-of-freedom (DOF) muscles, Multi-DOF wide area muscles, joint-covering muscles, and branched muscles. We created an upper arm structure using these muscles to demonstrate wide range of motion, lifting heavy objects, and movements through environmental contact. These experiments show that the proposed PSM is capable of operating in various shapes and environments, and is suitable for the muscles of musculoskeletal robots.

Exoskeletons for robot operation necessitate shoulders with high range of motions and high degrees of freedom to fit the operator’s shoulder girdle. These shoulder joints need high torque for force feedback on the operator. Existing exoskeletons struggle to simultaneously meet these requirements of high DOFs, wide ROM, and high torque due to spatial constraints. This study introduces an exoskeleton with a distal branching link mechanism that addresses this issue by concentrating on each link’s absolute and relative degrees of freedom. In the proposed exoskeleton, the end-effector’s absolute DOF, the forearm’s absolute DOF, and the end-effector and forearm’s relative DOF are matched between the operator and the exoskeleton. This is achieved while reducing the overall DOF by sharing the root link system’s DOF. Furthermore, by avoiding direct attachment of the operator to the exoskeleton’s shoulder, the design can accommodate the human shoulder’s high torque and high ROM. The study demonstrates that the branching exoskeleton outperforms existing link-fixed exoskeletons in terms of tracking the operator’s arms and the torque required by the exoskeleton’s joints. Utilizing this exoskeleton, we successfully maneuvered an actual humanoid robot to perform daily activities where the forearm posture is crucial.

In Japan, home robots have become popular and are widely loved. In particular, Sony’s aibo users have formed a big community, and a society where humans and robots live in harmony is becoming a reality. In order for robots to be more accepted in society, exploratory analysis and explicit description of their bonds are important. However, the scale for measuring relationships with robots has not yet been defined, so a suitable measure was created from the Friendship Function Scale to measure the relationship of friends and the Pet Awareness Scale to measure the attitude toward pets. Using the proposed scale, we investigated the role of home robots, subjective intimacy, and attitudes toward robots among 54 robot owners. The survey found that more than half of them considered robots to be pets or children. In addition, the average value of subjective intimacy was very high to indicate a very high level of attachment to the robot. Furthermore, as a result of factor analysis of the scale items, five factors were extracted: "Comfort," "Irreplaceability," "Trust from the robot," "Trust to the robot," and "Social life." At this time, there was no bias in how users felt about comfort and irreplaceability, but there was a bias in how users felt about trust. Additionally, those with the highest level of subjective intimacy felt significantly more "trusted by the robot" than those with the other level of intimacy, suggesting that recognition of trust from the robot may be an important bond-forming strategy. In the future, we will aim to form further bonds by investigating relationships with robots that play various roles, as well as in regions other than Japan.

Manipulation performance improvement is crucial for aerial robots. For aerial manipulators, the baselink position and attitude errors directly affect the precision at the end effector. To address this stability problem, fixed-body approaches such as perching on the environment using the rotor suction force are useful. Additionally, conventional arm-equipped multirotors, called rotor-concentrated manipulators, find it difficult to generate a large wrench at the end effector due to joint torque limitations. Using distributed rotors to each link, the thrust can support each link weight, decreasing the arm joints' torque. Based on this approach, rotor-distributed manipulators (RDMs) can increase feasible wrench and reachability of the end effector. This article introduces a minimal configuration of an RDM that can perch on surfaces, especially ceilings, using a part of their body. First, we design a minimal rotor-distributed arm considering the flight and end-effector performance. Second, a flight controller is proposed for this minimal RDM along with a perching controller adaptable for various types of aerial robots. Third, we propose a motion planning method based on inverse kinematics, considering specific constraints to the proposed RDMs, such as perching force. Finally, we evaluate flight and perching motions and confirm that the proposed manipulator can significantly improve the manipulation performance.

State recognition of objects and environment in robots has been conducted in various ways. In most cases, this is executed by processing point clouds, learning images with annotations, and using specialized sensors. In contrast, in this study, we propose a state recognition method that applies Visual Question Answering (VQA) in a Pre-Trained Vision-Language Model (PTVLM) trained from a large-scale dataset. By using VQA, it is possible to intuitively describe robotic state recognition in the spoken language. On the other hand, there are various possible ways to ask about the same event, and the performance of state recognition differs depending on the question. Therefore, in order to improve the performance of state recognition using VQA, we search for an appropriate combination of questions using a genetic algorithm. We show that our system can recognize not only the open/closed of a refrigerator door and the on/off of a display, but also the open/closed of a transparent door and the state of water, which have been difficult to recognize.

Humanoids operate in repeated contact and non-contact with their environment and so the motion of humanoids such as walking on uneven terrain or in a narrow space requires the accurate force and position control. Joint torque control systems are suitable for position and force control, but are prone to friction and other modeling errors. To solve this problem, methods have been proposed to realize torque control in combination with joint position control systems or by improving joint structures such as sensors and actuators, but these methods have problems such as response delay and increased weight and volume. Thus, it is difficult to achieve motion of life-sized humanoids by whole-body torque control. In this paper, we solve challenges not with one specific layer, but rather with multiple layers that complement each other. We propose a hierarchical whole-body torque control method using four layers: friction compensation based on a vibration-suppressed model, whole-body resolved acceleration control using priority, center-of-gravity acceleration control based on foot-guided control, and landing position time modification based on capture point. We verify through walking experiments that the proposed methods can control the life-sized humanoid robot driven by high-reduction ratio joints by whole-body torque control without a torque sensor or joint position control, and that it enables the robot to move and even transport an object on outdoor uneven terrain.

The range of robot activities is expanding from industries with fixed environments to diverse and changing environments, such as nursing care support and daily life support. In particular, autonomous construction of robots that are personalized for each user and task is required. Therefore, we develop an actuator module that can be reconfigured to various link configurations, can carry heavy objects using a locking mechanism, and can be easily operated by human teaching using a releasing mechanism. Given multiple target coordinates, a modular robot configuration that satisfies these coordinates and minimizes the required torque is automatically generated by Tree-structured Parzen Estimator (TPE), a type of black-box optimization. Based on the obtained results, we show that the robot can be reconfigured to perform various functions such as moving monitors and lights, serving food, and so on.

In order for a humanoid robot to balance on the movable ground, balance feedback control in response to its unpredictable movement is required. However, feedback control in response to ground movement has the following two issues, (A) Interaction between the ground dynamics and the balance control may cause vibration. (B) The balance control may rather deteriorate the stability due to the response delay. To solve these problems, this study proposes the support foot acceleration term in the walking stabilizer and gives its gain by considering the following two conditions, (A) Avoiding steady-state vibration in a two-mass linear inverted pendulum model on an arbitrary ground, and (B) reducing the influence of inertial forces resulting from the delay of ZMP feedback. Experiments with a life-size humanoid JAXON verified the steady-state vibration phenomenon and improved the stability of acceleration and deceleration when boarding the Two-Wheeled Scooter.

We need to build robot systems that can operate in multiple environments, including underwater. For a robot to be waterproof, it needs to be covered all over its body and have a waterproof structure, but this is expensive to produce and consumes hardware resources such as weight. In this study, we propose a method of constructing a robot system in which paraffin, which can change its phase between solid and liquid at different temperatures, is cloaked on the robot's body surface, and grease is inserted into the robot's body to give it an acquired the waterproof shell for waterproofing purposes. We have studied an automated method of the waterproof shell application system by combining a small spider-shaped multi-legged robot that can measure the temperature of the environment as a robot to which we apply the waterproof shell and a life-size arm robot that performs the operation applying the waterproof shell to the small robot as a system. Through the addition of the waterproof shell to the spider-shaped robot and the removing the shell in unnecessary areas by controlling temperature, and the realization of operation in land and water environments, we used the proposed robot system to add acquired functions to operate underwater robots, thereby expanding the supported area of robots which have many DOFs and sensors from conventional ground to underwater.

Imitation learning has been actively studied in recent years. In particular, skill acquisition by a robot with a fixed body, whose root link position and posture and camera angle of view do not change, has been realized in many cases. On the other hand, imitation of the behavior of robots with floating links, such as humanoid robots, is still a difficult task. In this study, we develop an imitation learning system using a biped robot with a floating link. There are two main problems in developing such a system. The first is a teleoperation device for humanoids, and the second is a control system that can withstand heavy workloads and long-term data collection. For the first point, we use the whole body control device TABLIS. It can control not only the arms but also the legs and can perform bilateral control with the robot. By connecting this TABLIS with the high-power humanoid robot JAXON, we construct a control system for imi-tation learning. For the second point, we will build a system that can collect long-term data based on posture optimization, and can simultaneously move the robot's limbs. We combine high-cycle posture generation with posture optimization methods, including whole-body joint torque minimization and contact force optimization. We designed an integrated system with the above two features to achieve various tasks through imitation learning. Finally, we demonstrate the effectiveness of this system by experiments of manipulating flexible fabrics such that not only the hands but also the head and waist move simultaneously, manipulating objects using legs characteristic of humanoids, and lifting heavy objects that require large forces.

Robot hands that imitate the shape of the human body have been actively studied, and various materials and mechanisms have been proposed to imitate the human body. Although the use of soft materials is advantageous in that it can imitate the characteristics of the human body's epidermis, it increases the number of parts and makes assembly difficult in order to perform complex movements. In this study, we propose a skin-skeleton integrated robot hand that has 15 degrees of freedom and consists of four parts. The developed robotic hand is mostly composed of a single flexible part produced by a 3D printer, and while it can be easily assembled, it can perform adduction, flexion, and opposition of the thumb, as well as flexion of four fingers.

It is important for daily life support robots to detect changes in their environment and perform tasks. In the field of anomaly detection in computer vision, probabilistic and deep learning methods have been used to calculate the image distance. These methods calculate distances by focusing on image pixels. In contrast, this study aims to detect semantic changes in the daily life environment using the current development of large-scale vision-language models. Using its Visual Question Answering (VQA) model, we propose a method to detect semantic changes by applying multiple questions to a reference image and a current image and obtaining answers in the form of sentences. Unlike deep learning-based methods in anomaly detection, this method does not require any training or fine-tuning, is not affected by noise, and is sensitive to semantic state changes in the real world. In our experiments, we demonstrated the effectiveness of this method by applying it to a patrol task in a real-life environment using a mobile robot, Fetch Mobile Manipulator. In the future, it may be possible to add explanatory power to changes in the daily life environment through spoken language.

In order for a humanoid robot to traverse uneven terrain without falling over, the robot must control its landing position appropriately. To determine the landing position, there are two difficulties in terrain recognition and leg motion control. In terrain recognition, it is difficult to recognize and avoid terrain such as steps and obstacles that cannot be landed on in real-time. In leg motion control, it is necessary to land at appropriate positions and times to control the CoG trajectory while limiting the velocity of the swing-leg to suppress the landing impact. For solving these problems, we propose a recognition and walking control system on uneven terrain. In terrain recognition, we improved the recognition accuracy while satisfying real-time performance by using a CNN that learns the relationship between the foot and the geometric information of the surrounding terrain. In the leg motion control, landing impact was reduced by modifying the landing position under not only (1) terrain constraint and (2) robot stability constraint, but also (3) leg velocity constraint. We verified the effectiveness of the proposed system through uneven terrain walking and push recovery experiments using the actual robot.

In this paper, we focus on the kangaroo, which has powerful legs capable of jumping and a soft and strong tail. To incorporate these unique structure into a robot for utilization, we propose a design method that takes into account both the feasibility as a robot and the kangaroo-mimetic structure. Based on the kangaroo's musculoskeletal structure, we determine the structure of the robot that enables it to jump by analyzing the muscle arrangement and prior verification in simulation. Also, to realize a tail capable of body support, we use an articulated, elastic structure as a tail. In order to achieve both softness and high power output, the robot is driven by a direct-drive, high-power wire-winding mechanism, and weight of legs and the tail is reduced by placing motors in the torso. The developed kangaroo robot can jump with its hind legs, moving its tail, and supporting its body using its hind legs and tail.

Eating with others enriches our lives and has many positive effects. However, numerous people are forced to eat alone in the current COVID-19 pandemic situation. We think that robots that exist with bodies and can be interacted with in real time can be good mealtime partners. In this study, we developed a meal partner robot platform called Mamoru’21, which can share eating behavior. We experimented to evaluate Mamoru’21. The results showed that Mamoru’21 could make eating more enjoyable than a conventional communication robot. We also investigated the preferable appearance of a meal partner robot and confirmed that Mamoru’21 met the requirements.

In this study, we examined scene selection methods and emotion-based descriptions for a robot’s daily diary. We proposed a scene selection method and an emotion description method that take into account semantic and affective information, and created several types of diaries. Experiments were conducted to examine the change in sentiment values and preference of each diary, and it was found that the robot’s feelings and impressions changed more from date to date when scenes were selected using the affective captions. Furthermore, we found that the robot’s emotion generally improves the preference of the robot’s diary regardless of the scene it describes. However, presenting negative or mixed emotions at once may decrease the preference of the diary or reduce the robot’s robot-likeness, and thus the method of presenting emotions still needs further investigation.

Holding hands can give people a sense of security. In this study, we developed a five-fingered robotic hand that can hold hands with a person and a guidance system that uses the developed hand to hold hands with them. In this system, a robot remembers the location where people have taught it and guides people to that point. We conducted an experiment to evaluate the sense of security in guidance with hand-holding compared with guidance without hand-holding. Participants watched the videos on these two conditions and answered a questionnaire. The results confirmed that holding hands can lead to a sense of security.

The humanoid robot that can transform itself into a form according to its purpose requires whole-body motions with complex contact state transitions such as recovery from a fall and transition to the target form. To make the robot behavior in simulations closer to that in the real world for planning complex target trajectories, we need a platform that can measure the body stiffness during the motion and verify its application without being damaged by repeated motions that are prone to tipping over. In this study, we propose a small, inexpensive, and robust humanoid-vehicle transformer platform with redundant sensors and a low rigidity multi degree-of-freedom body and observe the effects of body deflection and internal forces during whole-body posture transition. By comparing the results obtained from experiments in several environments with different friction and from the simulator using a rigid body model, we were able to verify the influence of body flexibility on whole-body motion and the relationship between deflection and wrench observed by redundant sensors and movement failure.

In recent years, an increasing number of research works have been focusing on the manipulation by aerial robots. Previous works using aerial robots with robotic arms have two problems: underactuation and external disturbances. We propose the fully-actuated control method and motion strategy using contact with the environment to solve these problems, along with the mechanical approach required. First, each propeller's 1 degree-of-freedom (DoF) thrust vectoring units are applied to enable fully-actuated flight control. In order to obtain the desired thrust and vectoring angle inputs for aerial manipulation satisfying hardware limits, we developed a fully-actuated control method using non-linear optimization. Second, we propose a manipulation motion strategy that treats the multilink robot body as a fixed manipulator by making contact with the environment. The contact mechanism attached to the link end is developed to maintain contact and resist external disturbances. In a real machine experiment, the robot successfully opened the door while in contact with the wall, demonstrating the feasibility of the proposed methods.

In the literature on object grasping, the robot often determines the grasp point and posture from visual information. They predict the grasping point uniquely from the object's shape characteristics. However, as a practical matter, there are cases where there are constraints on grasp point due to the object states, the limitation of the robot's hardware and the surrounding environment. In this study, we propose a neural network that can easily constrain the input. It determines the grasp pose from visual information and outputs the grasp success probability. The grasp pose is modified using backpropagation to increase the success rate of the grasp. As for the target object, we deal with some dirty tableware scattered on the table. We have developed a system that autonomously collects supervised data so that the robot can learn by itself whether it has succeeded in a grasp attempt. Finally, the robot can grasp an object which avoids dirty parts and find the suboptimal grasp pose.

While the musculoskeletal humanoid has various biomimetic benefits, its complex modeling is difficult, and many learning control methods have been developed. However, for the actual robot, the hysteresis of its joint angle tracking is still an obstacle, and realizing target posture quickly and accurately has been difficult. Therefore, we develop a feedback control method considering the hysteresis. To solve the problem in feedback controls caused by the closed-link structure of the musculoskeletal body, we update a neural network representing the relationship between the error of joint angles and the change in target muscle lengths online, and realize target joint angles accurately in a few trials. We compare the performance of several configurations with various network structures and loss definitions, and verify the effectiveness of this study on an actual musculoskeletal humanoid, Musashi.

In this study, seated walk, a movement of walking while sitting on a chair with casters, is realized on a musculoskeletal humanoid from human teaching. The body is balanced by using buttock-contact sensors implemented on the planar interskeletal structure of the human mimetic musculoskeletal robot. Also, we develop a constrained teaching method in which one-dimensional control command, its transition, and a transition condition are described for each state in advance, and a threshold value for each transition condition such as joint angles and foot contact sensor values is determined based on human teaching. Complex behaviors can be easily generated from simple inputs. In the musculoskeletal humanoid MusashiOLegs, forward, backward, and rotational movements of seated walk are realized.

The musculoskeletal humanoid is difficult to modelize due to the flexibility and redundancy of its body, whose state can change over time, and so balance control of its legs is challenging. There are some cases where ordinary PID controls may cause instability. In this study, to solve these problems, we propose a method of learning a correlation model among the joint angle, muscle tension, and muscle length of the ankle and the zero moment point to perform balance control. In addition, information on the changing body state is embedded in the model using parametric bias, and the model estimates and adapts to the current body state by learning this information online. This makes it possible to adapt to changes in upper body posture that are not directly taken into account in the model, since it is difficult to learn the complete dynamics of the whole body considering the amount of data and computation. The model can also adapt to changes in body state, such as the change in footwear and change in the joint origin due to recalibration. The effectiveness of this method is verified by a simulation and by using an actual musculoskeletal humanoid, Musashi.

There is a way to utilize humanoid robots to mimic human behavior by taking advantage of their human-like proportions. In general, motion capture is used; in this case, the posture of the body links can be taken. However, this method does not provide detailed information on the contact state, which is important for actions that involve contact with objects. In this study, we focused on the foot, which has not been paid much attention among the parts where contact and manipulation with objects are important, and developed a device to measure the contact pressure distribution at the outer edge of the sole. We proposed an index, SS-COP, which simply reflects the contact on the curved surface of the sole for this device and a robot foot with lateral force sensation and realized a behavior that imitates the foot condition of a humanoid robot by using this index.

When robots walk on uneven terrain, trajectory planning should take into account both the whole-body dy-namics and the ground geometry simultaneously. In uneven terrain environments, there are only a limited number of places where the robot is able to make stable contact with the ground without its feet wobbling or slipping because of the intricate round geometry. In such environments, the optional landing position and time to maintain the robot's balance and stable foot contact are not obvious and computationally expensive. In this study, we propose a robust walking system that integrates environment recognition using steppable regions and walking control for a humanoid robot to walk on uneven terrain. In this paper, a steppable region is defined as a two-dimensional convex hull that represents a region where a robot is capable of landing. We propose a method to compute the steppable region quickly by 2.SD projection of the environment points and spatial filtering. In this system, the walking controller integrates the steppable region with the Capture Region to modify the landing position from a two-dimensional geometric calculation. In addition, to cope with the environment recognition error, we have introduced a trajectory generation that allows the feet to penetrate the ground and hybrid control of position and torque. We verified the effectiveness of the proposed system through experiments in which a life-size humanoid robot walked on uneven terrain and recovered when pushed.

Humanoid robots are versatile platforms with the potential for multiple locomotion skills. However, this contact-switched system with only two contact feet is fragile to keep balance in many scenarios. Inspired by birds combining legs and wings, we propose the novel hybrid locomotion behavior for the humanoid robots with the aid of a thruster suit. To fully leverage their agility while guaranteeing efficient computation, we combine the neural controller based on reinforcement learning to handle the complexity of the highly non-linear system and the optimization-based controller to explicitly handle the constraint conditions of the safety-critical thruster module. Our learning framework is demonstrated on several thruster-aided humanoid platforms with hybrid walking and even dynamic locomotion skills. To our best knowledge, it is the first work that, 1. demonstrates agile hybrid whole-body locomotion skills on the thruster-aided humanoid robot; 2. achieves hybrid locomotion under the reinforcement learning settings.

Legged robots with high locomotive performance have been extensively studied, and various leg structures have been proposed. Especially, a leg structure that can achieve both continuous and high jumps is advantageous for moving around in a three-dimensional environment. In this study, we propose a parallel wire-driven leg structure, which has one DoF of linear motion and two DoFs of rotation and is controlled by six wires, as a structure that can achieve both continuous jumping and high jumping. The proposed structure can simultaneously achieve high controllability on each DoF, long acceleration distance and high power required for jumping. In order to verify the jumping performance of the parallel wire-driven leg structure, we have developed a parallel wire-driven monopedal robot, RAMIEL. RAMIEL is equipped with quasi-direct drive, high power wire winding mechanisms and a lightweight leg, and can achieve a maximum jumping height of 1.6 m and a maximum of seven continuous jumps.

For robots to move in the real world, they must first correctly understand the state of its own body and the tools that it holds. In this research, we propose DIJE, an algorithm to estimate the image Jacobian for every pixel. It is based on an optical flow calculation and a simplified Kalman Filter that can be efficiently run on the whole image in real time. It does not rely on markers nor knowledge of the robotic structure. We use the DIJE in a self-recognition process which can robustly distinguish between movement by the robot and by external entities, even when the motion overlaps. We also propose a visual servoing controller based on DIJE, which can learn to control the robot's body to conduct reaching movements or bimanual tool-tip control. The proposed algorithms were implemented on a physical musculoskeletal robot and its performance was verified. We believe that such global estimation of the visuomotor policy has the potential to be extended into a more general framework for manipulation.

Self-healing function is a promising approach for damage management of high-load robot applications such as legged robots. Although the function is getting major in soft robotics, its application to life-sized "stiff" robots is of relatively minor interest. Although the authors have devised several self-healing tensile modules for tendon-driven robots, the design guideline to satisfy the large load endurance and large stroke is still unclear. The paper focuses on the parametric design for unleaked liquid-assisted healing of low melting point alloy structure. The method was validated with a benchtop module test. Moreover, the module enabled tendon-driven monopod testbed to perform squat motion three times after the landing impact fracture and the self-healing sequence, which was never accomplished.

Precise aerial manipulation is important for multirotor robots. For multirotors equipped with arms, the root pose error due to the floating body affects the precision at the end effector. Fixed-root approaches, such as perching on surfaces using the rotor suction force, are useful to address this problem. Furthermore, it is difficult for arm-equipped multirotors to generate large wrenches at the end effector owing to joint torque limitations. For multilink aerial robots with rotors distributed to each link, the thrust of rotors can produce large torques. Therefore, such multirotor robots can generate comparatively large wrenches at the end effector. In this paper, we introduce a rotor-distributed multilink robot that can perch on surfaces. First, we designed a root footplate and arm module for a multilink aerial robot. During perching, the joint between these two links can be passive to prevent peeling. Second, we propose a quadratic programming (QP) based controller to calculate the desired thrust for perching motion, considering the static friction and zero moment point (ZMP) conditions on the footplate. Finally, we conducted root-body perching motion tests. The manipulations of the multilink aerial robot during perching become more accurate than those during flight because the root position adheres to the environment.

Quadrupedal robots are skillful at locomotion tasks while lacking manipulation skills, not to mention dexterous manipulation abilities. Inspired by the animal behavior and the duality between multi-legged locomotion and multi-fingered manipulation, we showcase a circus ball challenge on a quadrupedal robot, ANYmal. We employ a model-free reinforcement learning approach to train a deep policy that enables the robot to balance and manipulate a light-weight ball robustly using its limbs without any contact measurement sensor. The policy is trained in the simulation, in which we randomize many physical properties with additive noise and inject random disturbance force during manipulation, and achieves zero-shot deployment on the real robot without any adjustment. In the hardware experiments, dynamic performance is achieved with a maximum rotation speed of 15 °/s, and robust recovery is showcased under external poking. To our best knowledge, it is the first work that demonstrates the dexterous dynamic manipulation on a real quadrupedal robot.

The purpose of this paper is the robotic hanging manipulation of an object of various shapes that is not limited to a specific category. To achieve this, we propose a method that allows the estimator to learn many different shapes with hanging points without any manual annotation. A random shape generator using GAN solves the limitation of the number of 3D models and can handle objects of various shapes. In addition, hanging is repeated in the dynamics simulation, and hanging points are automatically generated. A large amount of training data is generated by rendering random-textured objects with hanging points in the random simulation environment. A deep neural network trained with these data was able to estimate hanging points of an unknown category object in the real world and achieved hanging manipulation by a robot.

We have implemented a force-based operational space controller on a physical musculoskeletal humanoid robot arm. The controller calculates muscle activations based on a biomimetic Hill-type muscle model. We propose a method to include the joint torque nullspace in the optimization process, which enables the robot to exploit the nullspace to gradually lower its overall muscle activation. We have verified in experiments that it can react compliantly to external disturbances while retaining its operational space task.

In this study, we propose a predictive model composed of a recurrent neural network including parametric bias and stochastic elements, and an environmentally adaptive robot control method including variance minimization using the model. Robots which have flexible bodies or whose states can only be partially observed are difficult to modelize, and their predictive models often have stochastic behaviors. In addition, the physical state of the robot and the surrounding environment change sequentially, and so the predictive model can change online. Therefore, in this study, we construct a learning-based stochastic predictive model implemented in a neural network embedded with such information from the experience of the robot, and develop a control method for the robot to avoid unstable motion with large variance while adapting to the current environment. This method is verified through a mobile robot in simulation and to the actual robot Fetch.

Musculoskeletal humanoids have various biomimetic advantages, and the redundant muscle arrangement allowing for variable stiffness control is one of the most important. In this study, we focus on one feature of the redundancy, which enables the humanoid to keep moving even if one of its muscles breaks, an advantage that has not been dealt with in many studies. In order to make the most of this advantage, the design of muscle arrangement is optimized by considering the maximization of minimum available torque that can be exerted when one muscle breaks. This method is applied to the elbow of a musculoskeletal humanoid Musashi with simulations, the design policy is extracted from the optimization results, and its effectiveness is confirmed with the actual robot.

We report the results of our study on whether taking a walk with a child-like robot that a person takes care of can generate interaction with the surrounding people whom the person has never met before. As the number of single-person households increases, it is expected that more people will live with not only robots that can be useful for people but also robots that people can take care of. Our study is important because we explored the possibilities of such a robot to create interaction between people in the community, which has been lost in recent years. In this paper, we designed the behavior of a robot that follows a person and learns about the scenery while walking together and implemented it using Pepper. Then, the first author walked around the university building with the robot and observed the initial reactions of the surrounding people. As we expected, the surrounding people interacted with the first author by talking to the robot as if it were a child and helping the robot. This paper contributes to taking a step forward a new research theme ‘taking a walk with a robot and a person’ by focusing on the important role of robots in the future relationship with a person: care-receiving from a person and communication mediation between a person and others.

We developed a method to enable a humanoid robot to carry stacked boxes. In order to transport objects efficiently, it is necessary to carry multiple objects at the same time, but in previous studies, humanoid robots have only been able to carry a single object. When a humanoid robot carries stacked boxes, the robot drops boxes when the positional relationship between un-grasped boxes changes. The causes for dropping the boxes can be divided into sudden changes attributed to robot making turns or losing balance, and the accumulation of small changes that occur because of the impact of landing while walking. We propose a method that prevents sudden changes in the stacked boxes by smoothing the hand trajectory and modifying the misalignment by tilting or shaking the entire stack. We verify the effectiveness of proposed method for enabling a humanoid robot to carry stacked boxes through experiments using a simulator and an actual robot.

In this paper, we aim to recognize function points of category-agnostic objects and perform object manipulation. To recognize function points of various shapes, it is necessary to train with a large amount of training data. Also, it is necessary to take into account not only visual information but also physics and interaction between objects. To solve these problems, we are working on the automatic generation of training data by detecting function points from a physical simulation. In the proposed system, we add simulation with target task operation and goal state, which allows a robot to acquire the target function point recognizer. We also use GAN to generate various random shapes and render them with random domains, and train Deep Neural Networks on these data. These enable the robot to recognize function points of unseen objects in the real world and realize manipulation.

Due to the recent lifestyle changes, instances of people eating alone have been increasing. We think robots can be good meal partners without having to risk disease transmission. Furthermore, people are able to eat with robots without worrying about mealtimes. In this study, we determine who are more likely to accept robots as eating partners and compare eating with a single robot to eating with multiple robots. The results revealed that people who have vast experience in interacting with robots and those who have relatively few opportunities to eat alone felt better about eating with robots, whereas those who have numerous opportunities to eat alone enjoyed eating with multiple robots.

In recent years, various types of sensors observe the real world. Especially, weather sensors are densely installed all over the world to observe current weather situations at various places. However, weather signals such as the temperature or humidity obtained by weather sensors are intuitively difficult for humans to understand. On the other hand, images captured by typical RGB cameras can tell weather situations at the captured places in a more comprehensible way for humans; however, cameras are only installed at limited places and are not necessarily open to public due to privacy issues. In order to solve this problem, the goal of our work is to generate images which can tell weather situations at arbitrary time and locations. This can be realized by using a conditional generative adversarial network architecture that takes an image and a condition to transform the image accordingly to the condition. Training such image generator requires a large number of image and condition pairs as the training data. Although weather signals can be easily collected from weather sensors, collecting their spatially and temporally synchronized outdoor images is not easy. Thus, we propose a semi-supervised method for training the image generator. A relatively small number of pairs of an outdoor image and weather signals is collected, each from different web services, by considering their semantic consistency. The collected pairs are used to train a predictor for predicting weather signals from a given outdoor image. Then, the image generator is trained by using a large number of pairs of an outdoor image and pseudo weather signals predicted by the predictor as the training data.

The flexible under-actuated musculoskeletal hand is superior in its adaptability and impact resistance. On the other hand, since the relationship between sensors and actuators cannot be uniquely determined, almost all its controls are based on feedforward controls. When grasping and using a tool, the contact state of the hand gradually changes due to the inertia of the tool or impact of action, and the initial contact state is hardly kept. In this study, we propose a system that trains the predictive network of sensor state transition using the actual robot sensor information, and keeps the initial contact state by a feedback control using the network. We conduct experiments of hammer hitting, vacuuming, and brooming, and verify the effectiveness of this study.

Carrying payloads in air is a major mission for multirotor aerial robot. However, the presence of payloads on multirotor aerial robot has a risk of degrading the performance of the flight controller. This concern becomes obvious especially when carrying objects not securely attached to the body or performing aerial manipulation. Therefore, controller with the ability to adapt itself to the effects of payloads on flight stability is needed. This paper proposes a novel nonlinear multiple-input and multiple-output (MIMO) model reference adaptive control (MRAC) system for attitude control of multirotor aerial robots which can dynamically compensate change in the position of center of gravity and inertia caused by payloads. Stability and robustness of the controller are experimentally confirmed in quadrotor and transformable multirotor, and experiments modeling practical applications are conducted for each aerial robot system, proving the utility of the controller.

Stable flight control under ceilings is difficult for multirotor Unmanned Aerial Vehicles (UAVs). The wake interaction between rotors and upper walls, called the "ceiling effect", causes an increase of rotor thrust. As a result of the thrust increase, multi-rotors are drawn upward abruptly and collide with ceilings. In previous work, several thrust models of the ceiling effect have been proposed for stable flight under ceilings, assuming that the airflow around rotors is in steady states. However, the airflow around rotors in vertical flight is not in steady states and each thrust model in previous work is skillfully determined based on large amounts of precise experimental data. In this paper, we introduce an aerodynamics-based thrust model and a stable control method under ceilings. This model is derived from the momentum theory and the relationship between vertical climbing/descending rates of rotors and an induced velocity. To confirm our proposed model, we collect thrust data at various vertical rates in flight. In addition, we use only onboard sensors to estimate selfstate for structural inspections. Consequently, we reveal that the proposed model is consistent with the experimental results. Based on an aerodynamic model, we need not collect large amounts of precise experimental data to realize stable flight. Furthermore, the vertical flight tests under ceilings demonstrate that our in-unsteady-state-model-based controller outperforms the conventional steady-state ones.

It is necessary to use not only foot but also hand, knee and other body parts to support body weight for locomotion in uneven terrain. Such multi-contact motion planning is an important research topic including lots of previous works; however, a problem of computational speed of planning is still remaining. In this paper, we propose a learning-based algorithm to speed up the planning. The algorithm reduces replanning of contact states by learning an evaluation function of key pose to reach goal. We investigated the learning performance by comparing three neural network configurations and two activation function. This research aims at achieving robust robotics system in unknown environments.

Regrasping is one of the most common and important manipulation skills used in our daily life. However, aerial regrasping has not been seriously investigated yet, since most of the aerial manipulator lacks dexterous manipulation abilities except for the basic pick-and-place. In this paper, we focus on pivoting a long box, which is one of the most classical problems among regrasping researches, using a transformable multilink aerial robot. First, we improve our previous controller by compensating for the external wrench. Second, we optimize the joints configuration of our transformable multilink drone for stable grasping form under the constraints of thrust force and joints effort. Third, we sequentially optimize the grasping force in the pivoting process. The optimization goal is to generate continous grasping force whilst maximizing the friction force in case of the downwash, which would influence the grasped object and is difficult to model. Fourth, we develop the impedance controller in joint space and admittance controller in task space. As far as we know, it is the first research to achieve extrinsic contact-aware regrasping task on aerial robots.

Humanoid robots are expected to play a big role at distress sites and disaster sites. There is a variety of multi-contact locomotion forms other than bipedal walking such as crawling through tightly, getting on the rubble by using its knees and elbows, or jumping in and rolling over the obstacles. If such multi-contact locomotion forms can be achieved, robots can reach environments that are currently unreachable, and be able to conduct tasks required at the environments. To achieve this, it is required for robots to bring various parts of its body into contact with the environment like a human. However, it is difficult for parts without 6-axis force sensors to achieve the target force while adapting to the environment against impact force. It is also difficult to measure contact wrenches without 6-axis force sensors. In this paper, by allowing the error of the contact state, we propose online system for realizing dynamic motion which impact force occurs on the parts of the whole body by contact to the environment. In the proposed system, we applied the current-based torque control for joints to make the whole body parts of the robot adapt to the environment, and we modified motion in real time to stabilize zmp by estimating contact wrenches at the contact positions where force sensors are not mounted. In addition, at the motion planning, we generated more feasible motions for a robot applying torque control by using evolutionary computation which advances the search with the behavior of torque control. We demonstrate that the proposed system is effective by showing experimental results of sitting posture locomotion using a JAXON robot in which impact force occur on the back of the thighs which have no force sensors.

In this work, we propose a system for humanoid robot fast motions. When a humanoid robot performs a motion such as a tennis forehand stroke motion, a whole-body fast motion in reaction to visual information is required. There are three problems to tackle. (1) Motion is desired to be quick. (2) Real-time visual processing considering visual noises is needed. (3) Real-time joint angle modification with balance keeping is needed. To solve the problem (1), we used an offline optimization system to enhance the motion speed. To solve the problem (2), we implement a ball trajectory prediction algorithm using the Extended Kalman Filter (EKF). To solve the trade-off between (1) and (3), we propose an offline optimization condition with an estimated balance margin. By using these methods, we achieved a non-step tennis forehand stroke motion with a humanoid robot by predicting a ball's trajectory with stereo cameras on the robot's head.

The musculoskeletal humanoid has various biomimetic benefits, and the redundant muscle arrangement is one of its most important characteristics. This redundancy can achieve fail-safe redundant actuation and variable stiffness control. However, there is a problem that the maximum joint angle velocity is limited by the slowest muscle among the redundant muscles. In this study, we propose two methods that can exceed the limited maximum joint angle velocity, and verify the effectiveness with actual robot experiments.

The musculoskeletal humanoid has various biomimetic benefits, and it is important that we can embed and evaluate human reflexes in the actual robot. Although stretch reflex has been implemented in lower limbs of musculoskeletal humanoids, we apply it to the upper limb to discover its useful applications. We consider the implementation of stretch reflex in the actual robot, its active/passive applications, and the change in behavior according to the difference of parameters.

The purpose of this study is to make life-sized humanoid robots acquire tool manipulation skills that require complicated force adjustment. The difficulty in acquisition of tool manipulation skills comes from the hardship in physical modeling. Recent research have revealed that deep reinforcement learning (DRL), a model-free approach, performs superior in such tasks. However, DRL in general has a drawback in sample efficiency, and this becomes critical in robot learning especially in life-sized humanoid robots. In this study, we propose an integrated system incorporating DRL method and active learning. Our method also leverages a variety of previous studies on life-sized humanoid robots to overcome the sample efficiency issue. We demonstrated the effectiveness of our proposed system through a hacksaw skill acquisition and a Japanese planer (Kanna) skill acquisition by a life-sized humanoid robot.

For mitigating joint impact torques, researchers have reduced joint stiffness by series elastic actuators, reflected inertia by low gear ratios, and friction torque from drive-trains. However, these impact mitigation methods may impair the control performance of contact forces or may increase motor and robot mass. This paper proposes a design method for achieving a balance between impact mitigation performance and force control fidelity. We introduce an inertia-to-square-torque ratio as a new index for integrating the parameters of torque generation (motor continuous torque limits, gear ratios, etc.) and the parameters of impact mitigation (joint stiffness, reflected inertia, etc.). In the process, we make a hypothesis that a motor mass is negatively correlated with the ratio. Based on the hypothesis, we calculate a joint breakdown region of impact torques, joint stiffnesses, and motor masses. Finally, we decide the drive-train specifications of JAXON3-P and demonstrate that the proposed method provides high impact mitigation and force control capabilities through several experiments including the jumping motion of 0.3 m COG height.

We consider the problem of acquiring mechanical knowledge through visual cues to help robots use objects in new situations. In this work, we propose a novel deep learning approach that allows a robot to acquire mechanical knowledge from 3D point clouds. This presents two main challenges. The first challenge is that a robot needs to infer novel objects' functions from its experience. Secondly, the robot should also need to know how to manipulate these novel objects. To solve these problems, we present a two-branch deep neural network. The first branch detects function parts from the point clouds while the second branch predicts offset poses. Fusing the results from these two branches, our approach can not only detect what functions the novel objects may have but also generate key object states which can be used to guide a robot to manipulate these objects. We show that even though most of the training samples are synthetic data, our model still learns useful features and outputs proper results. Finally, we evaluate our approach on a real robot to run a series of tasks. The experimental results show that our approach has the capability to transfer mechanical knowledge in new situations.

Juggling manipulation is one of difficult manipulation to acquire since some of such manipulation is unstable and also its physical model is unknown due to the complex non-prehensile manipulation. To acquire these unstable unknown-dynamics juggling manipulation, we propose a method for designing the predictive model of manipulation with a deep neural network, and a real-time optimal control law with some robustness and adaptability using backpropagation of the network. In this study, we apply this method to diabolo orientation stabilization, which is one of unstable unknown-dynamics juggling manipulation. We verify the effectiveness of the proposed method by comparing with basic controllers such as P Controller or PID Controller, and also check the adaptability of the proposed controller by some experiments with a real life-sized humanoid robot.

This paper describes basic implementation of an embedded controller board based on a hybrid architecture equipped with an Intel FPGA SoC and an NVIDIA GPU SoC. Embedded distributed network involving motor-drivers or other embedded boards is constructed with low-latency optical transmission link. The central controller for high-level motion planning is connected via Gigabit Ethernet. The controller board with the hybrid architecture provides lower-latency feedback control performance. Computing performance of the FPGA SoC, the GPU SoC, and the central controller is evaluated by computation time of matrix multiplication. Then, the total feedback latency is estimated to show the performance of the hybrid architecture.

In this research, we have taken a biomimetic approach to the control of musculoskeletal humanoids. A controller was designed based on the motor directional tuning phenomenon seen in the motor cortex of primates. Despite the simple implementation of the control scheme, complex coordinated movements such as reaching for target objects with its upper body was achieved, and is demonstrated in the accompanying video. The controller does not require an internal model, and instead constantly observes its body in relation to the external world to update motor commands. We claim that such an embodied approach to the control of musculoskeletal robots will be able to effectively take advantage of their complex bodies to achieve motion.

Eating with others enhances our dining experience, improves socialization, and has some health benefits. Although many people do not want to eat alone, there is an increase in the number of people who eat alone in Japan due to difficulty in matching mealtimes and places with others.In this paper, we develop a mixed reality (MR) system for coeating with a robot. In this system, a robot and a MR headset are connected enabling users to observe a robot putting food image into its mouth, as if eating. We conducted an experiment to evaluate the developed system with users that are at least 13 years old. Experimental results show that the users enjoyed their meal and felt more delicious when the robot ate with them than when the robot only talked without eating. Furthermore, they eat more when a robot eats, suggesting that a robot could influence people's eating behavior.

Recent progress of deep learning improved the capability of a robot to find a proper grasp of a novel object for different grasp modalities (e.g., pinch and suction). While these previous studies consider multiple modalities separately, several studies develop multi-modal grippers that can achieve simultaneous pinch and suction grasp (multi-modal grasp fusion) for more capable and stable object manipulation. However, the previous studies with these grippers restrict the situations: simple object geometry and uncluttered environments. To overcome these difficulties, we propose a system that consists of: 1) object-class-agnostic grasp modality detection; 2) object-class-agnostic instance segmentation; and 3) grasp template matching for different modalities. The key idea of our work is the introduction of instance segmentation to fuse multiple modalities regarding each instance eluding a grasp of multiple objects at once. In the experiments, we evaluated the proposed system on the real-world picking task in clutter. The experimental results show that the effectiveness of modality detection, instance segmentation, and the integrated system as a whole.

External wrench estimation is very helpful for aerial exploration and manipulation tasks. During the exploration, there might be unseen obstacles to cause dangerous collisions. The estimation of the external force and torque is also beneficial in aerial manipulation tasks. In this paper, we present a framework of estimating the external wrench for the aerial multilink robot based on the onboard inertial measurement unit (IMU) sensors, joints state and robot dynamic models. Compared to the conventional multirotor robot, the center of mass (CoM) is always changing when the robot transforms. The sensor could not be attached to CoM to observe the acceleration data. Consequently, we present a novel method by applying a distributed IMU system to estimate the CoM linear and angular accelerations for the external wrench estimation. With the help of the robot model, the position of the contact point could be estimated, which is useful in exploring tasks to safely interact with the physical world. We design the contact-aided navigation strategy and computationally efficient motion primitives library to help our robot react to the unexpected collision. We experimentally validate our framework with a two-dimensional multilink aerial robot to show the results of external wrench estimator and its further applications2.2Experiment video: https://youtu.be/R-WDReLnWWI

We present joint learning of instance and semantic segmentation for visible and occluded region masks. Sharing the feature extractor with instance occlusion segmentation, we introduce semantic occlusion segmentation into the instance segmentation model. This joint learning fuses the instance-and image-level reasoning of the mask prediction on the different segmentation tasks, which was missing in the previous work of learning instance segmentation only (instance-only). In the experiments, we evaluated the proposed joint learning comparing the instance-only learning on the test dataset. We also applied the joint learning model to 2 different types of robotic pick-and-place tasks (random and target picking) and evaluated its effectiveness to achieve real-world robotic tasks.

We present a novel transformable multilinked aerial robot which consists of link modules with 1 DoF thrust vectoring mechanism. Commonly used UAV is underactuated due to its simplicity and high flight duration, but can not control the position and orientation independently. To overcome this problem, fully actuated multirotor aerial robots have been developed. In our previous work we developed fully actuated multilinked aerial robot which can transform in the air. However, the transformation range was limited because of a singularity problem. In this paper we propose a new design of link module with a tilted rotor and 1 DoF thrust vectoring joint which enables to avoid singularity forms and keep the flight stable during transformation. We describe modeling and control for the fully actuated multilinked multirotor. Then we propose a transformation planning method utilizing the 1 DoF thrust vectoring angle with consideration of guaranteed minimum force/torque. Finally we perform an aerial transformation experiment with a real platform to demonstrate the feasibility of our proposed design and methods.

As robots start to execute complex manipulation tasks, they are expected to improve their skill set over time as humans do. A prominent approach to accomplish this is having robots to keep models of their actions based on their experiences in order to improve their action executions in the future. In this paper, we present such a methodology where robots start to execute some actions with random parameters and record their generic execution logs with semantic annotations in a symbolic knowledge base for robots. Using the data inside logs, multivariate Gaussian mixture models are fitted to the high-level action parameters for later executions. These affordance models are being updated whenever a new execution is carried on. In essence, robots can use these continuously-updated probabilistic model for improving their actions To prove the applicability we demonstrate opening-a-fridge-door experiments with a PR2 robot.

In this paper, we propose an approach for online whole-body control of position-controlled humanoid robot with 3D multi-contact to cope with contact wrench constraints and joint overload. In our method, robots are controlled under contact wrench constraints with three features: 1) internal wrench control to reduce joint load and prolong the time in which the high-load postures can be maintained 2) feasible utilization of self-collision to reduce joint load by turning off joint servo gains 3) handling degenerated degree of freedom by solving a quadratic optimization problem integrating wrench distribution and inverse kinematics in which internal wrench is controlled only in controllable directions.With our methods, HRP2-JSKNTS could pick up an object under a desk with squatting with the back of the upper leg on the back of the lower leg without sliding at the right arm. We also evaluated the effectiveness of our control to reduce joint load with another experiment.

We proposed the soft flexible wire-driven finger mechanism with the soft skin and the multi-joint skeletal structure using two coil springs. The multi-joint skeletal structure is mainly composed of two coil springs, multiple skeletal members and a fiber wire, and the soft skin is formed outside of them. The soft skin and the multi-joint skeletal structure make it possible to distribute the contact pressure, which is necessary for not only touching a living body such as a human or an agricultural crop, but also for touching an artificial object such as pastry or an industrial product without damaging it or its package. In this paper, we describe the design of the soft flexible wire-driven finger mechanism and development of the three-fingered hand using the finger mechanism we proposed.

To develop Musashi as a musculoskeletal humanoid platform to investigate learning control systems, we aimed for a body with flexible musculoskeletal structure, redundant sensors, and easily reconfigurable structure. For this purpose, we develop joint modules that can directly measure joint angles, muscle modules that can realize various muscle routes, and nonlinear elastic units with soft structures, etc. Next, we develop MusashiLarm, a musculoskeletal platform composed of only joint modules, muscle modules, generic bone frames, muscle wire units, and a few attachments. Finally, we develop Musashi, a musculoskeletal humanoid platform which extends MusashiLarm to the whole body design, and conduct several basic experiments and learning control experiments to verify the effectiveness of its concept.

The musculoskeletal humanoid has many benefits that human beings have, but the modeling of its complex flexible body is difficult. Although we have developed an online acquisition method of the nonlinear relationship between joints and muscles, we could not completely match the actual robot and its self-body image. When realizing a certain task, the direct relationship between the control input and task state needs to be learned. So, we construct a neural network representing the time-series relationship between the control input and task state, and realize the intended task state by applying the network to a real-time control. In this research, we conduct accelerator pedal control experiments as one application, and verify the effectiveness of this study.

In this paper, we propose walking balance control based on Caputurability. The proposed method consists of five strategies: (i) moving Zero Moment Point (ZMP) in the support polygon (ii) landing position modification (iii) landing timing modification (iv) angular momentum control (v) falling detection and fall control. Walking pattern generation calculates the ZMP so that the Capture Point (CP) reaches the position of the supporting foot at the end of the double support phase. Owing to the asymmetry of the reachable landing region, landing timing modification is different in the sagittal and lateral planes, and the step time is extended in the lateral plane depending on the direction of disturbances. The torque around the center of gravity to avoid falling is realized through whole-body inverse kinematics with constraints on the angular momentum. In addition, we propose falling detection considering the reachable landing region. We verified the effectiveness of the proposed method through experiments in which the biped robot was disturbed by pushing during tether-free walking. The robot could prevent breakdown by detecting possible falling and performed knee bending motions to suppress damage.

We present a torque controller for humanoid robot's arm and a method to execute heavy-load tasks under that controller. Torque control of arms can reduce the joint load when an impulsive force is applied to the robot's hand. This feature is important for robots that will work among many humans or obstacles because accidental collisions may happen in such situations. We also developed a static force filter for force sensors at the end-effectors. This filter is utilized for the compensation for the static reaction force during heavy-load tasks. A life-sized humanoid robot JAXON digs soil with a shovel and carries soil with a wheelbarrow using our proposed controller.

In order to achieve tasks in the real world environment, humanoid robots have motors and reduction drives optimized in relation to weight and size for providing the necessary torque and angle speed. Therefore, having torque or angle speed outside of the predicted range will usually cause the task to fail. In this research, we propose a joint module with a two-stage transmission mechanism during driving and a joint locking mechanism during non-driving so that the appropriate torque and joint speed can be attained during the task. By applying the joint module to a tricycle type robot and switching the driving state during the task execution, we were able to both reduce the motor load when lifting heavy objects at driving time and keep high rigidity of the joint at non-driving time.

Self-healing robotics has been of considerable interest. We believe the function will have a major role in legged robots as a typical high-load application of robotics. Some pioneering works have been ongoing on self-healing soft robots. However, the development of large load self-healing component and its system integration with a life-sized legged robot is a challenging task. This study is to try the problem by a constructing self-healing component oriented for facilitated investigation. Proposed part enhances visibility and manufacturing by specializing tension transmission system. The developed module was evaluated by several experiments. First, healing visualization experiment was conducted to evaluate healing progress. In addition, the module's strength was tested using a motor-driven tendon module previously developed in our laboratory. Results of these experiments suggested that the stirring process have a major role in performing self-healing behaviour. Finally, we conducted a preliminary experiment on a tendon-driven legged robot. The experiment demonstrated that the module functioned in a real robot once.

Humanoid robots are expected to conduct tasks on behalf of humans in places such as a disaster scattered environment. Although humanoid robots have potentials to walk on uneven ground unlike wheeled robots, it is difficult to reach a given destination without falling down based on only visual information. In this paper, to reach the destination safely, we propose the autonomous safe locomotion system applying vision and sole reaction force to the footstep planning. Considering force information in addition to visual information, the robot can plan a path avoiding unstable footholds. The planned path is safer than a path which is planned based on only visual information. In our system, the robot checks if the foothold is safe or not by the foothold ascertainment motion. In addition to that, the robot saves the results of the motion to the database with the foothold label given by the visual classifier. To judge foothold safety, stiffness of the foothold is estimated from the reaction force and stepping amount. We propose the system considering these requirements for safe locomotion for bipedal robots and show experimental results using a real bipedal robot CHIDORI.

Transformable aerial robots are favorable in aerial manipulation tasks for their flexible ability to change configuration during the flight. By assuming robot keeping in the mild motion, the previous researches sacrifice aerial agility to simplify the complex non-linear system into a single rigid body with a linear controller. In this paper, we present a framework towards agile swing motion for the transformable multi-links aerial robot. We introduce a computational-efficient non-linear model predictive controller and joints motion primitive frame-work to achieve agile transforming motions and validate with a novel robot named HYRURS-X. Finally, we implement our framework under a table tennis task to validate the online and agile performance.Supplementary MaterialThis paper is accompanied by a experiment video: http://www.jsk.t.u-tokyo.ac.jp/%7eshifan/paper/iros19/video.mp4.

When a humanoid robot manipulates a heavy object, balance and heavy loads become problems. To solve these problems, we provide a method to generate a key pose sequence of the robot and the object which balance constraints and joint torque limits are kept. As we consider the configurations of both the robot and the object, the key poses of them are optimized in a view of kinematics and statics. Moreover, we generate a smooth key pose sequence by adding an objective function which makes adjacent poses closer. By using the proposed method, we make a humanoid robot RHP4B place a heavy suitcase on a step.

To realize a robust automobile driving behavior of musculoskeletal tendon-driven humanoids, we developed a six-axis force measurement module with a core-shell structure. This sensor enables space saving, high load capacity and wholebody sensing at the same time. By developing a foot unit incorporating a core-shell structural force sensor on its toe, we realized behaviors of depressing a pedal and recovering from foot slipping during the depressing with a lifesized musculoskeletal humanoid ”Musashi”.

An integrated humanoid robot system, including from low-level hardware to high-level intelligence software and user interfaces, is required to build humanoid robot system meeting the expectation that robots work in the real environment such as disaster response. For creating such an integrated system, it is important that it has sustainable development potential, partially re-usability, and transparency to any type of robot. When creating a new robot, the conventional a system is desired to use just by changing a robot hardware. In order to realize such system, it is necessary to ensure that a software system is transparent to a robot hardware. On the other hand, sustainable development and partially reusability contribute robustness of the system and quickly building a complex system. In this paper, we describe the methodology to create an integrated humanoid robot system through actual humanoid robot system we have developed. Our system achieved to have sustainable development, partially re-usability, and transparency to any type of robot.

In this paper, we present the achievement of aerial grasping by shape adaptive transformation to the object shape, using a novel transformable aerial robot called HALO: Horizontal Plane Transformable Aerial Robot with Closed-loop Multilinks Structure. Aerial manipulation is an active research area and using multiple aerial robots is an effective solution for the large size object. However the cooperation is considered that there are some difficulties such as the synchronized flight control and collision with each other. Then, we focus on the transformable aerial robot with two-dimensional multilinks proposed in our previous works, which can transform to the suitable form for the target object and grasp it. However the transformable aerial robot with the serial-link structure could not achieve stable flight in terms of horizontal position and yaw control due to the low rigidity and large inertia in the case of more than 4 links. Thus, first we construct a novel type of multilinks with closed-loop structure to avoid the deformation and a new link module with a tilted propeller for fully-actuated control. Second, we describe transformation method with closed-loop multilinks. Third, we present the optimization planning method for the multilinks form to be adaptive to the two-dimensional shape of the target object. Finally, we present experimental results to demonstrate the feasibility of closed-loop aerial transformation and aerial grasping for the large size object.

To enable robots to perform human-level tasks flexibly in varying conditions, we need a mechanism that allows them to exchange knowledge between themselves for crowd-sourcing the knowledge gap problem. One approach to achieve this is to equip a cloud application with a range of encyclopedic knowledge (i.e. ontologies) and execution logs of different robots performing the same tasks in different environments. In this paper, we show how knowledge exchange between robots can be done using OPENEASE as the cloud application. We equipped OPENEASE with ontologies about the kitchen domain, execution logs of three robots operating in two different kitchens, and semantic descriptions of both environments. By addressing two different use cases, we show that two PR2 robots and one Fetch robot can successfully adapt each other's plan parameters and sub symbolic data to the experiments that they are conducting.

In this paper, we propose novel methods suitable for online real time whole body master-slave control with real life-sized humanoid robot. We conducted some dynamic whole body master-slave experiment with life-sized humanoid robot, and we achieved speedier and flexible master-slave operation compared to conventional study. Conventionally, master-slave operations with humanoid robots were available with only the upper body of the humanoid robot, and the COM movement was limited to be static. In our previous study, we introduced LIP model based restrictions to ensure the balance stability. In this study, we extend the safety restrictions by introducing foot landing delay prediction and trajectory smoothing method suitable for real robot. We conducted master-slave tennis swing experiment and high kick motion experiment with life-sized humanoid robot “JAXON”, and we evaluated the effectiveness of our proposed methods and system.

For robots to autonomously achieve manipulation tasks in various scenes, advanced operational skills such as tool use, learning from demonstration, and multi-robot/human-robot cooperation are necessary. In this research, we devise a method for robots to realize such operational skills in a unified manner by evaluating physical consistency (referred to as “physics reasoning”) based on the formulation of the manipulation statics constraints. First, we propose manipulation planning and estimation methods in which the operational feasibility and properties' likelihood are derived by physics reasoning. In addition, we propose a framework to manipulate an object with unknown physical properties by executing planning and estimation both sequentially and in parallel. We demonstrate the effectiveness of the proposed methods by performing experiments in which real humanoid robots achieve various manipulation tasks with advanced operational skills.

In this paper, we introduce a novel robotic prosthetic knee as shown in Fig. 1 (named as SuKnee) with variable transmission mechanism that could vary transmission ratio while knee angle varies during ambulation activities. A slider crank mechanism is utilized to transform linear motion of series elastic actuator to rotary motion of knee joint. And it contributes to variable transmission ratio with knee angle, which help obtain desired speed variation and torque output in different activities in one mechanism. This feature could uniquely give the SuKnee both: the torque necessary to assist with standing up from a chair and the speed necessary to swing the leg forward during walking. The knee has an active mode, where it operates with batteries and is capable of providing external power, and a passive mode, behaving like a passive prosthesis. Preliminary tests have been performed by a transfemoral amputee and SuKnee could provide user with power to assist walking on level ground and standing up from a chair. And a passive mode test shows it could work like passive prosthesis after battery exhaustion.

In this paper, we propose methods for walking on a steep slope using a rope by a humanoid robot. There are two difficulties for walking on a steep slope without a rope. First, range of motion of ankle joints get limited. Second, feet of a robot slip on a steep slope. For these problems, using a rope is effective solution because the robot can receive enough friction force from the slope and walk on a steep slope by pulling a rope with proper tension. In addition, the robot pulling a rope on a slope can relax limitations of ankle joints. Therefore, we propose methods to determine tension of a grasped rope by solving a linear least-square problem considering deformability of a rope. With these methods, a life-size humanoid robot HRP-2 could walk on a steep slope which angle is 40 degree.

For a robot to manipulate an object, it has to understand the functions and the actions that can be subjected to the object. This set of information is known as affordance of the object. Affordances are generally defined by the geometrical structures and physical properties of the objects. In this paper, we present an affordance detection network (ADNet) for detecting object affordances using multimodal input i.e., RGB-D data. The method is based on the state-of-the-art fully convolutional network with two encoding streams and one decoding stream. In the presented formulation, the network learns powerful discriminative features independently from the RGB and depth images, which enables it to abstract rich photometrical and geometrical properties of the objects. The multimodal encoding is combined at multiple stages of the network using the late-fusion strategy and used is for predicting the potential affordances of the objects.

To enable robots to work on real home environments, they have to not only consider common knowledge in the global society, but also be aware of existing rules there. Since such “local rules” are not describable beforehand, robot agents must acquire them through their lives after deployment. To achieve this, we developed a framework that a) lets robots record long-term episodic memories in their deployed environments, b) autonomously builds probabilistic object localization map as structurization of logged data and c) make adapted task plans based on the map. We equipped our framework on PR2 and Fetch robots operating and recording episodic memory for 41 days with semantic common knowledge of the environment. We also conducted demonstrations in which a PR2 robot tidied up a room, showing that the robot agent can successfully plan and execute local-rule-aware home assistive tasks by using our proposed framework.

Robotic picking of folded objects such as books is required for picking various objects. As a folded object is easily unfolded, it is difficult to carry it stably and place it in a desired pose due to its dangling part. For overcoming this difficulty, we propose a trial-and-error picking system using our Suction Pinching Hand, which can push the dangling part up with pinch grasp until the object lifted with suction grasp is folded. That system utilizes proximity sensors on the hand to predict whether folding will succeed with a current hand pose and decide whether to retry with another pose. Also, proximity sensors, flex sensors and an air pressure sensor are used to deal with uncertainty of the image recognition, the hand hardware and suction grasp. We evaluate our proposed system with experiments of picking and placing folded objects. It is confirmed that our proposed system realizes picking with the ability of our Suction Pinching Hand to carry folded objects stably and place them in desired poses. It is also proved that our proposed system is robust against the uncertainty.

Tendon-driven musculoskeletal humanoids have many benefits in terms of the flexible spine, multiple degrees of freedom, and variable stiffness. At the same time, because of its body complexity, there are problems in controllability. First, due to the large difference between the actual robot and its geometric model, it cannot move as intended and large internal muscle tension may emerge. Second, movements which do not appear as changes in muscle lengths may emerge, because of the muscle route changes caused by softness of body tissue. To solve these problems, we construct two models: ideal joint-muscle model and muscle-route change model, using a neural network. We initialize these models by a man-made geometric model and update them online using the sensor information of the actual robot. We validate that the tendon-driven musculoskeletal humanoid Kengoro is able to obtain a correct self-body image through several experiments.

The sequential online tuning for controller gains is required for the continuous action of the riding into parallel two-wheeled scooter and the speed governing after riding by humanoid robot. The implemented controllers are different between the riding and the speed governing, and these tuning strategies are also different. In particular, the riding requires the immediate tuning in the short riding phase and the speed governing requires the accurate tuning to regulate the speed of humanoid robot. To the above requirements, this paper proposes the Sequential Online Learning Control (SOLC)method composed of the cascade connection of SGD-based open-loop Learning Control (SLC)and Mini-batch-based closed-loop Learning Control (MLC). SLC contributes the damping gain online tuning for the foot torque control during execution of riding, and MLC contributes the PID gains online tuning for the speed governing control. Finally, we show the validity of SOLC through the sequential experiment of riding and speed governing for parallel two-wheeled scooter by life-sized humanoid robot HRP2-JSK.

Recently, self-supervised approach is common for robot grasping. Although this approach improves success rate, it requires a long time to execute a number of grasp trials, and single-arm grasping is only considered. However, robots can grasp more various objects with two arms, and dual-arm robots such as humanoid robots are expected to execute dual-arm manipulation and overcome the single-arm limitation. In this paper, we introduce dual-arm grasping as another possible strategy and propose a multi-stage learning method for selective dual-arm grasping using Convolutional Neural Networks (CNN)for grasping point prediction and semantic segmentation. In the first stage, the network learns grasping points with the automatic annotation. Although a robot learns both single-arm and dual-arm grasping efficiently with the annotation, the robot may not be able to grasp it because the annotation algorithm is designed by human. Therefore, for the second stage, the robot samples various grasping points with both grasping strategies and learns how to grasp in the real world. In this stage, the robot obtains new possible grasping points and prunes unsuccessful ones for both grasping strategies through the robot experience. In the experiments in the real world, the adapted network achieved high success rate 76.7% in 90 trials. Since the network trained with no adaptation stage resulted in lower success rate 56.7%, this result also shows the network was refined with less than 250 times of grasp sampling. As an application of our method, we demonstrated that our system worked well in warehouse picking task.

Human hands can not only grasp objects of various shape and size and manipulate them in hands but also exert such a large gripping force that they can support the body in the situations such as dangling a bar and climbing a ladder. On the other hand, it is difficult for most robot hands to manage both. Therefore in this paper we developed the hand which can grasp various objects and exert large gripping force. To develop such hand, we focused on the thumb CM joint with wide range of motion and the MP joints of four fingers with the DOF of abduction and adduction. Based on the hand with large gripping force and flexibility using machined spring, we applied above mentioned joint mechanism to the hand. The thumb CM joint has wide range of motion because of the combination of three machined springs and MP joints of four fingers have variable rigidity mechanism instead of driving each joint independently in order to move joint in limited space and by limited actuators. Using the developed hand, we achieved the grasping of various objects, supporting a large load and several motions with an arm.

Comparing to IK (Inverse Kinematics) based motion control, joint-space motion control is more advantageous in terms of not being restricted by kinematics singularity problem. In this paper, we start with SIMBICON (Simple Biped Locomotion Control) based controller, a joint-space motion control method, extend it for enhancing walking's robustness and versatility. We propose a motion optimization method considering walking robustness, desired walking velocity and energy efficient minimization for walking motion generation. This method enables us to achieve human-like walking motion, which has stretched-knee posture and robust to large push disturbances. We also apply our proposed method to a life-sized biped robot and validate its effectiveness with push recovery and walking on unknown debris experiments.

In this paper, we develop a bipedal robot control system that has an ability to perform instantaneous high power and flexibility to absorb an impulsive disturbance. We utilize a sensor-less whole body torque control method executed in a high responsive realtime distributed system. This system also includes a robust online walking controller that can avoid fall over caused by a strong collision with the robot's legs. We evaluated the proposed control system by hitting a rubber ball or adding a leg sweep disturbance and verified the functionality of the absorbing motion and the balance restoring motion.

Currently, most of commercially available ankle foot prosthesis are passive, which don't exhibit appropriate biomechanics during walking and could not adapt to dynamic property of able-bodied walking. In this paper, we present a novel robotic ankle foot prosthesis with variable transmission series elastic actuator (SEA). Slider crank mechanism is applied to transform linear motion of series elastic actuator to rotary motion of ankle foot joint. And this could contribute to variable transmission ratio while ankle angle varies. Because of variable transmission ratio, ankle joint torque is increasing while ankle angle is flexed from plantar flexion to dorsiflexion, whose feature has similar increase trend with human's ankle joint torque-angle relationship, and exhibits an appropriate characteristic for developing robotic ankle foot prosthesis. Larger torque could be obtained in powered plantar flexion, and this indicates that variable transmission mechanism would help reduce required motor torque compared with traditional mechanism. Energy stored in springs of series elastic actuator contribute a torque to powered plantar flexion. Preliminary experiments with a transtibial amputee and a transferomal amputee have been performed to test the prototype.

We present a robotic system for picking a target from a pile of objects that is capable of finding and grasping the target object by removing obstacles in the appropriate order. The fundamental idea is to segment instances with both visible and occluded masks, which we call `instance occlusion segmentation'. To achieve this, we extend an existing instance segmentation model with a novel `relook' architecture, in which the model explicitly learns the inter-instance relationship. Also, by using image synthesis, we make the system capable of handling new objects without human annotations. The experimental results show the effectiveness of the relook architecture when compared with a conventional model and of the image synthesis when compared to a human-annotated dataset. We also demonstrate the capability of our system to achieve picking a target in a cluttered environment with a real robot.

Robots need to adapt themselves to various surroundings in order to achieve robust object search and grasp in unknown environments. For this adaptation, robot motions should be implemented as combination of primitive motions which are based on sensor reaction. Among various sensing methods, non contact sensing is required as a means of preventing operation failures such as pushing objects. Especially, proximity sensors have been proved effective in avoiding occlusion problems. In this paper, we first develop a gripper on which proximity sensors are mounted all around, and then calculate distance between the gripper and objects using proposed calibration method. This enables robots to recognize detailed shapes of objects surrounding the gripper. We also propose primitive motions for object search and grasp, and describe the contents of each motion. The motions are based on sensor information obtained from the gripper. We verify the effectiveness of our system through an experiment in which a real robot performs complex tasks by combination of the primitive motions.

In this paper, we introduce the achievement of the flight motion to pass through small opening by the multilinked and transformable aerial robot. Previous works about such motion are based on under-actuated multirotors, indicating that aggressive maneuvering is necessary condition. This involves two crucial problems: i) enough free space for deceleration is necessary, otherwise the robot would collide with unknown obstacle after exiting opening; ii) the multirotor can not traverse the openings that are smaller than the robot body. The proposed transformable aerial robot in our work can solve these problems, since the multilinked model can not only guarantee the near-hover condition during the whole motion sequence, but also slowly traverse relative small openings by changing its form like a snake. We first propose an improved dynamics derivation and flight control method for this multilinked aerial robot based on our previous work. Then, we present the path planning method which takes the flight stability in the near-hover condition into account. Finally we demonstrate the experimental results of the motion to pass through a horizontal and small opening which also involves the borders (the floor and the ceiling).

To promote psychosocial development in children, a number of efforts are being made to improve child-child interactions. In this paper, we determine whether walking with a robot has a positive effect on child-child interactions. We developed a walking-together system consisting of two methods of walking: (1) walking hand in hand and (2) following a person by tracking the face. After applying the system to Pepper, a humanoid robot, we conducted the field experiment in a science museum in two phases. The first phase was in an open laboratory, setting in which people (including children) freely interacted with the robot. The second phase was the experimental setting in which 2-3 children simultaneously interacted with the robot. We observed the kinds of child-child interactions that occurred in the two phases of the experiment. We also determined what would be important to generate longer child-child interactions by walking together.

lack of social relationships has become a significant issue in modern society. A robot is expected to enhance communication between people by introducing each person using knowledge of the person which the robot has obtained through interactions on their daily life. Therefore, we need to clarify what kind of behavior is required when a robot interacts with a group of people in such a situation. The goal of this paper is to reveal the kind of communication that occurs between a group of people new to each other if a robot introduces each person. We conducted an experiment in which two robots communicated with 14 people (7 children and 7 adults) from 5 families in the following situation. First, the robots asked each person a simple question about their person or daily life. Then, all the people and the two robots gathered together. The two robots started chatting, including an introduction of each person. Based on the video coding and questionnaires, we found that the robots generated an opportunity for people new to each other to learn about each other. Furthermore, we discuss the design requirements of the robot behavior to enhance communication between people new to each other through the introduction of each person by a robot.

This paper presents an overview of the inaugural Amazon Picking Challenge along with a summary of a survey conducted among the 26 participating teams. The challenge goal was to design an autonomous robot to pick items from a warehouse shelf. This task is currently performed by human workers, and there is hope that robots can someday help increase efficiency and throughput while lowering cost. We report on a 28-question survey posed to the teams to learn about each team's background, mechanism design, perception apparatus, planning, and control approach. We identify trends in this data, correlate it with each team's success in the competition, and discuss observations and lessons learned based on survey results and the authors' personal experiences during the challenge.

Whole-body manipulation is necessary for a humanoid robot to achieve tasks such as carrying large objects. One difficulty for achieving a whole-body manipulation is that the robot needs to select the appropriate operation from a list of candidates, such as lifting, pushing, and tilting. The appropriate operation depends upon the target object's physical properties, including its mass, center of mass, and friction coefficient, which are difficult to measure directly. In order to select the appropriate manipulation motion online, we propose a method of estimating the object's physical properties and evaluating the feasibility of the object operation. We calculate the likelihood of the object's physical properties from sensor information during manipulation and update these properties' probabifity distribution periodically based on Bayesian methods. The operational feasibility probability is evaluated by physics-based stability determination, allowing the robot to perform manipulation tasks by selecting the appropriate operation. We show the effectiveness of the proposed method by an experiment in which a life-sized humanoid robot carries a large object.

This paper shows a method for achieving multi-contact motion for a humanoid robot on a movable object, such as climbing of a stepladder. Recent research has developed methods for achieving multi-contact motion that considers various constraints, such as joint limits, torques, balance constraints, reachability, and collision avoidance. In addition to these constraints, Motion On a Movable Object (MOMO) has the following features: it has to consider an object's balance during the changing of contact points; and it has to handle scenarios where the mass properties of an object are unknown. In this paper, in order to achieve a humanoid robot having MOMO, we propose balance constraints that consider the constraints imposed by an object as well as an online estimation of object's constraints. First, we use object-environment constraints as the robot's constraints, and then we show a method for estimating them based on information provided by the robot's sensors. Next, we show a method for applying the balance constraints to a humanoid motion planner and for executing planned motion with real-time sensor feedback controller. Finally, we evaluate our proposed method through experiments in which a life-sized humanoid robot climbs stepladders that have unknown mass properties.

In this paper, we introduce the achievement of the aerial manipulation by using the whole body of a transformable aerial robot, instead of attaching an additional manipulator. The aerial robot in our work is composed by two-dimensional multilinks which enable a stable aerial transformation and can be employed as an entire gripper. We propose a planning method to find the optimized grasping form for the multilinks while they are on the air, which is based on the original planar enveloping algorithm, along with the optimization of the internal force and joint torque for the force-closure. We then propose the aerial approach and grasp motion strategy, which is devoted to the determination of the form and position of the aerial robot to approach and grasp effectively the object from the air. Finally we present the experimental results of the aerial manipulation which involves grasping, carrying and dropping different types of object. These results validate the performance of aerial grasping based on our proposed whole-body grasp planning and motion control method.

In this paper, we show the achievement of a transformable aerial robot with internal communication system for multiple objects transportation. As it is not easy to make the flight endurance of an aerial robot longer, we study the problem to transport multiple objects at the same time to improve the efficiency of transportation. However, for conventional aerial robots, multiple objects transportation is difficult because the CoG position changes when the number of grasped objects changes, resulting in the instability of the flight. Therefore, to solve this problem, we focus on the multirotor with two-dimensional multilinks proposed in our previous work, which possesses the ability to modify the CoG position actively and can keep the flight stable. First, we introduce the hardware platform including the structure of link module and internal communication system to achieve the extensibility in terms of the link number. We then propose a method to find the optimal form for the multilinks based on the flight stability. Finally, we present experimental results which include aerial transformation and multiple objects transportation.

In recent years, applications of visual tracking algorithms has seen a substantial growth with deployments in intelligent robots such as drones for human tracking. The algorithms for such tasks has to be efficient in terms of computational cost while been robust, accurate and fast. Object tracking algorithms based on handcrafted heuristics and constraints are widely used in uav applications. The handcrafted heuristics are mostly implemented for task-oriented applications which limits the extensions in uav's capability beyond the predefined functions. This paper considers the challenges of tracking and landing an autonomous uav on a speed high moving target, and presents a visual tracking algorithm that integrates correlation filters with deep comparison network for real-time tracking with state-of-the-art accuracy. The method first tracks the target upto translation using an online learnt model via local search technique. The changes in scale is estimated by a deep comparison network (DCN) instead of the commonly used pyramidal approach. In a single network evaluation, DCN can estimate the changes in scale as well as compensate the drifting of the tracker by refining the object region estimated by the correlation filters. The network is end-to-end trained which attempts to learn a powerful matching function for object localization using a known template. Generally, the integrated framework can be viewed as coarse-to-fine level motion estimation. Moreover, the framework can redetect the lost target without a need for a separate detector.

Picking various objects in cluttered narrow space automatically is required for warehouse automation. In this space, multi-fingered robot hands have difficulty in grasping objects as objects are surrounded by obstacles. On the other hand, vacuum grippers have difficulty in grasping various objects stably. In this paper, we propose the Suction Pinching Hand, which has two underactuated fingers and one extendable and foldable suction finger whose fingertip has a suction cup. This hand can grasp objects in cluttered narrow space using the suction finger. In addition, it can grasp various objects stably using suction and pinch at the same time. The ability to grasp various objects stably of this hand is confirmed by tabletop experiments. We also propose a picking strategy using suction and pinch simultaneously in cluttered narrow space. We evaluate our proposed methods with shelf bin picking experiments. With our methods, a robot can pick various objects in cluttered narrow space.

In this study, we propose a novel method to operate whole body of a humanoid robot, which also includes both feet, dynamically and safely with the master-slave approach. The conventional whole body master-slave approaches need static balancing assumption or a certain time length of planning after operator's input. Then, we introduce a set of limitations that allows the robot to execute human's daily dynamic bipedal locomotion, but forbid dangerous motions like the COM will be gone outside of the support region. In the limitations, we regulate COM velocity based on a positional relation of the Divergent Component of Motion (Capture Point) and the both feet, and automatically modify the swing foot contact timing with judging the ZMP is inside or outside of the single foot support region. At last, we conducted some experiments of the real time master-slave locomotion with using two life-sized humanoid robots and confirmed the effectiveness of our novel limitation methods.

For humanoid robots, unexpected collision can cause instability of robot balancing and damage to both robots and environment. This paper presents a reactive bipedal walking controller against swing foot collision for humanoid robots. This controller is composed of following three components: 1) Swing Foot Trajectory Regenerator, 2) Swing Foot Collision Detector, and 3) Swing Foot Impact Mitigation Controller. By regenerating swing foot trajectory depending on situations, humanoid robots can avoid falling down. However, although humanoid robots detect collision and regenerate a swing foot, collision impact can cause bad effects such as damage and posture rotation. Therefore, to mitigate strong impact, we propose Swing Foot Impact Mitigation Controller, which is composed of two controllers. The proposed method is validated through the experiments by actual humanoid robot CHIDORI. We confirm that CHIDORI can avoid falling down against collision in two situations: walking on the flat ground, and stepping up a stair.

In this paper, we described a new developed life-size humanoid robot. A purpose of the developed robot is to realize continuous operation for a long time and to improve an action autonomously. we considered three aspects of robustness, mechanical robustness, functional robustness and robustness of an action. Mechanical robustness was confirmed by the experiment that the robot fell down without mechanical failures and continued to work after falling down by using hard points. Functional robustness was designed to use power cable and to wear a suit which can be changed by required functionality. Robustness of an action was achieved as a standing up action using “StateNet”, which realized autonomous error recovery. Finally, we present a methodology to develop a humanoid robot platform which can continue to work in the real world.

The human forearm is composed of two long, thin bones called the radius and the ulna, and rotates using two axle joints. We aimed to develop a forearm based on the body proportion, weight ratio, muscle arrangement, and joint performance of the human body in order to bring out its benefits. For this, we need to miniaturize the muscle modules. To approach this task, we arranged two muscle motors inside one muscle module, and used the space effectively by utilizing common parts. In addition, we enabled the muscle module to also be used as the bone structure. Moreover, we used miniature motors and developed a way to dissipate the motor heat to the bone structure. Through these approaches, we succeeded in developing a forearm with a radioulnar joint based on the body proportion, weight ratio, muscle arrangement, and joint performance of the human body, while keeping maintainability and reliability. Also, we performed some motions such as soldering, opening a book, turning a screw, and badminton swinging using the benefits of the radioulnar structure, which have not been discussed before, and verified that Kengoro can realize skillful motions using the radioulnar joint like a human.

Human can not only support their body during standing or walking, but also support them by hand, so that they can dangle a bar and others. But most humanoid robots support their body only in the foot and they use their hand just to manipulate objects because their hands are too weak to support their body. Strong hands are supposed to enable humanoid robots to act in much broader scene. Therefore, we developed new life-size five-fingered hand that can support the body of life-size humanoid robot. It is tendon-driven and underactuated hand and actuators in forearms produce large gripping force. This hand has flexible joints using machined springs, which can be designed integrally with the attachment. Thus, it has both structural strength and impact resistance in spite of small size. As other characteristics, this hand has force sensors to measure external force and the fingers can be flexed along objects though the number of actuators to flex fingers is less than that of fingers. We installed the developed hand on musculoskeletal humanoid “Kengoro” and achieved two self-weight supporting motions: push-up motion and dangling motion.

This paper explains low-latency joint torque feedback control based on torque estimation on each joint for gear-driven humanoid robots with harmonic drives. Force control of gear-driven robots has an advantage in its fully variable stiffness in comparison with elastic robots. However, feedback latency makes gear-driven robots vulnerable to impact rising in several milliseconds. It would be resolved by low-latency torque feedback loop in a single joint, but torque sensors are too large for life-sized humanoid robots. We estimate joint torque from motor current and rotation observed in each joint, and give artificial elasticity to joints using compliance control and shock absorption control. Our controller performance is demonstrated by landing experiments.

This paper presents a COM trajectory generation method for 3D walking and skating motion by nonlinear optimization. In our method, we solve the following problems: (1) dealing with both walking and skating motion in the same framework, (2) generating center of mass (COM) trajectory faster than execution time, (3) executing motion with large acceleration. For solving (1) and (2), we calculate the COM trajectory at every step and introduce frictional constraints to the Divergent Component of Motion as terminal conditions. By changing the terminal condition, we can generate both skating and walking motion. Besides, the nonlinear constrained optimization using Gauss Pseudospectral Method is introduced for solving (2) and (3). Thanks to this method, we generate the 3D COM trajectory considering contact constraints and kinematic constraints faster than execution time. Finally, the walking and skating experiment were carried out to confirm the effectiveness of our method using life-sized humanoid HRP-2. Applying the proposed method, HRP-2 could successfully walk at 0.4 [m/s] and skate at 1.0 [m/s].

Probabilistic 3D map has been applied to object segmentation with multiple camera viewpoints, however, conventional methods lack of real-time efficiency and functionality of multilabel object mapping. In this paper, we propose a method to generate three-dimensional map with multilabel occupancy in real-time. Extending our previous work [1] in which only target label occupancy is mapped, we achieve multilabel object segmentation in a single looking around action. We evaluate our method by testing segmentation accuracy with 39 different objects, and applying it to a manipulation task of multiple objects in the experiments. Our mapping-based method outperforms the conventional projection-based method by 40-96% relative (12.6 mean IU3d), and robot successfuly recognizes (86.9%) and manipulates multiple objects (60.7%) in an environment with heavy occlusions.

Groping behavior based on contact sensors is necessary for manipulation in an unknown environment. For those situations, it is effective for a robot to accumulate contact information as an environment map, and to plan the motions for executing the safe trial motion. We first propose a method of updating the occupancy grid map of the manipulation region from the contact information by introducing the contact sensor model. Using this map, we propose a method of sampling-based motion planning that enables the execution of the safe trial motion based on the criteria of feasibility and safety. To verify the effectiveness, we show the experimentally obtained results, showing that a real robot plans and executes the manipulation with groping behavior in the occluded environment.

We propose a human mimetic foot structure for musculoskeletal humanoids. We designed the foot structure by inspiring from human foot abilities of the multi-bone connected structure for flexibility and the distributed force sensor system. The foot has multi-DOFs structure including toe DOF that is composed of fingers. The distributed force sensing system is composed of 12 an-axis force sensors. In order to demonstrate those effectiveness, we implement the foot into musculoskeletal humanoid Kengoro and conduct several experiments. As a result, we confirmed effectiveness of the foot from tiptoe motion and balancing behavior by utilizing the foot characteristics.

To enable robotic sensing for tasks with requirements on weight, size, and cost, we develop an ultra-tiny line laser range sensor based on the Time-of-Flight (TOF) principle. With delicate circuit design and optical attachments, we create a sensor as small as 35[mm] × 27[mm] × 30[mm] and as light as 20[g]. The line sensor samples 272 pixels (256 effective pixels) uniformly distributed within the measurement field of view customizable using different laser lenses. The optimal measurement range of the sensor is 0.05[m] ~ 2[m]. Higher sampling rates can be achieved with a shorter range. The sensor can also extend its range to 3[m] with reduced accuracy. We model the overall errors of the sensor and formulate calibration methods, achieving repeatable accuracy and measurement bias both within 2[cm] with our tested ambient lighting conditions and measurement ranges. The sensor is applicable to range sensing tasks including humanoid hand-eye measurement, UAV safe landing, tiny robot range sensing, and object detection.

Humanoid robot has the potential to manipulate wide range of tools in daily life. Arms and legs of humanoid robot contribute this ability. Above all, manipulation tasks for vehicles which are the same size as a life-sized humanoid or larger size than it require the operational motion by both arms and legs of humanoid robot. In addition to the arms and legs cooperative motion control, it is also important for humanoid robot to stabilize self posture during driving vehicle. In this research, we focus on the arms-legs-integrated manipulation task for tricycle controlled by humanoid robot. We propose dual manipulators control law that is defined as active manipulator which works movable objects such as handle and crank, and passive manipulator which follows the movement of this objects. We discuss the self stabilizing strategy for humanoid robot by both active manipulating legs as well as manipulation strategy for objects. Furthermore, this paper contributes the strategy of recognition and planning for outside obstacle situations and configures the tricycle manipulation system. Applying this proposed system, we show the experimental result for tricycle manipulation by life-sized humanoid robot HRP2-JSK on obstacle-mixed situation.

In this paper, we develop a waterproof suit for humanoid robots and propose an underwater walking control method. Although very few life-sized humanoid robots are completely waterproof, we can easily make these humanoid robots watertight by putting a waterproof suit on them. In water, humanoid robots are influenced by the two forces due to the water: buoyancy and drag force. We take buoyancy into account when generating a walking pattern because the force is large and easy to estimate before walking. However, drag force is small and difficult to precisely predict and therefore, we treat the force as an unknown disturbance. In our method, we modify footsteps based on the Capture Point in order to deal with large disturbances. We verify the effectiveness of the proposed methods through an experiment in which a life-sized humanoid robot walks on a floor, stairs and debris in water.

In this paper we propose a novel method to utilize the skeletal structure not only for supporting force but for releasing heat by latent heat.

To achieve tasks in unknown environments with high reliability, highly accurate localization during task execution is necessary for humanoid robots. In this paper, we discuss a localization system which can be applied to a humanoid robot when executing tasks in the real world. During such tasks, humanoid robots typically do not possess a referential to a constant horizontal plane which can in turn be used as part of fast and cost efficient localization methods. We solve this problem by first computing an improved odometry estimate through fusing visual odometry, feedforward commands from gait generator and orientation from inertia sensors. This estimate is used to generate a 3D point cloud from the accumulation of successive laser scans and such point cloud is then properly sliced to create a constant height horizontal virtual scan. Finally, this slice is used as an observation base and fed to a 2D SLAM method. The fusion process uses a velocity error model to achieve greater accuracy, which parameters are measured on the real robot. We evaluate our localization system in a real world task execution experiment using the JAXON robot and show how our system can be used as a practical solution for humanoid robots localization during complex tasks execution processes.

For generating motions of robots, global search in configuration space is time consuming although it is sometimes indispensable (e.g. collision avoidance in complex environment). Our idea is to use global sampling algorithm not in the state space but in the task nullspace, which is the redundant degrees of freedom of the state space with respect to the task space. Because the task nullspace is smaller than the original search space (state space), fast global sampling is possible. For embedding this hidden task nullspace parameters, we propose a new deep-auto-encoder-based neural network structure. Our approach learns the map from task and task nullspace towards robot's state (Task-State Map, TSM). As the demonstration, the relationship between 28-dof joint angles (state) and the end-effector coordinates of all limbs (task) is learned, and egress postures and reaching postures are generated.

In this paper, we propose a real-time control method for skating motion of humanoid robots. There are three problems for skating motion: (1) keeping dynamic balance, (2) adequately controlling foot force to suppress slipping at the foot, (3) controlling full-body motion in real-time. For solving these problems, we propose the Skating Motion Generator and the Skating Motion Stabilizer. In the Skating Motion Generator, we separate the slip suppression from motion generation for (3). The separation enables us to generate skating motions in real-time. In the Skating Motion Stabilizer, we adjust the sole pressure distribution of each foot to solve the contradiction between (1) and (2). We show the effectiveness of the proposed controller through the experiments, in which life-sized humanoid HRP-2 pushes the ground and skates on the skateboard. Applying the proposed controller, HRP-2 could successfully accelerate and skate on the skateboard at 0.5[m/s].

We present an experience-based planning framework called Thunder that learns to reduce computation time required to solve high-dimensional planning problems in varying environments. The approach is especially suited for large configuration spaces that include many invariant constraints, such as those found with whole body humanoid motion planning. Experiences are generated using probabilistic sampling and stored in a sparse roadmap spanner (SPARS), which provides asymptotically near-optimal coverage of the configuration space, making storing, retrieving, and repairing past experiences very efficient with respect to memory and time. The Thunder framework improves upon past experience-based planners by storing experiences in a graph rather than in individual paths, eliminating redundant information, providing more opportunities for path reuse, and providing a theoretical limit to the size of the experience graph. These properties also lead to improved handling of dynamically changing environments, reasoning about optimal paths, and reducing query resolution time. The approach is demonstrated on a 30 degrees of freedom humanoid robot and compared with the Lightning framework, an experience-based planner that uses individual paths to store past experiences. In environments with variable obstacles and stability constraints, experiments show that Thunder is on average an order of magnitude faster than Lightning and planning from scratch. Thunder also uses 98.8% less memory to store its experiences after 10,000 trials when compared to Lightning. Our framework is implemented and freely available in the Open Motion Planning Library.

Humanoid robot is able to execute various behavior to manipulate objects because of high degree-of-freedom around the whole-body. Although hands contact with objects and exert force in ordinary pushing motion by robot, pushing motion contacting with the object at various regions of whole-body has potential for extending the scope of feasible manipulation. We derive the fundamental formulas of humanoid robot in the situation that the external force is applied to the arbitrary region of whole-body, and then propose the method to generate and execute the pushing motion based on the formulas. The proposed method is generalized for enabling to select a contact point with an object from whole-body regions and control the pushing force applied to the sensorless region. In order to verify the effectiveness, we show the experimental result that a lifesized humanoid carries large and heavy objects by pushing with various regions of whole-body.

We propose a sensor-driver integrated muscle module by integrating necessarily components for tendon-driven robot which is likely to complicate. The module has abilities of high-tension measurability and flexible tension control. In order to achieve flexible tension control, we developed the new tension measurement mechanism with high-tension measurability and the new motor driver which enables current based motor control. We demonstrate the tension control ability of the module by several experiments. Furthermore, utilizing the module advantage of design facilitation, we made two types of tendon-driven robots and confirmed effectiveness of the module.

We present a vision cognition framework for tomato harvesting humanoid robot based on geometrical and physical reasoning. Inspired from the natural human harvesting behaviour, our goal is to build a humanoid robot to pick tomatoes autonomously or with minimal human efforts. The proposed vision approach uses fusion of calibrated observation data from two RGB-D sensors installed on the head and the hand of the humanoid. We observe the natural human harvesting behaviour and equip our robot with similar grippers to follow the same picking processes for a specific fruit. In the vision approach, we mainly focus on modelling fruits in one branch and then estimating the pedicel direction of each fruit in a branch. Through pointcloud model segmentation, the primitive shape model of each fruit can be obtained and we consider a simple fact that crops in one branch should remain stable with respect to gravity and interaction forces from neighbouring crops in the branch. According to this assumption, a probabilistic model is created and the picking order in the branch is assigned under the evaluated geometrical structure. In the experiments, we tested harvesting of real tomatoes on actual branches and evaluated the successful harvesting rate.

This paper presents a novel control method to stabilize the whole-body motion of humanoid robots when climbing vertical ladders and transitioning between ladders and catwalks. In such environments, the body of the robot tends to incline and rotate because of the slippery surfaces. The inclination and rotation may cause the robot to fail to grasp and thus collide with the rungs. The proposed method modifies the subsequent contact position in real time based on the error of the current robot posture estimated with inertial measurement units (IMUs) and actual joint angles. This paper also presents a method of generating motion by minimizing the contact wrench. This method satisfies hardware limitations, such as collision avoidance, joint torque limits, and joint limits. Applying these methods to a humanoid robot, we realize the robust climbing and descending of multiple rungs of a vertical ladder and bidirectional transitioning from ladders to catwalks.

We have developed a robot with a new control mechanism in order to collect information on flying robots in multiple fields. We aimed for a function that could rotate the tilt angle continuously and without limit and a function for flying maintaining any desired tilt angle with a structure that could efficiently use the thrust generated by the propellers. We devised a mechanism that connected two bicopter modules, each of which combines two of the four propellers into one set and named this mechanism the “Bi2Copter”. This mechanism provided movements including landing, take-off, and flying with any desired tilt angle. This ability of this mechanism to fly walls with continuously changing surface angles and full 360° spherical coverage makes possible applications in investigation, measurement, etc. This report covers the design concepts of this flying robot, the structure design, basic control and operations verification.

In situations where humanoid robots with constrained posture walk through a narrow space (e.g. manufacturing plants and kitchens), shuffling motions that are stepless and possess wide foot supporting area are effective. One of the difficulties of humanoid's shuffle translations is the load distribution between both feet. If sole loads are not distributed appropriately, the humanoid robot cannot maintain target contact states of each foot, and it will result in slipping both feet or falling down. In this paper, we propose Slide Friction Control (S.F.C.): offline pattern generator and Slide Contact Stabilizer (S.C.S.): online controller. First, Slide Friction Control determines reference foot forces and COM trajectories by adjusting sole loads and considering kinematic friction. The appropriate load distribution of S.F.C. enables humanoid robots to maintain target foot contact states. Second, Slide Contact Stabilizer controls each foot by using damping control to realize reference foot forces determined by S.F.C. S.C.S. enables humanoid robots to slide foot smoothly by suppressing friction vibrations. We also take into consideration the dynamic balance of humanoid robots such as previous waking stabilizers. Finally, we demonstrate that the proposed system enables humanoid robot to slide their feet smoothly using a life-sized humanoid robot, HRP-2.

The main goal of this paper is to design and evaluate a spine structure which withstands various motions. The structure around the neck has a prevailing importance since it is involved in various motions of the upper half of the body. The new design method we introduce essentially shows how to design all 7 cervical vertebrae (the part of spine in the neck) in a limited space, actuated by the wires winded around the motors. Then we show the muscle arrangements around the upper half of the spine. More specifically, we make use of a so called planar muscle mechanism. An abduction experiment which requires great force around the spine is made to show its stability. Finally, we show a variable stiffness system which enables the spine to resist an impulsive force. We have tested the system in the situation of whiplash injury which is a case of extreme external forces which can occur in car crash accidents. As such we have evaluated the strength of the design and the viability of our robot to act as a human body simulator.

Whole-body holding manipulation is effective for carrying the handleless large object. In order to keep the object stability, the dexterous transition motion is necessary. From geometric and physical conditions of object manipulation, we propose the general method of generating the transition graph, which represents the object pose and grasp contact. By searching the path on the graph, the transition motion is planned automatically with considering the object motion and contact switching simultaneously. By generating and modifying the whole-body holding motion, the planned object motion is achieved stably. We show the effectiveness of the proposed method by the experiments, in which robot lifts up a large object with whole-body contact by the planned transition motion.

For generating whole-body behavior involving contacts with environments such as climbing ladder behavior or walking on terrain behavior, “contact-before-motion” approach was used in some previous researches. By separating contact search process and motion search process, whole-body behavior generation was achieved. However, in previous researches, there were few examinations about contact transition strategies while generating behavior. For changing contact states, there are many kinds of contact transition strategies such as walking, sliding, rotating and so on. For example, while generating standing up behavior, it is important to switch contact transition strategies because the lack of degree of freedom makes it difficult to detach limb contacts from ground, and it may be desirable not to detach contacts but to slide them. In this study, we propose a novel whole-body behavior generation algorithm which involves contact transition strategies switching function. Especially, in this paper, we focus on the walk-type and slide-type transition strategies switching. We call walk-type transition as the contact transition process which detaches some contacts, moves them, and attaches them again. Besides, we call slide-type transition as the contact transition process which keeps on attaching contacts, and slides to move them. By using this algorithm, it is possible to generate whole-body behaviors which are difficult or impossible to achieve only with walk-type transition and which are more desirable by comparing multiple transition strategies. Finally, we evaluated this algorithm by generating standing up behavior and sitting on chair behavior.

Recent research in psychology argue the importance of “context” in emotion perception. According to these recent studies, facial expressions do not possess discrete emotional meanings; rather the meaning depends on the social situation of how and when the expressions are used. These research results imply that the emotion expressivity depends on the appropriate combination of context and expression, and not the distinctiveness of the expressions themselves. Therefore, it is inferable that relying on facial expressions may not be essential. Instead, when appropriate pairs of context and expression are applied, emotional internal states perhaps emerge. This paper first discusses how facial expressions of robots limit their head design, and can be hardware costly. Then, the paper proposes a way of expressing context-based emotions as an alternative to facial expressions. The paper introduces the mechanical structure for applying a specific non-facial contextual expression. The expression was originated from Japanese animation, and the mechanism was applied to a real desktop size humanoid robot. Finally, an experiment on whether the contextual expression is capable of linking humanoid motions and its emotional internal states was conducted under a sound-context condition. Although the results are limited in cultural aspects, this paper presents the possibilities of future robotic interface for emotion-expressive and interactive humanoid robots.

Object handover is a basic task in many human-robot interactive scenarios and therefore, it is important for assistive robots to be able to perform proper handovers. We previously designed a human-inspired grip-force-varying handover controller for a robot giver and showed on a Willow Garage PR2 robot that the controller yields human-like and human-preferred handovers. The PR2 robot had a non-compliant fully-actuated gripper. However, recently, compliant underactuated grippers have been gaining more popularity. Although compliant underactuated grippers can provide more flexibility in manipulation, it is generally difficult to accurately measure and control the amount of applied grip force. In this paper, we present an implementation of the human-inspired handover controller on a Kawada Industries HRP4R robot, which has compliant underactuated hands, using joint position error measurement for estimating the amount of applied grip force. Through an experiment, we show that we are able to achieve safe, smooth, and intuitive robot-human handovers despite the lack of accurate grip force control on our robot.

Life-size humanoids which have the same joint arrangement as humans are expected to help in the living environment. In this case, they require high load operations such as gripping and conveyance of heavy load, and holding people at the care spot. However, these operations are difficult for existing humanoids because of their low joint output. Therefore, the purpose of this study is to develop the highoutput life-size humanoid robot. We first designed a motor driver for humanoid with featuring small, water-cooled, and high output, and it performed higher joint output than existing humanoids utilizing. In this paper, we describe designed humanoid arm and leg with this motor driver. The arm is featuring the designed 2-axis unit and the leg is featuring the water-cooled double motor system. We demonstrated the arm's high torque and high velocity experiment and the leg's high performance experiment based on water-cooled double motor compared with air-cooled and single motor. Then we designed and developed a life-size humanoid with these arms and legs. We demonstrated some humanoid's experiment operating high load to find out the arm and leg's validity.

This paper proposes a humanoid dance motion generation system that deals with a huge variety of leg motions. While previous research only tackled on a few kinds of leg motions, original human dance leg motions contain various foot touch states such as slide, turn, and heel contact, as well as complex motions such as kick and twist. According to the dance literature, we found that there are seven major foot touch states that make dance motion more “dance-like”. Thus we present a method to classify the seven kinds of foot touch state from human dance motion data, and describe the various dance leg motions by using combinations of the foot touch states and key-frames. Based on these methods, we designed the humanoid dance motion generation system that enables humanoid robots not only to satisfy the geometric condition but also to imitate various human dance leg motions. Finally we show an experiment using a life-sized humanoid, HRP-2.

In case that a robot carries large and heavy objects with unknown physical parameters such as mass automatically, the autonomous decision and execution of the manipulation strategy are necessary. The method to decide the proper strategy from the various candidates depending on the object is a difficult problem and not researched widely. We consider the operation as the mapping from the physical parameter space to the object motion space. Based on the concept of mapping, we define the strategy proving operation (SPO) for determination of strategy feasibility. We introduce two examples of SPO and construct the system for deciding strategy from lifting, pushing, and pivoting. Executing the strategy in the situation that physical parameters are not known is also necessary. We construct the generator and controller for the full-body manipulation, which can be employed regardless of strategy. The controller enables the robot to exert adequate force while keeping balance. We clarify the applicable scope of the proposed method and show that a life-sized humanoid decides the strategy and carries various large and heavy objects autonomously through the experiment.

For humanoid robots to perform whole-body motions, a motion planner should generate feasible motions satisfying various constraints including joint torque limitation, friction, balancing, collision, and so on. Furthermore, for life-size humanoid robots to perform higher-load motions, such as climbing ladders, safely, it is important to generate motions which requirements are not too close to the limitations. In this paper, we propose a humanoid motion planner based on Body Retention Load Vector (BRLV), which is a novel index for representing severity of physical constraints: limitation of joint Torque, contact Force, and contact Moment (TFM limitations). By minimizing the norm of BRLV, we obtain humanoid motions that are farthest from TFM limitations. Finally, we evaluate the proposed motion planner in simulation and confirm the effectiveness of the planner through experiments in which a life-size humanoid robot climbs a ladder and a car.

We present a method for enabling robots to determine appropriate grasp configurations for handovers - i.e., where to grasp, and how to orient an object when handing it over. In our method, a robot first builds a knowledge base by observing demonstrations of how certain objects are used and their proper handover grasp configurations. Objects in the knowledge base are then organized based on their movements and inter-object interaction features. The key point in this process is that similarity in affordances should be recognized. When subsequently asked to handover an object, the robot then computes an appropriate grasp configuration based on the object's recognized affordances. Experimental results show that our method was able to differentiate and group together objects according to their affordances. Furthermore, when given a new object, our method was able to generalize data in the knowledge base and determine an appropriate grasp configuration.

This paper describes a textureless object segmentation approach for autonomous service robots acting in human living environments. The proposed system allows a robot to effectively segment textureless objects in cluttered scenes by leveraging its manipulation capabilities. In our pipeline, the cluttered scenes are first statically segmented using state-of-the-art classification algorithm and then the interactive segmentation is deployed in order to resolve this possibly ambiguous static segmentation. In the second step the RGBD (RGB + Depth) sparse features, estimated on the RGBD point cloud from the Kinect sensor, are extracted and tracked while motion is induced into a scene. Using the resulting feature poses, the features are then assigned to their corresponding objects by means of a graph-based clustering algorithm. In the final step, we reconstruct the dense models of the objects from the previously clustered sparse RGBD features. We evaluated the approach on a set of scenes which consist of various textureless flat (e.g. box-like) and round (e.g. cylinder-like) objects and the combinations thereof.

This paper presents a method for daily assistive robots to manipulate objects whose contact relationships are important. In our approach, contact relationships between multiple objects such as “on” and “into” are estimated using the hierarchical relationships between the objects' regions in the images. The hierarchical states are used for failure recovery processes so that the target contact state is achieved. We tested our method in two daily tasks by HRP-2VZ humanoid robot: filing, and serving an egg with a spatula. These results suggested the effectiveness of our approach.

Human knee joint has a yaw-axis rotational DOF and a locking mechanism called screw-home mechanism. We focus on this mechanism and implement it to a musculoskeletal humanoid through hardware design. The importance of developing a knee joint with screw-home mechanism is that such a joint is capable of working yaw-axis properly and generating enough pitch joint torque for supporting whole body motion. In this paper, as an evaluation of our developed knee joint, we first checked the moment arm of the yaw rotational axis of the knee. Moreover, we also checked the yaw angle displacement during squat motion. From these results, we confirmed that the mechanism worked properly. Second, in order to check whether enough pitch joint torque is generated during movement, we conducted several experiments with whole body motions such as squatting. Lastly, as unique and integrated motions that involve the use of yaw DOF derived from the mechanism, we tested knee joint Open-Close, Right-to-Left and whole body twist squat motion. Our results demonstrated the feasibility of musculoskeletal humanoids with screw-home mechanism and showed that we have achieved humanlike twisting motion.

This paper presents a multi-field universal vehicle that is able to work at land, sea and air. The vehicle consists of a quad-copter with variable-pitch propellers that enable the vehicle to stand on the ground at a given tilt angle, roll on the ground like a wheel, and float and move on the water, in addition to flying like a conventional quad-copter. This article clarifies the behavioral objectives, structural design, basic control mechanism of the ring-shaped robot, and examples of 3D measurements.

This paper presents a design methodology for humanoid upper limb based on human anatomy. Kenshiro is a full body tendon driven humanoid robot and is designed from the data of average 14 year old Japanese boy. The design of his upper limb is realizing detail features of muscles, bones and the adhesive relation of the two. Human mimetic design is realized by focusing on the fact that joints are being stabled by muscles winding around the bones, and by accurately mimicking the bone shape this was enabled. In this paper we also introduce details of mechanical specifications of the upper limb. By having muscles, bones, and joint structures based on human anatomy, Kenshiro can move flexibly. The use as human body simulator can be expected by measuring sensor data which can correspond to biological data.

In the case of object manipulation by a humanoid robot, it is important to deal with contact states between objects, a robot, and an environment both to avoid falling down and to achieve objective manipulations. We propose a method to describe and uniformly execute various object manipulations by a humanoid robot. In description, we focus on the contact states and define manipulation phases according to the contact states. In execution, the humanoid's controller autonomously switches manipulation phases and substantiates the contact-force controller. According to switching of the manipulation phases, the humanoid's manipulation system switches the inputs for the contact-force controller, which includes the estimation of object's information and motion generation. We evaluated our proposed system through experiments in which the HRP-2 robot manipulates four objects without information about the objects' masses and necessary operational forces.

Many of today's mobile robots are supposed to perform everyday manipulation tasks autonomously. However, in large-scale environments, a task-related object might be out of the robot's reach. Hence, the robot first has to search for the object in its environment before it can perform the task. In this paper, we present a decision-theoretic approach for searching objects in large-scale environments using probabilistic environment models and utilities associated with object locations. We demonstrate the feasibility of our approach by integrating it into a robot system and by conducting experiments where the robot is supposed to search different objects with various strategies in the context of fetch-and-delivery tasks within a multi-level building.

Pushing heavy and large objects in a plane requires generating correct operational forces that compensate for unpredictable ground-object friction forces. This is a challenge because the reaction forces from the heavy object can easily cause a humanoid robot to slip at its feet or lose balance and fall down. Although previous research has addressed humanoid robot balancing problems to prevent falling down while pushing an object, there has been little discussion about the problem of avoiding slipping due to the reaction forces from the object. We extend a full-body balancing controller by simultaneously controlling the reaction forces of both hands using dual-arm force control. The main contribution of this paper is a method to calculate dual-arm reference forces considering the moments around the vertical axis of the humanoid robot and objects. This method involves estimating friction forces based on force measurements and controlling reaction forces to follow the reference forces. We show experimental results on the HRP-2 humanoid robot pushing a 90[kg] wheelchair.

Humanoid robots working in a household environment need 3D geometric shape models of objects for recognizing and managing them properly. In this paper, we make humanoid robots creating models by themselves with dual-arm re-grasping (Fig.1). When robots create models by themselves, they should know how and where they can grasp objects, how their hands occlude object surfaces, and when they have seen every surface on an object. In addition, to execute efficient observation with less failure, it is important to reduce the number of re-grasping. Of course when the shape of objects is unknown, it is difficult to get a sequence of grasp positions which fulfills these conditions. This determination problem of a sequence of grasp positions can be expressed through a graph search problem. To solve this graph, we propose a heuristic method for selecting the next grasp position. This proposed method can be used for creating object models when 3D shape information is updated on-line. To evaluate it, we compare the result of the re-grasping sequence from this method with the optimal sequence coming out of breadth first search which use 3D shape information. Also, we propose an observation system with dual-arm re-grasping considering the points when humanoid robots execute observation in the real world. Finally, we show the experiment results of construction of 3D shape models in the real world using the heuristic method and the observation system.

In order to know human dynamics, humanoid as a human body simulator is increasing its importance. Such humanoid is expected to have human musculoskeletal structure as close as possible. From this viewpoint, we are trying to create new musculoskeletal humanoid which has detailed human imitating structure, such as bi-articular muscle, muscle arrangement, joint structure and so on. In this paper, we address the design of lower thigh. The concepts of the thigh include leg configuration, new knee joint and link, and artificial muscle arrangements, especially knee joint structure imitating human flexible motion. The knee joint has yaw axis DOF and its locking mechanism which is usually simplified in robotics. Finally, we conduct extension, flexion and rotation as basic experiment to confirm the joint characteristics. Also, we conduct rotation experiment in the ground state to confirm the contribution of yaw axis DOF for human-like motion.

To design a robot with humanlike body structure, this paper presents a design methodology for a humanoid upper limb by tendon driven system. We newly designed an upper limb and rib cage like thorax for a musculoskeletal humanoid robot, based on the knowledge of anatomy. The robot consists of muscle, bone, and joint structure based on human and is expected to move flexibly and dynamically. This paper describes how to design such an upper limb and proposes the key mechanical design points, which is “rib surface thorax”, “muscle cushion”, “planar muscle”, and “open type ball joint”. To show that these mechanisms is effective in making a musculoskeletal humanoid robot, we examine the motion range of the robot. One of our goals is to enable robots to do the same movements as humans do through mimicking the human body structure, finding some important elements of human nature. This robots explained in this paper is the prototype for a new life size robot “Kenshiro” project.

We have been developing and studying musculoskeletal humanoids. Our goal is to realize more human-like humanoids which can do natural and dynamic motions as well as humans. Especially motions with complex contact with environments, which is jumping, running, catching a ball, and landing on one's hands, etc, are difficult to be achieved by humanoids. To achieve that motions, robots have to absorb shock force so as not to break ones' body structures, such as gears, motors, body links and so on. Musculo-skeletal humanoids are suited for these situations, because they can easily have mechanical flexibilit for shock absorption by adding elastic units, which is nonlinear spring units, to its own tendons. In this paper, we propose shock absorption methods by musculoskeletal humanoids, which uses its own mechanical flexibilit and simple refle of each muscles based on tension sensor. This strategy is inspired by human's motion control, and it can achieve shock absorption tasks without any fast sensor feedback controls and any prediction ocntrols used by conventional robots with rigid bodies. We chose a catching a ball task as an example of complex contact motions, implemented the proposed strategy to musculoskeletal humanoid Kenzoh and confirme the feasibility of proposed method by actually catching a ball demonstration.

Manipulation of structured objects connected to the environment by a kinematics chain involves two problems: (a) The objects have movable directions and unmovable directions. An undesired reaction force in the unmovable directions prevents a robot from successful manipulation; (b) The reaction forces from the objects could fluctuate during manipulation. Related works have enabled robots to manipulate objects by integrating position control in movable directions and force control in unmovable directions at the hands. However, in the case of a humanoid robot, too large undesired reaction forces in movable directions cause the robot's falling down and slipping. In this paper, we propose a controller system controlling reaction forces at the hands and successively updating reference forces based on reaction forces. For problem (a), we apply force control both to the movable and unmovable directions in order to satisfy both maintaining full-body balance and achieving manipulation. For problem (b), the update of the reference forces enables the humanoid robot to adapt to fluctuation of the reaction forces. We show experimental results on the cmanipulating four doors and a drawer.

A walking biped robot is required to change its walking direction and speed with minimal delay and not to tip off even when subjected to an unexpected external force. A major drawback of Zero Moment Point (ZMP) based online walking pattern generation methods is that arbitrary ZMPs cannot be achieved without divergence of the Conter of Mass (CoM). In this paper, we propose a new online walking trajectory generation method that utilizes nondivergence conditions of ZMP-CoM in a successive manner. This method enables changes in the walking direction and speed of a robot and push recovery under an unknown external force in unifie form. Experiments confir walking change of up to 4.05 km/h in the speed and changes in the walking direction with minimum delay and successful push recovery under an implse of 22 Ns.

In this paper we propose a “Direct-Mapping Method” and describe a wearable device which has a similar muscle arrangement as a tendon-driven humanoid robot. This device has linear-encoders as many as the muscles of the robot to control, and they are arranged like the robot's muscle alignment. Tendon-driven humanoids have advantages of their human size, multiple Degrees of freedoms and their musclo-skeletal structure, therefore they are able to make more human-like poses than shaft-driven humanoid robots. If we can teach human poses to them, they become more human-like so that we feel a sense of familiarity with robots. However, because of their complex structure, sometimes they make incorrect poses. There must be a feedback system to control their bones in detail. To make human-like pose naturally and to make precise motion by feed back to the user, the best way may be a wearable device. With this device, we can control tendon-driven humanoids without making their simulation model in the computer. We discuss the hardware settings of the device and the controlling system in this paper. Experiments of controlling a tendon-driven humanoid were conducted to demonstrate the effectiveness of the device. When controlling the robot, a situation that some muscles are loose or too tight occured. However, this device's behavior-teaching framework contributes to detailed robotic movement in the future.

Many countries around the world face three major issues associated with their aging societies: a declining population, an increasing proportion of seniors, and an increasing number of single-person households. To explore assistive technologies that can help solve the problems faced by aging societies, we have tested several information and robot technologies. This paper introduces research on a home-assistant robot, which improves the ease and productivity of home activities. For people who work hard outside the home, the assistant robot performs chores in their home environment while they are away. A case study of a life-sized robot with a humanlike functional body performing daily chores is presented. An integrated software system incorporating modeling, recognition, and manipulation skills, as well as a motion generation approach based on the software system, is explained. Moreover, because housekeepers perform chores one after another in their daily environment, we also aim to develop a system for continuously performing a series of tasks by including failure detection and recovery.

A humanoid robot working in a household environment with people needs to localize and continuously update the locations of obstacles and manipulable objects. Achieving such system, requires strong perception method to efficiently update the frequently changing environment. We propose a method for mapping a household environment using multiple stereo and depth cameras located on the humanoid head and the environment. The method relies on colored 3D point cloud data computed from the sensors. We achieve robot localization by matching the point clouds from the robot sensor data directly with the environment sensor data. Object detection is performed using Iterative Closest Point (ICP) with a database of known point cloud models. In order to guarantee accurate object detection results, objects are only detected within the robot sensor data. Furthermore, we utilize the environment sensor data to map out of the obstacles as bounding convex hulls. We show experimental results creating a household environment map with known object labels and estimate the robot position in this map.

We propose a strategy for a robot to operate in an environment with movable obstacles using only onboard sensors, with no previous knowledge of the objects in that environment. Movable obstacles are detected using active sensing and a color range sensor, and when an obstacle is moved, the perception of the environment is reconstructed. Active sensing is defined as the classification of an object as either movable or static after the robot tries to push the object using its arm. This classification is collectively based on force sensor inputs, joint angles, and color range sensor inputs. In order to gather information from the environment, we use a color range sensor consisting of a TOF (Time of Flight) range sensor and conventional stereo cameras. Finally, we show experimental result in the environment with movable obstacles such as a table and chairs. Humanoid robot HRP-2 detects that a chair is a movable obstacle, moves the chair to clear a path to its goal, and then reaches the goal.

In this paper we propose a new method to manipulate heavy objects for a humanoid robot. In this method the manipulation strategy is determined based on on-line estimation of the operational force. We integrate these functions with a real-time controller that controls the external force and maintains full-body balance. The feature point of our work is that since a full-body control system includes switching of the manipulation strategy based on the operational force estimated on-line the system enables a humanoid robot to manipulate heavy objects as well as light objects. The effectiveness of our whole system is confirmed in our experiments, in which a humanoid robot manipulates up to 12[kg] while estimating the object's weight.

The high power ability of humanoid is desired for application of nursing or running or jumping motions. Achievement of the actuator of light and powerful equivalent to humans is required. In this paper, we propose a method to extract inherent performance from motors by an active temperature control. The method safely improves the output of motors. The active temperature control is achieved by combining the estimation of an internal temperature of the motor with the forced cooling by liquid. We also developed high power motor drivers for the proposed method. An experiment of a high power joint test bench is shown. In this paper, we show a high power prototype biped robot for application of nursing, running or jumping motions. High power actuator system and robust internal body network are developed for high power robot. Basic demonstration experiments of high power motion are shown.

This paper describes a software system integration of daily assistive robots. Several tasks related to cleaning and tidying up rooms are focused on. Recognition and motion generation functions needed to perform daily assistance are developed, and these functions are used to design various behaviors involved in daily assistance. In our approach, the robot behaviours are divided into simple units which consist of 3 functions as check/plan/do, it provides us with high reusable and flexible development environment. Because sequential task execution can be achieved only after functions about failure detection and recovery, we also try to implement such functions in keeping with this approach. In addition to using simple behavior unit, multilayer error handling is effective. Experiments doing several daily tasks with handling daily tools showed the effectiveness of our system.

Model-based approaches in recognition and planning of robots work effectively, and these approaches can apply to model-less situation using autonomous model construction by an agent. There are problems about segmentation or shape fitting of various objects with different scales or shapes. In this paper, we construct a Head-mounted 3D multi sensor for 3D environment modeling and propose a method of 3D reconstruction for various objects using intentional behavior of human.

In this paper we present a new method for picking up dishes based on active groping. Though a bird's eye view is commonly used to recognize dishes, this method tends to produce errors in the presence of large occlusion, A multisensory robot hand can be used in a kitchen environment to probe and grasp dishes that are placed close to each other. Though sensing is an effective method, it is difficult to recognize a dish globally by this method because of the locality of measurement. To solve this difficulty, we propose a sensing method to aquire the geometric information about dishes by tracing their surface. We demonstrate that a robot can pick up three types of dishes on the basis of this active groping strategy by conducting an experiment.

Humanoid robots for home daily assistance need to have an autonomous behavior selection system. To realize this, situation recognition capability is important. In this paper, we propose a situation recognition system where the use of daily life sounds enables to recognize situations difficult to understand using only visual sensor data and where the use of time series of information enables robust situation recognition. We apply cepstrum feature for recognition of daily life sounds and Dynamic Bayesian Networks(DBN) for robust situation recognition. As an example of situation recognition, we show some experiments and results targeting some situations that a human is in a kitchen.

Methods for a daily assistive humanoid robot to manipulate and recognize the objects incorporating joints and learn the manipulation knowledge are presented. It is necessary for humanoid robots to use the objects incorporating joints such as some furniture and tools to provide daily assistance. We have been tried to make an integrated humanoid robots recognition and manipulation system of the objects and tools in the real world. We extend the system for the objects incorporating joints. In this paper, a recognition system in which the robots recognizes the objects incorporating joints by the visual 3D object recognition method with multi-cue integration using particle filter technique and a manipulation system of them are shown. The search areas of the joints are automatically generated based on the manipulation knowledge. We present three key techniques to recognize and manipulate the objects incorporating rotational and linear joints. 1) Knowledge description for manipulation and recognition of these objects; 2) Motion planning method to manipulate them; and 3) Recognition method of them closely related to the manipulation knowledge. Moreover, a method for a person to teach the handle, one of manipulation knowledge, visually to the robot is shown. Finally, a daily assistive task experiment in the real world using these elements is shown.

In this paper, we treat with wheelchair support by a life-sized humanoid robot. It is quite essential to integrate whole-body motion, recognition of environment and human-interface behind the user in order to achieve this task. Contributions of this paper is whole-body control including pushing motion using the offset of the ZMP and observation of the attitude outlier, recognition of the wheelchair using particle filter and human-interface behind the person using face detection and recognition of gesture.

This paper describes daily assistive task experiments that conducting on the HRP2JSK humanoid robot. We present overall action and recognition integrated system design to realize daily assistive behaviors autonomously and robustly, along with the demonstration that the HRP2JSK pours tea from a bottle to a cup and wash it after human drink it. To obtain autonomy and robustness, visual recognition and behavior control through perception information are important.

This paper proposes the new robot programming environment in which robot motion programming environment and dynamics simulator are integrated. This allows robot motion programs to include simulation descriptions. Additionally, a new implementation of simulation that is composed by simulation modules is presented, on the other hand, conventional simulators are monolithic and implanted every function. This makes it difficult to add new simulation functions such as new sensors on a simulator by its users. In the new method, the users of the environment can add new modules easily. The simulation function of this system is evaluated by showing new robot motion simulations like brooming, seesaw and so on. The experiment that shows how the simulation embedded brain changes the motion planning of block moving problem is illustrated in the end this paper.

A vision based object recognition subsystem on knowledge-based humanoid robot system is presented. Humanoid robot system for real world service application must integrate an object recognition subsystem and a motion planning subsystem in both mobility and manipulation tasks. These requirements involve the vision system capable of self-localization for navigation tasks and object recognition for manipulation tasks, while communicating with the motion planning subsystem. In this paper, we describe a design and implementation of knowledge based visual 3D object recognition system with multi-cue integration using particle filter technique. The particle filter provides very robust object recognition performance and knowledge based approach enables robot to perform both object localization and self localization with movable/fixed information. Since this object recognition subsystem share knowledge with a motion planning subsystem, we are able to generate vision-guided humanoid behaviors without considering visual processing functions. Finally, in order to demonstrate the generality of the system, we demonstrated several vision-based humanoid behavior experiments in a daily life environment.

This paper presents a work toward the old dream of the housekeeping robot. One humanoid robot will cooperate with the user to cook simple dishes. The system combines predefined tasks and dialogues to find a plan in which both robot and user help each other in the kitchen. The kitchen problem allows the demonstration of a large variety of actions, and then the necessity to find and to plan those actions. With this problem the task planner can be fully used to enhance the robot reasoning capacity. Furthermore the robot must also use motion planning to have general procedures to cope with the action planned. We focus on the planning problems and the interactions of these two planning methods

Online self collision detection system for humanoid robots is an essentially important function for developing sensor based behaviors without worrying about breaking hardware. In this paper, we propose a practical and real-time self collision detection system for humanoid robots that satisfy both enough a range of movement and a safety margin. Previous researches usually add a safety margin around each link to cope with errors in both modeling and control. However this margin significantly decreases a range of movement of a joint, especially a compound joint, a joint between adjacent links and composed of 2 or 3 revolute joints whose axis intersect in a same point. In order to gain enough a range of movement and safety margin, we developed hybrid approach that uses both table based collision checking for compound joints and online geometrical model checking with a simplified link shape for other joints. We have experimentally evaluated our self collision detection system using a HRP2-JSK humanoid robot. Our demonstration shows that the robot automatically stops its motion when self collision occurs

In this paper, we describe the real-time and precise self collision detection system that does not reduce the number of polygons and checks more than 100 collision pairs in real-time by using AABB based collision detection libraries. Previous researches on collision detection of humanoid robots which reduce collision pairs or simplify a shape of a robot has disadvantages such as increasing the dangerousness or decreasing range of movement. However our self collision detection system uses detailed geometric model and collision pairs as many as possible. We have experimentally evaluated collision detection libraries on a real-time self collision detection application of a humanoid robot. This experiment suggests that AABBs based method is much faster than conventional OBBs based method. Finally, we demonstrated real-time collision detection and avoidance function that automatically stops entire motion if self collision occurs using HRP2 humanoid robot.

This paper describes vision-based 3D walking system of a humanoid robot by combining a precise 3D planar surface detection method and a practical 3D footstep planner method. The walking control system requires vision system with 10[mm] accuracy. Then we developed the precise 3D planar surface recognition system by combining the 3D Hough transformation method and the robust estimation method. We also developed practical 3D foot step planner by considering kinematics and dynamics restriction of robot hardware. Finally, we realized vision based 3D walking experiments that a humanoid robot steps upon an unknown obstacle are shown.

This paper describes the design and development of system software for humanoid robots such that researchers who specialize not only in biped walking but also in various fields are able to use humanoid robots as a research tool. For this purpose, the system for a humanoid must integrate and organize each subsystem such as control, recognition, dialogue, planning and so on, and it must provide efficient full-body motion control by specifying fewer degrees of freedom than all joints. Our system design provides a common interface among subsystems by implementing each function as a method call through a three-dimensional model of the robot for good integration, and it also provides a motion planning technique based full-body posture sequence and walking pattern generation. Finally, we show integrated behavior experiments with vision, planning and motion control using the developed system software for a life-sized humanoid robot, HRP2.

We propose the portable situation reporting system by a small robot for daily life, which demands the rapidly system reconstitution for daily life support. In this paper, we established a system by an attachment mechanism assisted the robot system based on device-distributed approach and the patterned processing by the tree structuring of sensor information processing. The attachment mechanism facilitates the hardware system reconstitution and the patterned processing simplifies the software system restructuring. We evaluated the reconfigurable system not to re-write programs and judged the effectiveness of the system by the experiments.

In this paper, we describe a planner for a humanoid robot that is capable of finding a path in an environment with movable objects, whereas previous motion planner only deals with an environment with fixed objects. We address an environment manipulation problem for a humanoid robot that finds a walking path from the given start location to the goal location while displacing obstructing objects on the walking path. This problem requires more complex configuration space than previous researches using a mobile robot especially in a manipulation phase, since a humanoid robot has many degrees of freedom in its arm than a forklift type robot. Our approach is to build environment manipulation task graph that decompose the given task into subtasks which are solved using navigation path planner or whole body motion planner. We also propose a standing location search and a displacing obstacle location search for connecting subtasks. Efficient method to solve manipulation planning that relies on whole body inverse kinematics and motion planning technology is also shown. Finally, we show experimental results in an environment with movable objects such as chairs and trash boxes. The planner finds an action sequence consists of walking paths and manipulating obstructing objects to walk from the start position to the goal position.

We present an integrated humanoid locomotion and online terrain modeling system using stereo vision. From a 3D depth map, a 2.5D probabilistic description of the nearby terrain is generated. The depth map is calculated from a pair of stereo camera images, correlation-based localization is performed, and candidate planar walking surfaces are extracted. The results are used to update a probabilistic map of the terrain, which is input to an online footstep planning system. Experimental results are shown using the humanoid robot H7, which was designed as a research platform for intelligent humanoid robotics.

This paper describes an experimental stereo vision based motion planning system for humanoid robots. The goal is to automatically generate arm trajectories that avoid obstacles in unknown environments from high-level task commands. Our system consists of three components: 1) environment sensing using stereo vision with disparity map generation and on-line consistency checking, 2) probabilistic mesh modeling in order to accumulate continuous vision input, and 3) motion planning for the robot arm using RRTs (rapidly exploring random trees). We demonstrate results from experiments using an implementation designed for the humanoid robot H7.

To realize humanoid robots in unknown environment, sensor based navigation system is required as one of an essential function. This paper describes vision-based navigation system for humanoid robots, which has following features: 1) To recognize floor regions from a view of vision of a humanoid robot in unknown environment, we utilized existing technique called Plane Segment Finder, which is able to extract arbitrary planner surface regions from depth image. 2) Path planning for wheeled robots usually models a robot as a 2D circle, however path planning system for humanoid robot requires capable of modeling a robot as a 3D cylinder model, convex hull model, rigid model and so on, according to a situation such as a robot carries a large object or a robot opens its arms. Finally, we show a humanoid robot HOAP-1 with enhanced stereo vision system for navigation task and a result of path planning using generated local map through stereo vision system which uses real images as an input.

This paper describes a rapid development system for humanoid vision-based behavior consisting of both real and virtual robots in a simulation environment. Previous robotics simulators were limited to verify dynamic motions such as walking, or to plan or learn in a simple environment. However, our system is able to simulate vision based behavior, i.e. motion and perception behavior, of a robot as a whole, since it can simulate both dynamics and collisions in the environment. We regard the real-time aspect of the simulation as important so that so that users can develop vision based behavior rapidly and efficiently. We employ a Common Interface API to share behavior software between real and virtual robots. Moreover, visual processing functions such as color extraction and depth map generation are available. As a result, vision based behavior consisting of local map generation, planning and navigation of the humanoid in simulation is presented. We also show that software developed in the simulation environment is applicable to real robots.

This paper describes the development of a plane segment finder, which is able to detect three-dimensional planar surfaces from input images in real-time. We propose an algorithm for detecting plane segments, that includes: 1) plane segment candidate extraction using the 3D Hough transform from depth map information; and 2) fitting the plane segment candidates to the depth map in order to detect the partial plane segment, since the extracted plane segment candidates are general planes, with no boundary. To achieve real-time plane segment finding system, we apply: 1) the recursive correlation method for depth map generation; and 2) the randomized Hough transform for plane segment extraction. Finally, experimental results using an implementation of our system along with a humanoid robot are shown.

This paper shows walking human avoidance and detection behavior of a mobile robot using the 3D depth flow. The 3D depth flow proposed is able to measure the 3D motion vector of every pixels between two time sequential images. First, a definition of the 3D depth flow and a simple 3D depth flow calculation method are presented. Then an implementation of the real-time 3D depth flow generation system using a standard PC is described, and the experimental results are given. Finally, as an application, the walking human detection and avoidance task using a mobile robot in real environments is shown.

This paper proposes a 3D depth flow generation method which measures 3D motion vector of every pixels between two time sequential images. First, the definition of 3D depth flow and a simple method in order to generate the 3D depth flow are described. Then the implementation of a real time 3D depth flow generation system using only a PC is presented, and experimental results are given. Finally, as an application, a walking human tracking task using mobile robot is included.

This paper describes the design and implementation of real-time depth map generation system with four key issues: 1) recursive (normalized) correlation technique; 2) cache optimization; 3) online consistency checking method; and 4) applying the MMX/SSE/sup R/ multimedia instructions set. The system is implemented on a standard PC/AT hardware with simple image capture board. Implementation details and system evaluations are also denoted. Furthermore, experiments with robots in real world are shown.

This paper describes a vision system which recognizes 3D objects in real-time by modeling the shapes of objects and matching the generated models. We develop the following methods for practically solving important problems of integration such as the estimation of sensor accuracy as well as real-time processing: 1) we reduce the computation of signed-distance, which is necessary to apply the marching cubes algorithm, and select the optimal resolution of models to be generated using an octree, thereby enabling us to generate hierarchical mesh models in real-time; 2) we apply spin-image matching by selecting the resolution of generated models and the coarse to-fine algorithm, consequently, we are able to efficiently match multiple objects of different sizes.

A legged robot "JROB-1" is developed for a robotics research platform as a result of inter-university research program on intelligent robotics supported by the Ministry of Education Grant-in-Aid for Scientific Research on Priority Areas in Japan. The JROB-1 features: 1) self-contained, 2) RT-Linux running on PG/AT processes vision and sensor processing, motion planning and control, 3) connected to a network via radio Ethernet as to utilize networked resources, 4) Fujitsu color tracking vision board and Hitachi general purpose vision processing board, 5) all parts are commercially available, and 6) it is extensible with respect to sensor, sensor processing hardware and software. JROB-1 is expected to be a common testbed for experiment and intelligent robotics research by integrating perception and motion.

View changing because of vibration while walking is one of the most fundamental problem for a vision based legged robot. To overcome this difficulty, three key issues are denoted: a) integration of color segmentation, optical flow and stereo, which is able to apply to vibrating view using correlation hardware, b) software servo loop implemented as a kernel module for the purpose of soft actuation, and c) smooth walking pattern generation using solid model and dynamics simulator. Furthermore, a quadruped legged robot "JROB-1" is developed as a platform for the research on perception-action coupling in intelligent behavior of robots.