In recent years, multimodal locomotion capabilities have enabled robots to maneuver in both terrestrial and aerial domains. However, most of these robots are designed only for locomotion, and few possess the manipulation capabilities required for practical tasks. By adding a manipulator, ground robots can perform manipulation, and some drones with robotic arms have demonstrated aerial manipulation. Nonetheless, such multirotors cannot be directly used for manipulation on the ground, and this configuration itself is unsuitable for air-ground hybrid locomotion. This is because their thruster-centralized structure makes it difficult to achieve both sufficient degrees of freedom (DoF) for manipulation and stable motion with contact and transformation. Therefore, in this work, we develop a new multilink multirotor with thrusters on each link and capable of contact with the environments. This robot can perform terrestrial rolling locomotion, aerial flight locomotion, and manipulation in multiple environments using joint actuation. First, we introduce a minimal configuration design of the proposed robot. We also describe a kinematic model and propose a design for each component based on this model. Second, we propose a real-time control method based on nonlinear optimization that considers contact and joint motion, which can be applied to various multirotors. Third, we propose motion strategies that include contact constraints specific to air-ground hybrid multilink multirotors, and analyze the limitations of manipulation capabilities based on multi-contact model. Finally, we demonstrate a variety of motions in both domains using the implemented prototype. To the best of our knowledge, this is the first demonstration of air-ground hybrid locomotion and manipulation by a multilink multirotor.

Log4j has become a widely adopted logging library for Java programs due to its long history and high reliability. Its widespread use is notable not only because of its maturity but also due to the complexity and depth of its features, which have made it an essential tool for many developers. However, Log4j 1.x, which reached its end of support (deprecated), poses significant security risks and has numerous deprecated features that can be exploited by attackers. Despite this, some clients may still rely on this library. We aim to understand whether clients are still using Log4j 1.x despite its official support ending. We utilized the Mining Software Repositories 2025 challenge dataset, which provides a large and representative sample of open-source software projects. We analyzed over 10,000 log entries from the Mining Software Repositories 2025 challenge dataset using the Goblin framework to identify trends in usage rates for both Log4j 1.x and Log4j-core 2.x. Specifically, our study addressed two key issues: (1) We examined the usage rates and trends for these two libraries, highlighting any notable differences or patterns in their adoption. (2) We demonstrate that projects initiated after a deprecated library has reached the end of its support lifecycle can still maintain significant popularity. These findings highlight how deprecated are still popular, with the next step being to understand the reasoning behind these adoptions.

Science Fiction Prototyping (SFP) is a method that uses science fiction to imagine future technologies and foster innovation. It is considered effective for exploring human-robot relationships and envisioning better robot designs. However, robot embodiment influences human perception, which plays a crucial role in interaction. Simply imagining future scenarios with robots through SFP may overlook these aspects. We propose an approach called Experiential Science Fiction Prototyping (ESFP), which adds a phase of experiencing the story to the traditional SFP process. To explore the effects of ESFP, we conducted a workshop with Japanese teenagers under the theme of designing a robot that contributes to a sense of “ibasho”—a Japanese concept referring to a space or relationship where one feels accepted and comfortable. ESFP unfolds in three phases: Storytelling, where participants envision future lives with robots and create stories; Experience, where they bring these stories to life through interaction with a physical robot; and Discussion, where they reflect on the story they created and experienced. The results suggested that, through the experiential phase, participants developed new ideas about interaction with robots and expanded their imagination about future relationships. Experiencing the story helped participants connect more closely with the envisioned robot interactions and inspired new reflections and expectations. This study contributes by proposing the ESFP method, detailing its implementation, and discussing its potential through a case study.

Library-based methods are known to be very effective for fast motion planning by adapting an experience retrieved from a precomputed library. This article presents CoverLib, a principled approach for constructing and utilizing such a library. CoverLib iteratively adds an experience-classifier-pair to the library, where each classifier corresponds to an adaptable region of the experience within the problem space. This iterative process is an active procedure, as it selects the next experience based on its ability to effectively cover the uncovered region. During the query phase, these classifiers are utilized to select an experience that is expected to be adaptable for a given problem. Experimental results demonstrate that CoverLib effectively mitigates the tradeoff between plannability and speed observed in global (e.g., sampling-based) and local (e.g., optimization-based) methods. As a result, it achieves both fast planning and high success rates over the problem domain. Moreover, due to its adaptation-algorithm-agnostic nature, CoverLib seamlessly integrates with various adaptation methods, including nonlinear programming-based and sampling-based algorithms.

Various robots have been developed so far; however, we face challenges in modeling the low-rigidity bodies of some robots. In particular, the deflection of the body changes during tool-use due to object grasping, resulting in significant shifts in the tool-tip position and the body’s center of gravity. Moreover, this deflection varies depending on the weight and length of the tool, making these models exceptionally complex. However, there is currently no control or learning method that takes all of these effects into account. In this study, we propose a method for constructing a neural network that describes the mutual relationship among joint angle, visual information, and tactile information from the feet. We aim to train this network using the actual robot data and utilize it for tool-tip control. Additionally, we employ Parametric Bias to capture changes in this mutual relationship caused by variations in the weight and length of tools, enabling us to understand the characteristics of the grasped tool from the current sensor information. We apply this approach to the whole-body tool-use on KXR, a low-rigidity plastic-made humanoid robot, to validate its effectiveness.

In this study, we present an implementation strategy for a robot that performs peg transfer tasks in Fundamentals of Laparoscopic Surgery (FLS) via imitation learning, aimed at the development of an autonomous robot for laparoscopic surgery. Robotic laparoscopic surgery presents two main challenges: (1) the need to manipulate forceps using ports established on the body surface as fulcrums, and (2) difficulty in perceiving depth information when working with a monocular camera that displays its images on a monitor. Especially, regarding issue (2), most prior research has assumed the availability of depth images or models of a target to be operated on. Therefore, in this study, we achieve more accurate imitation learning with only monocular images by extracting motion constraints from one exemplary motion of skilled operators, collecting data based on these constraints, and conducting imitation learning based on the collected data. We implemented an overall system using two Franka Emika Panda Robot Arms and validated its effectiveness.

We propose the method to adapt humanoids the ability to change the body structures that modular robots have by using Attach-Lock-Detachable Magnetic Couplings(ALDMag) to give the ability to detach and attach the robot body with an arm-type robot, and the system to manage the connection state of modularized body elements. Robots and we can use the ALDMag to attach and detach mechanical and electrical connections without actuators. Using xacro for writing the file of the robot model description of each module, we can construct a system that allows the robot to attach and detach modules during task operation. We demonstrated the effectiveness of the proposed method by achieving assembly experiments of a small robot with a life-size arm and experiments with environmental contacts by the small robot.

In this paper, we explain the Morphable StateNet as the StateNet with pseudo-generalized behaviors for robots with various degree-of-freedom arrangements and link lengths. Pseudo-generalization is performed by analytically calculating joint angles that satisfy the desired support conditions, focusing on link lengths and antigravity joints that contribute to motion, with constraints placed on the contact conditions between the environment and the robot body. We apply Morphable StateNet to the standing-up motion of humanoids with variable body structures and conduct evaluation experiments. We have demonstrated the usefulness of the proposed method in environments with low friction coefficients with the environment by conducting evaluations using both a simulator and an actual humanoid.

A robotic hand for social interaction should be capable of comfortable touch with humans. However, it is difficult to mount skin, tactile sensors, and driving mechanism required for human contact, especially holding hands, on a slender finger. In addition, in order to unitize the hand for easy use with any robot and maintainability, the mechanism must be contained within the small space of the fingers and palms. In this paper, we propose a human-sized five-fingered robotic hand named WARABI Hand. It is covered with multi-layored rubber skin to realize human-like soft and pleasant feel. Force sensors on each finger link detect contact with humans and adjust gripping force. We conducted experiments in which a humanoid equipped with WARABI Hand grasped forearm, held hands, and interlocked fingers with a person. The performance for object grasping was also evaluated. We demonstrated that our proposed hand is useful for interaction with humans including receiving and handing over things.

Transferring human motion skills to humanoid robots remains a significant challenge. In this study, we introduce a Wasserstein adversarial imitation learning system, allowing humanoid robots to replicate natural whole-body locomotion patterns and execute seamless transitions by mimicking human motions. First, we present a unified primitive-skeleton motion retargeting to mitigate morphological differences between arbitrary human demonstrators and humanoid robots. An adversarial critic component is integrated with Reinforcement Learning (RL) to guide the control policy to produce behaviors aligned with the data distribution of mixed reference motions. Additionally, we employ a specific Integral Probabilistic Metric (IPM), namely the Wasserstein-1 distance with a novel soft boundary constraint to stabilize the training process and prevent model collapse. Our system is evaluated on a full-sized humanoid JAXON in the simulator. The resulting control policy demonstrates a wide range of locomotion patterns, including standing, push-recovery, squat walking, humanlike straight-leg walking, and dynamic running. Notably, even in the absence of transition motions in the demonstration dataset, the robot showcases an emerging ability to transit naturally between distinct locomotion patterns as desired speed changes.

For humanoid whole body contact motions, it is important to recognize the existence of whole body contacts and the contact forces. The challenges in recognizing the existence of whole body contacts and the contact forces in life-size humanoids are: 1) the measurement part with low mechanical strength must be tolerant of high load and 2) it is difficult to model thick elastic bodies with high impact tolerance and uneven sensor placements when applied to various shapes of the whole body. This paper proposes a method of constructing a load tolerant tactile sensor by separating the loaded part from the measuring part with magnetism and protecting the measuring part inside the frame of the robot. For modeling difficulties, this paper proposes learning the relationship between the change in the detected physical quantity due to deformation of the elastic body and the contact force distribution. This paper shows through experiments that the proposed tactile sensor based on a robot frame is load tolerant enough to support the weight of a life-sized humanoid, and that it can acquire contact force distribution and the robot is able to acclimate to external forces.

A wire-driven parallel robot is a type of robotic system where multiple wires are used to control the movement of a end-effector. The wires are attached to the end-effector and anchored to fixed points on external structures. This configuration allows for the separation of actuators and end-effectors, enabling lightweight and simplified movable parts in the robot. However, its range of motion remains confined within the space formed by the wires, limiting the wire-driven capability to only within the pre-designed operational range. Here, in this study, we develop a wire-driven robot, CubiX, capable of connecting to and utilizing the environment. CubiX connects itself to the environment using up to 8 wires and drives itself by winding these wires. By integrating actuators for winding the wires into CubiX, a portable wire-driven parallel robot is realized without limitations on its workspace. Consequently, the robot can form parallel wire-driven structures by connecting wires to the environment at any operational location.

Robots generally have a structure that combines rotational joints and links in a serial fashion. On the other hand, various joint mechanisms are being utilized in practice, such as prismatic joints, closed links, and wire-driven systems. Previous research have focused on individual mechanisms, proposing methods to design robots capable of achieving given tasks by optimizing the length of links and the arrangement of the joints. In this study, we propose a method for the design optimization of robots that combine different types of joints, specifically rotational and prismatic joints. The objective is to automatically generate a robot that minimizes the number of joints and link lengths while accomplishing a desired task, by utilizing a black-box multi-objective optimization approach. This enables the simultaneous observation of a diverse range of body designs through the obtained Pareto solutions. Our findings confirm the emergence of practical and known combinations of rotational and prismatic joints, as well as the discovery of novel joint combinations.

To give the single robot system the ability to realize many behaviors and to realize tasks with shifting environments and objectives, it is necessary to abstract the robot’s body ability to the extent that they can be detected by sensors in the body so that we can plan as the problem of state transition.In this paper, to abstract the transformer robot system that performs heavy lifting and environmental attachment tasks in an aquatic and terrestrial environment, we extend the graphical representation of the robot’s body to manage joint capability and the body adaptability for the environment. To abstract the body ability, we divide the body into elements and define Connection between them at three different granularities. And using Connection, we propose the Connection Modification Feature(CMF) as the representation for changing body ability. To implement the Connection Modification Feature, we perform the abstract description and extract Connection to construct Body Ability Graph, a graph for the robot to manage its body ability. We show that it is possible to plan to manipulate and use its own Connection Modification Feature through multiple experiments by defining Normal Action that does not change body abilities and Body Ability Modifying Action that manipulate body abilities.

The complex ways in which humans utilize their bodies in sports and martial arts are remarkable, and human motion analysis is one of the most effective tools for robot body design and control. On the other hand, motion analysis is not easy, and it is difficult to measure complex body motions in detail due to the influence of numerous muscles and soft tissues, mainly ligaments. In response, various musculoskeletal simulators have been developed and applied to motion analysis and robotics. However, none of them reproduce the ligaments but only the muscles, nor do they focus on the shoulder complex, including the clavicle and scapula, which is one of the most complex parts of the body. Therefore, in this study, a detailed simulation model of the shoulder complex including ligaments is constructed. The model will mimic not only the skeletal structure and muscle arrangement but also the ligament arrangement and maximum muscle strength. Through model predictive control based on the constructed simulation, we confirmed that the ligaments contribute to joint stabilization in the first movement and that the proper distribution of maximum muscle force contributes to the equalization of the load on each muscle, demonstrating the effectiveness of this simulation.

In order to make robots more useful in a variety of environments, they need to be highly portable so that they can be transported to wherever they are needed, and highly storable so that they can be stored when not in use. We propose "on-site robotics", which uses parts procured at the location where the robot will be active, and propose a new solution to the problem of portability and storability. In this paper, as a proof of concept for on-site robotics, we describe a method for estimating the kinematic model of a robot by using inertial measurement units (IMU) sensor module on rigid links, estimating the relative orientation between modules from angular velocity, and estimating the relative position from the measurement of centrifugal force.At the end of this paper, as an evaluation for this method, we present an experiment in which a robot made up of wooden sticks reaches a target position. In this experiment, even if the combination of the links is changed, the robot is able to reach the target position again immediately after estimation, showing that it can operate even after being reassembled. Our implementation is available on https://github.com/hiroya1224/urdf_estimation_with_imus.

Muscles of the human body are composed of tiny actuators made up of myosin and actin filaments. They can exert force in various shapes such as curved or flat, under contact forces and deformations from the environment. On the other hand, muscles in musculoskeletal robots so far have faced challenges in generating force in such shapes and environments. To address this issue, we propose Patterned Structure Muscle (PSM), artificial muscles for musculoskeletal robots. PSM utilizes patterned structures with anisotropic characteristics, wire-driven mechanisms, and is made of flexible material Thermoplastic Polyurethane (TPU) using FDM 3D printing. This method enables the creation of various shapes of muscles, such as simple 1 degree-of-freedom (DOF) muscles, Multi-DOF wide area muscles, joint-covering muscles, and branched muscles. We created an upper arm structure using these muscles to demonstrate wide range of motion, lifting heavy objects, and movements through environmental contact. These experiments show that the proposed PSM is capable of operating in various shapes and environments, and is suitable for the muscles of musculoskeletal robots.

Exoskeletons for robot operation necessitate shoulders with high range of motions and high degrees of freedom to fit the operator’s shoulder girdle. These shoulder joints need high torque for force feedback on the operator. Existing exoskeletons struggle to simultaneously meet these requirements of high DOFs, wide ROM, and high torque due to spatial constraints. This study introduces an exoskeleton with a distal branching link mechanism that addresses this issue by concentrating on each link’s absolute and relative degrees of freedom. In the proposed exoskeleton, the end-effector’s absolute DOF, the forearm’s absolute DOF, and the end-effector and forearm’s relative DOF are matched between the operator and the exoskeleton. This is achieved while reducing the overall DOF by sharing the root link system’s DOF. Furthermore, by avoiding direct attachment of the operator to the exoskeleton’s shoulder, the design can accommodate the human shoulder’s high torque and high ROM. The study demonstrates that the branching exoskeleton outperforms existing link-fixed exoskeletons in terms of tracking the operator’s arms and the torque required by the exoskeleton’s joints. Utilizing this exoskeleton, we successfully maneuvered an actual humanoid robot to perform daily activities where the forearm posture is crucial.

In Japan, home robots have become popular and are widely loved. In particular, Sony’s aibo users have formed a big community, and a society where humans and robots live in harmony is becoming a reality. In order for robots to be more accepted in society, exploratory analysis and explicit description of their bonds are important. However, the scale for measuring relationships with robots has not yet been defined, so a suitable measure was created from the Friendship Function Scale to measure the relationship of friends and the Pet Awareness Scale to measure the attitude toward pets. Using the proposed scale, we investigated the role of home robots, subjective intimacy, and attitudes toward robots among 54 robot owners. The survey found that more than half of them considered robots to be pets or children. In addition, the average value of subjective intimacy was very high to indicate a very high level of attachment to the robot. Furthermore, as a result of factor analysis of the scale items, five factors were extracted: "Comfort," "Irreplaceability," "Trust from the robot," "Trust to the robot," and "Social life." At this time, there was no bias in how users felt about comfort and irreplaceability, but there was a bias in how users felt about trust. Additionally, those with the highest level of subjective intimacy felt significantly more "trusted by the robot" than those with the other level of intimacy, suggesting that recognition of trust from the robot may be an important bond-forming strategy. In the future, we will aim to form further bonds by investigating relationships with robots that play various roles, as well as in regions other than Japan.

Manipulation performance improvement is crucial for aerial robots. For aerial manipulators, the baselink position and attitude errors directly affect the precision at the end effector. To address this stability problem, fixed-body approaches such as perching on the environment using the rotor suction force are useful. Additionally, conventional arm-equipped multirotors, called rotor-concentrated manipulators, find it difficult to generate a large wrench at the end effector due to joint torque limitations. Using distributed rotors to each link, the thrust can support each link weight, decreasing the arm joints' torque. Based on this approach, rotor-distributed manipulators (RDMs) can increase feasible wrench and reachability of the end effector. This article introduces a minimal configuration of an RDM that can perch on surfaces, especially ceilings, using a part of their body. First, we design a minimal rotor-distributed arm considering the flight and end-effector performance. Second, a flight controller is proposed for this minimal RDM along with a perching controller adaptable for various types of aerial robots. Third, we propose a motion planning method based on inverse kinematics, considering specific constraints to the proposed RDMs, such as perching force. Finally, we evaluate flight and perching motions and confirm that the proposed manipulator can significantly improve the manipulation performance.

State recognition of objects and environment in robots has been conducted in various ways. In most cases, this is executed by processing point clouds, learning images with annotations, and using specialized sensors. In contrast, in this study, we propose a state recognition method that applies Visual Question Answering (VQA) in a Pre-Trained Vision-Language Model (PTVLM) trained from a large-scale dataset. By using VQA, it is possible to intuitively describe robotic state recognition in the spoken language. On the other hand, there are various possible ways to ask about the same event, and the performance of state recognition differs depending on the question. Therefore, in order to improve the performance of state recognition using VQA, we search for an appropriate combination of questions using a genetic algorithm. We show that our system can recognize not only the open/closed of a refrigerator door and the on/off of a display, but also the open/closed of a transparent door and the state of water, which have been difficult to recognize.

Humanoids operate in repeated contact and non-contact with their environment and so the motion of humanoids such as walking on uneven terrain or in a narrow space requires the accurate force and position control. Joint torque control systems are suitable for position and force control, but are prone to friction and other modeling errors. To solve this problem, methods have been proposed to realize torque control in combination with joint position control systems or by improving joint structures such as sensors and actuators, but these methods have problems such as response delay and increased weight and volume. Thus, it is difficult to achieve motion of life-sized humanoids by whole-body torque control. In this paper, we solve challenges not with one specific layer, but rather with multiple layers that complement each other. We propose a hierarchical whole-body torque control method using four layers: friction compensation based on a vibration-suppressed model, whole-body resolved acceleration control using priority, center-of-gravity acceleration control based on foot-guided control, and landing position time modification based on capture point. We verify through walking experiments that the proposed methods can control the life-sized humanoid robot driven by high-reduction ratio joints by whole-body torque control without a torque sensor or joint position control, and that it enables the robot to move and even transport an object on outdoor uneven terrain.

The range of robot activities is expanding from industries with fixed environments to diverse and changing environments, such as nursing care support and daily life support. In particular, autonomous construction of robots that are personalized for each user and task is required. Therefore, we develop an actuator module that can be reconfigured to various link configurations, can carry heavy objects using a locking mechanism, and can be easily operated by human teaching using a releasing mechanism. Given multiple target coordinates, a modular robot configuration that satisfies these coordinates and minimizes the required torque is automatically generated by Tree-structured Parzen Estimator (TPE), a type of black-box optimization. Based on the obtained results, we show that the robot can be reconfigured to perform various functions such as moving monitors and lights, serving food, and so on.

In order for a humanoid robot to balance on the movable ground, balance feedback control in response to its unpredictable movement is required. However, feedback control in response to ground movement has the following two issues, (A) Interaction between the ground dynamics and the balance control may cause vibration. (B) The balance control may rather deteriorate the stability due to the response delay. To solve these problems, this study proposes the support foot acceleration term in the walking stabilizer and gives its gain by considering the following two conditions, (A) Avoiding steady-state vibration in a two-mass linear inverted pendulum model on an arbitrary ground, and (B) reducing the influence of inertial forces resulting from the delay of ZMP feedback. Experiments with a life-size humanoid JAXON verified the steady-state vibration phenomenon and improved the stability of acceleration and deceleration when boarding the Two-Wheeled Scooter.

We need to build robot systems that can operate in multiple environments, including underwater. For a robot to be waterproof, it needs to be covered all over its body and have a waterproof structure, but this is expensive to produce and consumes hardware resources such as weight. In this study, we propose a method of constructing a robot system in which paraffin, which can change its phase between solid and liquid at different temperatures, is cloaked on the robot's body surface, and grease is inserted into the robot's body to give it an acquired the waterproof shell for waterproofing purposes. We have studied an automated method of the waterproof shell application system by combining a small spider-shaped multi-legged robot that can measure the temperature of the environment as a robot to which we apply the waterproof shell and a life-size arm robot that performs the operation applying the waterproof shell to the small robot as a system. Through the addition of the waterproof shell to the spider-shaped robot and the removing the shell in unnecessary areas by controlling temperature, and the realization of operation in land and water environments, we used the proposed robot system to add acquired functions to operate underwater robots, thereby expanding the supported area of robots which have many DOFs and sensors from conventional ground to underwater.

Imitation learning has been actively studied in recent years. In particular, skill acquisition by a robot with a fixed body, whose root link position and posture and camera angle of view do not change, has been realized in many cases. On the other hand, imitation of the behavior of robots with floating links, such as humanoid robots, is still a difficult task. In this study, we develop an imitation learning system using a biped robot with a floating link. There are two main problems in developing such a system. The first is a teleoperation device for humanoids, and the second is a control system that can withstand heavy workloads and long-term data collection. For the first point, we use the whole body control device TABLIS. It can control not only the arms but also the legs and can perform bilateral control with the robot. By connecting this TABLIS with the high-power humanoid robot JAXON, we construct a control system for imi-tation learning. For the second point, we will build a system that can collect long-term data based on posture optimization, and can simultaneously move the robot's limbs. We combine high-cycle posture generation with posture optimization methods, including whole-body joint torque minimization and contact force optimization. We designed an integrated system with the above two features to achieve various tasks through imitation learning. Finally, we demonstrate the effectiveness of this system by experiments of manipulating flexible fabrics such that not only the hands but also the head and waist move simultaneously, manipulating objects using legs characteristic of humanoids, and lifting heavy objects that require large forces.

Robot hands that imitate the shape of the human body have been actively studied, and various materials and mechanisms have been proposed to imitate the human body. Although the use of soft materials is advantageous in that it can imitate the characteristics of the human body's epidermis, it increases the number of parts and makes assembly difficult in order to perform complex movements. In this study, we propose a skin-skeleton integrated robot hand that has 15 degrees of freedom and consists of four parts. The developed robotic hand is mostly composed of a single flexible part produced by a 3D printer, and while it can be easily assembled, it can perform adduction, flexion, and opposition of the thumb, as well as flexion of four fingers.

It is important for daily life support robots to detect changes in their environment and perform tasks. In the field of anomaly detection in computer vision, probabilistic and deep learning methods have been used to calculate the image distance. These methods calculate distances by focusing on image pixels. In contrast, this study aims to detect semantic changes in the daily life environment using the current development of large-scale vision-language models. Using its Visual Question Answering (VQA) model, we propose a method to detect semantic changes by applying multiple questions to a reference image and a current image and obtaining answers in the form of sentences. Unlike deep learning-based methods in anomaly detection, this method does not require any training or fine-tuning, is not affected by noise, and is sensitive to semantic state changes in the real world. In our experiments, we demonstrated the effectiveness of this method by applying it to a patrol task in a real-life environment using a mobile robot, Fetch Mobile Manipulator. In the future, it may be possible to add explanatory power to changes in the daily life environment through spoken language.

In order for a humanoid robot to traverse uneven terrain without falling over, the robot must control its landing position appropriately. To determine the landing position, there are two difficulties in terrain recognition and leg motion control. In terrain recognition, it is difficult to recognize and avoid terrain such as steps and obstacles that cannot be landed on in real-time. In leg motion control, it is necessary to land at appropriate positions and times to control the CoG trajectory while limiting the velocity of the swing-leg to suppress the landing impact. For solving these problems, we propose a recognition and walking control system on uneven terrain. In terrain recognition, we improved the recognition accuracy while satisfying real-time performance by using a CNN that learns the relationship between the foot and the geometric information of the surrounding terrain. In the leg motion control, landing impact was reduced by modifying the landing position under not only (1) terrain constraint and (2) robot stability constraint, but also (3) leg velocity constraint. We verified the effectiveness of the proposed system through uneven terrain walking and push recovery experiments using the actual robot.

In this paper, we focus on the kangaroo, which has powerful legs capable of jumping and a soft and strong tail. To incorporate these unique structure into a robot for utilization, we propose a design method that takes into account both the feasibility as a robot and the kangaroo-mimetic structure. Based on the kangaroo's musculoskeletal structure, we determine the structure of the robot that enables it to jump by analyzing the muscle arrangement and prior verification in simulation. Also, to realize a tail capable of body support, we use an articulated, elastic structure as a tail. In order to achieve both softness and high power output, the robot is driven by a direct-drive, high-power wire-winding mechanism, and weight of legs and the tail is reduced by placing motors in the torso. The developed kangaroo robot can jump with its hind legs, moving its tail, and supporting its body using its hind legs and tail.

Eating with others enriches our lives and has many positive effects. However, numerous people are forced to eat alone in the current COVID-19 pandemic situation. We think that robots that exist with bodies and can be interacted with in real time can be good mealtime partners. In this study, we developed a meal partner robot platform called Mamoru’21, which can share eating behavior. We experimented to evaluate Mamoru’21. The results showed that Mamoru’21 could make eating more enjoyable than a conventional communication robot. We also investigated the preferable appearance of a meal partner robot and confirmed that Mamoru’21 met the requirements.

In this study, we examined scene selection methods and emotion-based descriptions for a robot’s daily diary. We proposed a scene selection method and an emotion description method that take into account semantic and affective information, and created several types of diaries. Experiments were conducted to examine the change in sentiment values and preference of each diary, and it was found that the robot’s feelings and impressions changed more from date to date when scenes were selected using the affective captions. Furthermore, we found that the robot’s emotion generally improves the preference of the robot’s diary regardless of the scene it describes. However, presenting negative or mixed emotions at once may decrease the preference of the diary or reduce the robot’s robot-likeness, and thus the method of presenting emotions still needs further investigation.

Holding hands can give people a sense of security. In this study, we developed a five-fingered robotic hand that can hold hands with a person and a guidance system that uses the developed hand to hold hands with them. In this system, a robot remembers the location where people have taught it and guides people to that point. We conducted an experiment to evaluate the sense of security in guidance with hand-holding compared with guidance without hand-holding. Participants watched the videos on these two conditions and answered a questionnaire. The results confirmed that holding hands can lead to a sense of security.

The humanoid robot that can transform itself into a form according to its purpose requires whole-body motions with complex contact state transitions such as recovery from a fall and transition to the target form. To make the robot behavior in simulations closer to that in the real world for planning complex target trajectories, we need a platform that can measure the body stiffness during the motion and verify its application without being damaged by repeated motions that are prone to tipping over. In this study, we propose a small, inexpensive, and robust humanoid-vehicle transformer platform with redundant sensors and a low rigidity multi degree-of-freedom body and observe the effects of body deflection and internal forces during whole-body posture transition. By comparing the results obtained from experiments in several environments with different friction and from the simulator using a rigid body model, we were able to verify the influence of body flexibility on whole-body motion and the relationship between deflection and wrench observed by redundant sensors and movement failure.

In recent years, an increasing number of research works have been focusing on the manipulation by aerial robots. Previous works using aerial robots with robotic arms have two problems: underactuation and external disturbances. We propose the fully-actuated control method and motion strategy using contact with the environment to solve these problems, along with the mechanical approach required. First, each propeller's 1 degree-of-freedom (DoF) thrust vectoring units are applied to enable fully-actuated flight control. In order to obtain the desired thrust and vectoring angle inputs for aerial manipulation satisfying hardware limits, we developed a fully-actuated control method using non-linear optimization. Second, we propose a manipulation motion strategy that treats the multilink robot body as a fixed manipulator by making contact with the environment. The contact mechanism attached to the link end is developed to maintain contact and resist external disturbances. In a real machine experiment, the robot successfully opened the door while in contact with the wall, demonstrating the feasibility of the proposed methods.

In the literature on object grasping, the robot often determines the grasp point and posture from visual information. They predict the grasping point uniquely from the object's shape characteristics. However, as a practical matter, there are cases where there are constraints on grasp point due to the object states, the limitation of the robot's hardware and the surrounding environment. In this study, we propose a neural network that can easily constrain the input. It determines the grasp pose from visual information and outputs the grasp success probability. The grasp pose is modified using backpropagation to increase the success rate of the grasp. As for the target object, we deal with some dirty tableware scattered on the table. We have developed a system that autonomously collects supervised data so that the robot can learn by itself whether it has succeeded in a grasp attempt. Finally, the robot can grasp an object which avoids dirty parts and find the suboptimal grasp pose.

While the musculoskeletal humanoid has various biomimetic benefits, its complex modeling is difficult, and many learning control methods have been developed. However, for the actual robot, the hysteresis of its joint angle tracking is still an obstacle, and realizing target posture quickly and accurately has been difficult. Therefore, we develop a feedback control method considering the hysteresis. To solve the problem in feedback controls caused by the closed-link structure of the musculoskeletal body, we update a neural network representing the relationship between the error of joint angles and the change in target muscle lengths online, and realize target joint angles accurately in a few trials. We compare the performance of several configurations with various network structures and loss definitions, and verify the effectiveness of this study on an actual musculoskeletal humanoid, Musashi.

In this study, seated walk, a movement of walking while sitting on a chair with casters, is realized on a musculoskeletal humanoid from human teaching. The body is balanced by using buttock-contact sensors implemented on the planar interskeletal structure of the human mimetic musculoskeletal robot. Also, we develop a constrained teaching method in which one-dimensional control command, its transition, and a transition condition are described for each state in advance, and a threshold value for each transition condition such as joint angles and foot contact sensor values is determined based on human teaching. Complex behaviors can be easily generated from simple inputs. In the musculoskeletal humanoid MusashiOLegs, forward, backward, and rotational movements of seated walk are realized.

The musculoskeletal humanoid is difficult to modelize due to the flexibility and redundancy of its body, whose state can change over time, and so balance control of its legs is challenging. There are some cases where ordinary PID controls may cause instability. In this study, to solve these problems, we propose a method of learning a correlation model among the joint angle, muscle tension, and muscle length of the ankle and the zero moment point to perform balance control. In addition, information on the changing body state is embedded in the model using parametric bias, and the model estimates and adapts to the current body state by learning this information online. This makes it possible to adapt to changes in upper body posture that are not directly taken into account in the model, since it is difficult to learn the complete dynamics of the whole body considering the amount of data and computation. The model can also adapt to changes in body state, such as the change in footwear and change in the joint origin due to recalibration. The effectiveness of this method is verified by a simulation and by using an actual musculoskeletal humanoid, Musashi.

There is a way to utilize humanoid robots to mimic human behavior by taking advantage of their human-like proportions. In general, motion capture is used; in this case, the posture of the body links can be taken. However, this method does not provide detailed information on the contact state, which is important for actions that involve contact with objects. In this study, we focused on the foot, which has not been paid much attention among the parts where contact and manipulation with objects are important, and developed a device to measure the contact pressure distribution at the outer edge of the sole. We proposed an index, SS-COP, which simply reflects the contact on the curved surface of the sole for this device and a robot foot with lateral force sensation and realized a behavior that imitates the foot condition of a humanoid robot by using this index.

When robots walk on uneven terrain, trajectory planning should take into account both the whole-body dy-namics and the ground geometry simultaneously. In uneven terrain environments, there are only a limited number of places where the robot is able to make stable contact with the ground without its feet wobbling or slipping because of the intricate round geometry. In such environments, the optional landing position and time to maintain the robot's balance and stable foot contact are not obvious and computationally expensive. In this study, we propose a robust walking system that integrates environment recognition using steppable regions and walking control for a humanoid robot to walk on uneven terrain. In this paper, a steppable region is defined as a two-dimensional convex hull that represents a region where a robot is capable of landing. We propose a method to compute the steppable region quickly by 2.SD projection of the environment points and spatial filtering. In this system, the walking controller integrates the steppable region with the Capture Region to modify the landing position from a two-dimensional geometric calculation. In addition, to cope with the environment recognition error, we have introduced a trajectory generation that allows the feet to penetrate the ground and hybrid control of position and torque. We verified the effectiveness of the proposed system through experiments in which a life-size humanoid robot walked on uneven terrain and recovered when pushed.

Humanoid robots are versatile platforms with the potential for multiple locomotion skills. However, this contact-switched system with only two contact feet is fragile to keep balance in many scenarios. Inspired by birds combining legs and wings, we propose the novel hybrid locomotion behavior for the humanoid robots with the aid of a thruster suit. To fully leverage their agility while guaranteeing efficient computation, we combine the neural controller based on reinforcement learning to handle the complexity of the highly non-linear system and the optimization-based controller to explicitly handle the constraint conditions of the safety-critical thruster module. Our learning framework is demonstrated on several thruster-aided humanoid platforms with hybrid walking and even dynamic locomotion skills. To our best knowledge, it is the first work that, 1. demonstrates agile hybrid whole-body locomotion skills on the thruster-aided humanoid robot; 2. achieves hybrid locomotion under the reinforcement learning settings.

Legged robots with high locomotive performance have been extensively studied, and various leg structures have been proposed. Especially, a leg structure that can achieve both continuous and high jumps is advantageous for moving around in a three-dimensional environment. In this study, we propose a parallel wire-driven leg structure, which has one DoF of linear motion and two DoFs of rotation and is controlled by six wires, as a structure that can achieve both continuous jumping and high jumping. The proposed structure can simultaneously achieve high controllability on each DoF, long acceleration distance and high power required for jumping. In order to verify the jumping performance of the parallel wire-driven leg structure, we have developed a parallel wire-driven monopedal robot, RAMIEL. RAMIEL is equipped with quasi-direct drive, high power wire winding mechanisms and a lightweight leg, and can achieve a maximum jumping height of 1.6 m and a maximum of seven continuous jumps.

For robots to move in the real world, they must first correctly understand the state of its own body and the tools that it holds. In this research, we propose DIJE, an algorithm to estimate the image Jacobian for every pixel. It is based on an optical flow calculation and a simplified Kalman Filter that can be efficiently run on the whole image in real time. It does not rely on markers nor knowledge of the robotic structure. We use the DIJE in a self-recognition process which can robustly distinguish between movement by the robot and by external entities, even when the motion overlaps. We also propose a visual servoing controller based on DIJE, which can learn to control the robot's body to conduct reaching movements or bimanual tool-tip control. The proposed algorithms were implemented on a physical musculoskeletal robot and its performance was verified. We believe that such global estimation of the visuomotor policy has the potential to be extended into a more general framework for manipulation.

Self-healing function is a promising approach for damage management of high-load robot applications such as legged robots. Although the function is getting major in soft robotics, its application to life-sized "stiff" robots is of relatively minor interest. Although the authors have devised several self-healing tensile modules for tendon-driven robots, the design guideline to satisfy the large load endurance and large stroke is still unclear. The paper focuses on the parametric design for unleaked liquid-assisted healing of low melting point alloy structure. The method was validated with a benchtop module test. Moreover, the module enabled tendon-driven monopod testbed to perform squat motion three times after the landing impact fracture and the self-healing sequence, which was never accomplished.

Precise aerial manipulation is important for multirotor robots. For multirotors equipped with arms, the root pose error due to the floating body affects the precision at the end effector. Fixed-root approaches, such as perching on surfaces using the rotor suction force, are useful to address this problem. Furthermore, it is difficult for arm-equipped multirotors to generate large wrenches at the end effector owing to joint torque limitations. For multilink aerial robots with rotors distributed to each link, the thrust of rotors can produce large torques. Therefore, such multirotor robots can generate comparatively large wrenches at the end effector. In this paper, we introduce a rotor-distributed multilink robot that can perch on surfaces. First, we designed a root footplate and arm module for a multilink aerial robot. During perching, the joint between these two links can be passive to prevent peeling. Second, we propose a quadratic programming (QP) based controller to calculate the desired thrust for perching motion, considering the static friction and zero moment point (ZMP) conditions on the footplate. Finally, we conducted root-body perching motion tests. The manipulations of the multilink aerial robot during perching become more accurate than those during flight because the root position adheres to the environment.

Quadrupedal robots are skillful at locomotion tasks while lacking manipulation skills, not to mention dexterous manipulation abilities. Inspired by the animal behavior and the duality between multi-legged locomotion and multi-fingered manipulation, we showcase a circus ball challenge on a quadrupedal robot, ANYmal. We employ a model-free reinforcement learning approach to train a deep policy that enables the robot to balance and manipulate a light-weight ball robustly using its limbs without any contact measurement sensor. The policy is trained in the simulation, in which we randomize many physical properties with additive noise and inject random disturbance force during manipulation, and achieves zero-shot deployment on the real robot without any adjustment. In the hardware experiments, dynamic performance is achieved with a maximum rotation speed of 15 °/s, and robust recovery is showcased under external poking. To our best knowledge, it is the first work that demonstrates the dexterous dynamic manipulation on a real quadrupedal robot.

The purpose of this paper is the robotic hanging manipulation of an object of various shapes that is not limited to a specific category. To achieve this, we propose a method that allows the estimator to learn many different shapes with hanging points without any manual annotation. A random shape generator using GAN solves the limitation of the number of 3D models and can handle objects of various shapes. In addition, hanging is repeated in the dynamics simulation, and hanging points are automatically generated. A large amount of training data is generated by rendering random-textured objects with hanging points in the random simulation environment. A deep neural network trained with these data was able to estimate hanging points of an unknown category object in the real world and achieved hanging manipulation by a robot.

We have implemented a force-based operational space controller on a physical musculoskeletal humanoid robot arm. The controller calculates muscle activations based on a biomimetic Hill-type muscle model. We propose a method to include the joint torque nullspace in the optimization process, which enables the robot to exploit the nullspace to gradually lower its overall muscle activation. We have verified in experiments that it can react compliantly to external disturbances while retaining its operational space task.

There is increasing demand for robots that provide a mode of transportation in environments in which people coexist. However, conventional mobile robots, especially those carrying people, are limited in terms of their environments and tasks. For example, wheeled robots are limited to moving on flat ground. Walking robots are limited to entertainment and so on. The originality of the present paper is the development of a novel movement mechanism for a mobility apparatus that can handle various daily use scenes. We first clarify functional requirements for daily use. We then propose a transformable human-carrying wheel–leg mobility. The leg is based on a serial link and compactly uses a parallel link mechanism such that the motor is placed on top, which improves responsiveness when having a high payload and compact shape. By conducting simulations and experiments with the prototype, it is confirmed that expected operations can be realized. In particular, it is confirmed that the weight of the leg tips is reduced, such that the shaking of the waist in the direction of travel during the ascent of a step is reduced by 68%. The above results reveal that the prototype can be used in daily life.

In this study, we propose a predictive model composed of a recurrent neural network including parametric bias and stochastic elements, and an environmentally adaptive robot control method including variance minimization using the model. Robots which have flexible bodies or whose states can only be partially observed are difficult to modelize, and their predictive models often have stochastic behaviors. In addition, the physical state of the robot and the surrounding environment change sequentially, and so the predictive model can change online. Therefore, in this study, we construct a learning-based stochastic predictive model implemented in a neural network embedded with such information from the experience of the robot, and develop a control method for the robot to avoid unstable motion with large variance while adapting to the current environment. This method is verified through a mobile robot in simulation and to the actual robot Fetch.

Musculoskeletal humanoids have various biomimetic advantages, and the redundant muscle arrangement allowing for variable stiffness control is one of the most important. In this study, we focus on one feature of the redundancy, which enables the humanoid to keep moving even if one of its muscles breaks, an advantage that has not been dealt with in many studies. In order to make the most of this advantage, the design of muscle arrangement is optimized by considering the maximization of minimum available torque that can be exerted when one muscle breaks. This method is applied to the elbow of a musculoskeletal humanoid Musashi with simulations, the design policy is extracted from the optimization results, and its effectiveness is confirmed with the actual robot.

We report the results of our study on whether taking a walk with a child-like robot that a person takes care of can generate interaction with the surrounding people whom the person has never met before. As the number of single-person households increases, it is expected that more people will live with not only robots that can be useful for people but also robots that people can take care of. Our study is important because we explored the possibilities of such a robot to create interaction between people in the community, which has been lost in recent years. In this paper, we designed the behavior of a robot that follows a person and learns about the scenery while walking together and implemented it using Pepper. Then, the first author walked around the university building with the robot and observed the initial reactions of the surrounding people. As we expected, the surrounding people interacted with the first author by talking to the robot as if it were a child and helping the robot. This paper contributes to taking a step forward a new research theme ‘taking a walk with a robot and a person’ by focusing on the important role of robots in the future relationship with a person: care-receiving from a person and communication mediation between a person and others.

We developed a method to enable a humanoid robot to carry stacked boxes. In order to transport objects efficiently, it is necessary to carry multiple objects at the same time, but in previous studies, humanoid robots have only been able to carry a single object. When a humanoid robot carries stacked boxes, the robot drops boxes when the positional relationship between un-grasped boxes changes. The causes for dropping the boxes can be divided into sudden changes attributed to robot making turns or losing balance, and the accumulation of small changes that occur because of the impact of landing while walking. We propose a method that prevents sudden changes in the stacked boxes by smoothing the hand trajectory and modifying the misalignment by tilting or shaking the entire stack. We verify the effectiveness of proposed method for enabling a humanoid robot to carry stacked boxes through experiments using a simulator and an actual robot.

In this paper, we aim to recognize function points of category-agnostic objects and perform object manipulation. To recognize function points of various shapes, it is necessary to train with a large amount of training data. Also, it is necessary to take into account not only visual information but also physics and interaction between objects. To solve these problems, we are working on the automatic generation of training data by detecting function points from a physical simulation. In the proposed system, we add simulation with target task operation and goal state, which allows a robot to acquire the target function point recognizer. We also use GAN to generate various random shapes and render them with random domains, and train Deep Neural Networks on these data. These enable the robot to recognize function points of unseen objects in the real world and realize manipulation.

Due to the recent lifestyle changes, instances of people eating alone have been increasing. We think robots can be good meal partners without having to risk disease transmission. Furthermore, people are able to eat with robots without worrying about mealtimes. In this study, we determine who are more likely to accept robots as eating partners and compare eating with a single robot to eating with multiple robots. The results revealed that people who have vast experience in interacting with robots and those who have relatively few opportunities to eat alone felt better about eating with robots, whereas those who have numerous opportunities to eat alone enjoyed eating with multiple robots.

The flexible under-actuated musculoskeletal hand is superior in its adaptability and impact resistance. On the other hand, since the relationship between sensors and actuators cannot be uniquely determined, almost all its controls are based on feedforward controls. When grasping and using a tool, the contact state of the hand gradually changes due to the inertia of the tool or impact of action, and the initial contact state is hardly kept. In this study, we propose a system that trains the predictive network of sensor state transition using the actual robot sensor information, and keeps the initial contact state by a feedback control using the network. We conduct experiments of hammer hitting, vacuuming, and brooming, and verify the effectiveness of this study.

Carrying payloads in air is a major mission for multirotor aerial robot. However, the presence of payloads on multirotor aerial robot has a risk of degrading the performance of the flight controller. This concern becomes obvious especially when carrying objects not securely attached to the body or performing aerial manipulation. Therefore, controller with the ability to adapt itself to the effects of payloads on flight stability is needed. This paper proposes a novel nonlinear multiple-input and multiple-output (MIMO) model reference adaptive control (MRAC) system for attitude control of multirotor aerial robots which can dynamically compensate change in the position of center of gravity and inertia caused by payloads. Stability and robustness of the controller are experimentally confirmed in quadrotor and transformable multirotor, and experiments modeling practical applications are conducted for each aerial robot system, proving the utility of the controller.

Stable flight control under ceilings is difficult for multirotor Unmanned Aerial Vehicles (UAVs). The wake interaction between rotors and upper walls, called the "ceiling effect", causes an increase of rotor thrust. As a result of the thrust increase, multi-rotors are drawn upward abruptly and collide with ceilings. In previous work, several thrust models of the ceiling effect have been proposed for stable flight under ceilings, assuming that the airflow around rotors is in steady states. However, the airflow around rotors in vertical flight is not in steady states and each thrust model in previous work is skillfully determined based on large amounts of precise experimental data. In this paper, we introduce an aerodynamics-based thrust model and a stable control method under ceilings. This model is derived from the momentum theory and the relationship between vertical climbing/descending rates of rotors and an induced velocity. To confirm our proposed model, we collect thrust data at various vertical rates in flight. In addition, we use only onboard sensors to estimate selfstate for structural inspections. Consequently, we reveal that the proposed model is consistent with the experimental results. Based on an aerodynamic model, we need not collect large amounts of precise experimental data to realize stable flight. Furthermore, the vertical flight tests under ceilings demonstrate that our in-unsteady-state-model-based controller outperforms the conventional steady-state ones.

It is necessary to use not only foot but also hand, knee and other body parts to support body weight for locomotion in uneven terrain. Such multi-contact motion planning is an important research topic including lots of previous works; however, a problem of computational speed of planning is still remaining. In this paper, we propose a learning-based algorithm to speed up the planning. The algorithm reduces replanning of contact states by learning an evaluation function of key pose to reach goal. We investigated the learning performance by comparing three neural network configurations and two activation function. This research aims at achieving robust robotics system in unknown environments.

Regrasping is one of the most common and important manipulation skills used in our daily life. However, aerial regrasping has not been seriously investigated yet, since most of the aerial manipulator lacks dexterous manipulation abilities except for the basic pick-and-place. In this paper, we focus on pivoting a long box, which is one of the most classical problems among regrasping researches, using a transformable multilink aerial robot. First, we improve our previous controller by compensating for the external wrench. Second, we optimize the joints configuration of our transformable multilink drone for stable grasping form under the constraints of thrust force and joints effort. Third, we sequentially optimize the grasping force in the pivoting process. The optimization goal is to generate continous grasping force whilst maximizing the friction force in case of the downwash, which would influence the grasped object and is difficult to model. Fourth, we develop the impedance controller in joint space and admittance controller in task space. As far as we know, it is the first research to achieve extrinsic contact-aware regrasping task on aerial robots.

Humanoid robots are expected to play a big role at distress sites and disaster sites. There is a variety of multi-contact locomotion forms other than bipedal walking such as crawling through tightly, getting on the rubble by using its knees and elbows, or jumping in and rolling over the obstacles. If such multi-contact locomotion forms can be achieved, robots can reach environments that are currently unreachable, and be able to conduct tasks required at the environments. To achieve this, it is required for robots to bring various parts of its body into contact with the environment like a human. However, it is difficult for parts without 6-axis force sensors to achieve the target force while adapting to the environment against impact force. It is also difficult to measure contact wrenches without 6-axis force sensors. In this paper, by allowing the error of the contact state, we propose online system for realizing dynamic motion which impact force occurs on the parts of the whole body by contact to the environment. In the proposed system, we applied the current-based torque control for joints to make the whole body parts of the robot adapt to the environment, and we modified motion in real time to stabilize zmp by estimating contact wrenches at the contact positions where force sensors are not mounted. In addition, at the motion planning, we generated more feasible motions for a robot applying torque control by using evolutionary computation which advances the search with the behavior of torque control. We demonstrate that the proposed system is effective by showing experimental results of sitting posture locomotion using a JAXON robot in which impact force occur on the back of the thighs which have no force sensors.

In this work, we propose a system for humanoid robot fast motions. When a humanoid robot performs a motion such as a tennis forehand stroke motion, a whole-body fast motion in reaction to visual information is required. There are three problems to tackle. (1) Motion is desired to be quick. (2) Real-time visual processing considering visual noises is needed. (3) Real-time joint angle modification with balance keeping is needed. To solve the problem (1), we used an offline optimization system to enhance the motion speed. To solve the problem (2), we implement a ball trajectory prediction algorithm using the Extended Kalman Filter (EKF). To solve the trade-off between (1) and (3), we propose an offline optimization condition with an estimated balance margin. By using these methods, we achieved a non-step tennis forehand stroke motion with a humanoid robot by predicting a ball's trajectory with stereo cameras on the robot's head.

The musculoskeletal humanoid has various biomimetic benefits, and the redundant muscle arrangement is one of its most important characteristics. This redundancy can achieve fail-safe redundant actuation and variable stiffness control. However, there is a problem that the maximum joint angle velocity is limited by the slowest muscle among the redundant muscles. In this study, we propose two methods that can exceed the limited maximum joint angle velocity, and verify the effectiveness with actual robot experiments.

The musculoskeletal humanoid has various biomimetic benefits, and it is important that we can embed and evaluate human reflexes in the actual robot. Although stretch reflex has been implemented in lower limbs of musculoskeletal humanoids, we apply it to the upper limb to discover its useful applications. We consider the implementation of stretch reflex in the actual robot, its active/passive applications, and the change in behavior according to the difference of parameters.

The purpose of this study is to make life-sized humanoid robots acquire tool manipulation skills that require complicated force adjustment. The difficulty in acquisition of tool manipulation skills comes from the hardship in physical modeling. Recent research have revealed that deep reinforcement learning (DRL), a model-free approach, performs superior in such tasks. However, DRL in general has a drawback in sample efficiency, and this becomes critical in robot learning especially in life-sized humanoid robots. In this study, we propose an integrated system incorporating DRL method and active learning. Our method also leverages a variety of previous studies on life-sized humanoid robots to overcome the sample efficiency issue. We demonstrated the effectiveness of our proposed system through a hacksaw skill acquisition and a Japanese planer (Kanna) skill acquisition by a life-sized humanoid robot.

For mitigating joint impact torques, researchers have reduced joint stiffness by series elastic actuators, reflected inertia by low gear ratios, and friction torque from drive-trains. However, these impact mitigation methods may impair the control performance of contact forces or may increase motor and robot mass. This paper proposes a design method for achieving a balance between impact mitigation performance and force control fidelity. We introduce an inertia-to-square-torque ratio as a new index for integrating the parameters of torque generation (motor continuous torque limits, gear ratios, etc.) and the parameters of impact mitigation (joint stiffness, reflected inertia, etc.). In the process, we make a hypothesis that a motor mass is negatively correlated with the ratio. Based on the hypothesis, we calculate a joint breakdown region of impact torques, joint stiffnesses, and motor masses. Finally, we decide the drive-train specifications of JAXON3-P and demonstrate that the proposed method provides high impact mitigation and force control capabilities through several experiments including the jumping motion of 0.3 m COG height.

We consider the problem of acquiring mechanical knowledge through visual cues to help robots use objects in new situations. In this work, we propose a novel deep learning approach that allows a robot to acquire mechanical knowledge from 3D point clouds. This presents two main challenges. The first challenge is that a robot needs to infer novel objects' functions from its experience. Secondly, the robot should also need to know how to manipulate these novel objects. To solve these problems, we present a two-branch deep neural network. The first branch detects function parts from the point clouds while the second branch predicts offset poses. Fusing the results from these two branches, our approach can not only detect what functions the novel objects may have but also generate key object states which can be used to guide a robot to manipulate these objects. We show that even though most of the training samples are synthetic data, our model still learns useful features and outputs proper results. Finally, we evaluate our approach on a real robot to run a series of tasks. The experimental results show that our approach has the capability to transfer mechanical knowledge in new situations.

Juggling manipulation is one of difficult manipulation to acquire since some of such manipulation is unstable and also its physical model is unknown due to the complex non-prehensile manipulation. To acquire these unstable unknown-dynamics juggling manipulation, we propose a method for designing the predictive model of manipulation with a deep neural network, and a real-time optimal control law with some robustness and adaptability using backpropagation of the network. In this study, we apply this method to diabolo orientation stabilization, which is one of unstable unknown-dynamics juggling manipulation. We verify the effectiveness of the proposed method by comparing with basic controllers such as P Controller or PID Controller, and also check the adaptability of the proposed controller by some experiments with a real life-sized humanoid robot.

This paper describes basic implementation of an embedded controller board based on a hybrid architecture equipped with an Intel FPGA SoC and an NVIDIA GPU SoC. Embedded distributed network involving motor-drivers or other embedded boards is constructed with low-latency optical transmission link. The central controller for high-level motion planning is connected via Gigabit Ethernet. The controller board with the hybrid architecture provides lower-latency feedback control performance. Computing performance of the FPGA SoC, the GPU SoC, and the central controller is evaluated by computation time of matrix multiplication. Then, the total feedback latency is estimated to show the performance of the hybrid architecture.

In this research, we have taken a biomimetic approach to the control of musculoskeletal humanoids. A controller was designed based on the motor directional tuning phenomenon seen in the motor cortex of primates. Despite the simple implementation of the control scheme, complex coordinated movements such as reaching for target objects with its upper body was achieved, and is demonstrated in the accompanying video. The controller does not require an internal model, and instead constantly observes its body in relation to the external world to update motor commands. We claim that such an embodied approach to the control of musculoskeletal robots will be able to effectively take advantage of their complex bodies to achieve motion.

Eating with others enhances our dining experience, improves socialization, and has some health benefits. Although many people do not want to eat alone, there is an increase in the number of people who eat alone in Japan due to difficulty in matching mealtimes and places with others.In this paper, we develop a mixed reality (MR) system for coeating with a robot. In this system, a robot and a MR headset are connected enabling users to observe a robot putting food image into its mouth, as if eating. We conducted an experiment to evaluate the developed system with users that are at least 13 years old. Experimental results show that the users enjoyed their meal and felt more delicious when the robot ate with them than when the robot only talked without eating. Furthermore, they eat more when a robot eats, suggesting that a robot could influence people's eating behavior.

Recent progress of deep learning improved the capability of a robot to find a proper grasp of a novel object for different grasp modalities (e.g., pinch and suction). While these previous studies consider multiple modalities separately, several studies develop multi-modal grippers that can achieve simultaneous pinch and suction grasp (multi-modal grasp fusion) for more capable and stable object manipulation. However, the previous studies with these grippers restrict the situations: simple object geometry and uncluttered environments. To overcome these difficulties, we propose a system that consists of: 1) object-class-agnostic grasp modality detection; 2) object-class-agnostic instance segmentation; and 3) grasp template matching for different modalities. The key idea of our work is the introduction of instance segmentation to fuse multiple modalities regarding each instance eluding a grasp of multiple objects at once. In the experiments, we evaluated the proposed system on the real-world picking task in clutter. The experimental results show that the effectiveness of modality detection, instance segmentation, and the integrated system as a whole.

External wrench estimation is very helpful for aerial exploration and manipulation tasks. During the exploration, there might be unseen obstacles to cause dangerous collisions. The estimation of the external force and torque is also beneficial in aerial manipulation tasks. In this paper, we present a framework of estimating the external wrench for the aerial multilink robot based on the onboard inertial measurement unit (IMU) sensors, joints state and robot dynamic models. Compared to the conventional multirotor robot, the center of mass (CoM) is always changing when the robot transforms. The sensor could not be attached to CoM to observe the acceleration data. Consequently, we present a novel method by applying a distributed IMU system to estimate the CoM linear and angular accelerations for the external wrench estimation. With the help of the robot model, the position of the contact point could be estimated, which is useful in exploring tasks to safely interact with the physical world. We design the contact-aided navigation strategy and computationally efficient motion primitives library to help our robot react to the unexpected collision. We experimentally validate our framework with a two-dimensional multilink aerial robot to show the results of external wrench estimator and its further applications2.2Experiment video: https://youtu.be/R-WDReLnWWI

We present joint learning of instance and semantic segmentation for visible and occluded region masks. Sharing the feature extractor with instance occlusion segmentation, we introduce semantic occlusion segmentation into the instance segmentation model. This joint learning fuses the instance-and image-level reasoning of the mask prediction on the different segmentation tasks, which was missing in the previous work of learning instance segmentation only (instance-only). In the experiments, we evaluated the proposed joint learning comparing the instance-only learning on the test dataset. We also applied the joint learning model to 2 different types of robotic pick-and-place tasks (random and target picking) and evaluated its effectiveness to achieve real-world robotic tasks.

We present a novel transformable multilinked aerial robot which consists of link modules with 1 DoF thrust vectoring mechanism. Commonly used UAV is underactuated due to its simplicity and high flight duration, but can not control the position and orientation independently. To overcome this problem, fully actuated multirotor aerial robots have been developed. In our previous work we developed fully actuated multilinked aerial robot which can transform in the air. However, the transformation range was limited because of a singularity problem. In this paper we propose a new design of link module with a tilted rotor and 1 DoF thrust vectoring joint which enables to avoid singularity forms and keep the flight stable during transformation. We describe modeling and control for the fully actuated multilinked multirotor. Then we propose a transformation planning method utilizing the 1 DoF thrust vectoring angle with consideration of guaranteed minimum force/torque. Finally we perform an aerial transformation experiment with a real platform to demonstrate the feasibility of our proposed design and methods.

As robots start to execute complex manipulation tasks, they are expected to improve their skill set over time as humans do. A prominent approach to accomplish this is having robots to keep models of their actions based on their experiences in order to improve their action executions in the future. In this paper, we present such a methodology where robots start to execute some actions with random parameters and record their generic execution logs with semantic annotations in a symbolic knowledge base for robots. Using the data inside logs, multivariate Gaussian mixture models are fitted to the high-level action parameters for later executions. These affordance models are being updated whenever a new execution is carried on. In essence, robots can use these continuously-updated probabilistic model for improving their actions To prove the applicability we demonstrate opening-a-fridge-door experiments with a PR2 robot.

In this paper, we propose an approach for online whole-body control of position-controlled humanoid robot with 3D multi-contact to cope with contact wrench constraints and joint overload. In our method, robots are controlled under contact wrench constraints with three features: 1) internal wrench control to reduce joint load and prolong the time in which the high-load postures can be maintained 2) feasible utilization of self-collision to reduce joint load by turning off joint servo gains 3) handling degenerated degree of freedom by solving a quadratic optimization problem integrating wrench distribution and inverse kinematics in which internal wrench is controlled only in controllable directions.With our methods, HRP2-JSKNTS could pick up an object under a desk with squatting with the back of the upper leg on the back of the lower leg without sliding at the right arm. We also evaluated the effectiveness of our control to reduce joint load with another experiment.

We proposed the soft flexible wire-driven finger mechanism with the soft skin and the multi-joint skeletal structure using two coil springs. The multi-joint skeletal structure is mainly composed of two coil springs, multiple skeletal members and a fiber wire, and the soft skin is formed outside of them. The soft skin and the multi-joint skeletal structure make it possible to distribute the contact pressure, which is necessary for not only touching a living body such as a human or an agricultural crop, but also for touching an artificial object such as pastry or an industrial product without damaging it or its package. In this paper, we describe the design of the soft flexible wire-driven finger mechanism and development of the three-fingered hand using the finger mechanism we proposed.

To develop Musashi as a musculoskeletal humanoid platform to investigate learning control systems, we aimed for a body with flexible musculoskeletal structure, redundant sensors, and easily reconfigurable structure. For this purpose, we develop joint modules that can directly measure joint angles, muscle modules that can realize various muscle routes, and nonlinear elastic units with soft structures, etc. Next, we develop MusashiLarm, a musculoskeletal platform composed of only joint modules, muscle modules, generic bone frames, muscle wire units, and a few attachments. Finally, we develop Musashi, a musculoskeletal humanoid platform which extends MusashiLarm to the whole body design, and conduct several basic experiments and learning control experiments to verify the effectiveness of its concept.

The musculoskeletal humanoid has many benefits that human beings have, but the modeling of its complex flexible body is difficult. Although we have developed an online acquisition method of the nonlinear relationship between joints and muscles, we could not completely match the actual robot and its self-body image. When realizing a certain task, the direct relationship between the control input and task state needs to be learned. So, we construct a neural network representing the time-series relationship between the control input and task state, and realize the intended task state by applying the network to a real-time control. In this research, we conduct accelerator pedal control experiments as one application, and verify the effectiveness of this study.

In this paper, we propose walking balance control based on Caputurability. The proposed method consists of five strategies: (i) moving Zero Moment Point (ZMP) in the support polygon (ii) landing position modification (iii) landing timing modification (iv) angular momentum control (v) falling detection and fall control. Walking pattern generation calculates the ZMP so that the Capture Point (CP) reaches the position of the supporting foot at the end of the double support phase. Owing to the asymmetry of the reachable landing region, landing timing modification is different in the sagittal and lateral planes, and the step time is extended in the lateral plane depending on the direction of disturbances. The torque around the center of gravity to avoid falling is realized through whole-body inverse kinematics with constraints on the angular momentum. In addition, we propose falling detection considering the reachable landing region. We verified the effectiveness of the proposed method through experiments in which the biped robot was disturbed by pushing during tether-free walking. The robot could prevent breakdown by detecting possible falling and performed knee bending motions to suppress damage.

We present a torque controller for humanoid robot's arm and a method to execute heavy-load tasks under that controller. Torque control of arms can reduce the joint load when an impulsive force is applied to the robot's hand. This feature is important for robots that will work among many humans or obstacles because accidental collisions may happen in such situations. We also developed a static force filter for force sensors at the end-effectors. This filter is utilized for the compensation for the static reaction force during heavy-load tasks. A life-sized humanoid robot JAXON digs soil with a shovel and carries soil with a wheelbarrow using our proposed controller.

In order to achieve tasks in the real world environment, humanoid robots have motors and reduction drives optimized in relation to weight and size for providing the necessary torque and angle speed. Therefore, having torque or angle speed outside of the predicted range will usually cause the task to fail. In this research, we propose a joint module with a two-stage transmission mechanism during driving and a joint locking mechanism during non-driving so that the appropriate torque and joint speed can be attained during the task. By applying the joint module to a tricycle type robot and switching the driving state during the task execution, we were able to both reduce the motor load when lifting heavy objects at driving time and keep high rigidity of the joint at non-driving time.

Self-healing robotics has been of considerable interest. We believe the function will have a major role in legged robots as a typical high-load application of robotics. Some pioneering works have been ongoing on self-healing soft robots. However, the development of large load self-healing component and its system integration with a life-sized legged robot is a challenging task. This study is to try the problem by a constructing self-healing component oriented for facilitated investigation. Proposed part enhances visibility and manufacturing by specializing tension transmission system. The developed module was evaluated by several experiments. First, healing visualization experiment was conducted to evaluate healing progress. In addition, the module's strength was tested using a motor-driven tendon module previously developed in our laboratory. Results of these experiments suggested that the stirring process have a major role in performing self-healing behaviour. Finally, we conducted a preliminary experiment on a tendon-driven legged robot. The experiment demonstrated that the module functioned in a real robot once.

Humanoid robots are expected to conduct tasks on behalf of humans in places such as a disaster scattered environment. Although humanoid robots have potentials to walk on uneven ground unlike wheeled robots, it is difficult to reach a given destination without falling down based on only visual information. In this paper, to reach the destination safely, we propose the autonomous safe locomotion system applying vision and sole reaction force to the footstep planning. Considering force information in addition to visual information, the robot can plan a path avoiding unstable footholds. The planned path is safer than a path which is planned based on only visual information. In our system, the robot checks if the foothold is safe or not by the foothold ascertainment motion. In addition to that, the robot saves the results of the motion to the database with the foothold label given by the visual classifier. To judge foothold safety, stiffness of the foothold is estimated from the reaction force and stepping amount. We propose the system considering these requirements for safe locomotion for bipedal robots and show experimental results using a real bipedal robot CHIDORI.

Transformable aerial robots are favorable in aerial manipulation tasks for their flexible ability to change configuration during the flight. By assuming robot keeping in the mild motion, the previous researches sacrifice aerial agility to simplify the complex non-linear system into a single rigid body with a linear controller. In this paper, we present a framework towards agile swing motion for the transformable multi-links aerial robot. We introduce a computational-efficient non-linear model predictive controller and joints motion primitive frame-work to achieve agile transforming motions and validate with a novel robot named HYRURS-X. Finally, we implement our framework under a table tennis task to validate the online and agile performance.Supplementary MaterialThis paper is accompanied by a experiment video: http://www.jsk.t.u-tokyo.ac.jp/%7eshifan/paper/iros19/video.mp4.

When a humanoid robot manipulates a heavy object, balance and heavy loads become problems. To solve these problems, we provide a method to generate a key pose sequence of the robot and the object which balance constraints and joint torque limits are kept. As we consider the configurations of both the robot and the object, the key poses of them are optimized in a view of kinematics and statics. Moreover, we generate a smooth key pose sequence by adding an objective function which makes adjacent poses closer. By using the proposed method, we make a humanoid robot RHP4B place a heavy suitcase on a step.

To realize a robust automobile driving behavior of musculoskeletal tendon-driven humanoids, we developed a six-axis force measurement module with a core-shell structure. This sensor enables space saving, high load capacity and wholebody sensing at the same time. By developing a foot unit incorporating a core-shell structural force sensor on its toe, we realized behaviors of depressing a pedal and recovering from foot slipping during the depressing with a lifesized musculoskeletal humanoid ”Musashi”.

An integrated humanoid robot system, including from low-level hardware to high-level intelligence software and user interfaces, is required to build humanoid robot system meeting the expectation that robots work in the real environment such as disaster response. For creating such an integrated system, it is important that it has sustainable development potential, partially re-usability, and transparency to any type of robot. When creating a new robot, the conventional a system is desired to use just by changing a robot hardware. In order to realize such system, it is necessary to ensure that a software system is transparent to a robot hardware. On the other hand, sustainable development and partially reusability contribute robustness of the system and quickly building a complex system. In this paper, we describe the methodology to create an integrated humanoid robot system through actual humanoid robot system we have developed. Our system achieved to have sustainable development, partially re-usability, and transparency to any type of robot.

In this paper, we present the achievement of aerial grasping by shape adaptive transformation to the object shape, using a novel transformable aerial robot called HALO: Horizontal Plane Transformable Aerial Robot with Closed-loop Multilinks Structure. Aerial manipulation is an active research area and using multiple aerial robots is an effective solution for the large size object. However the cooperation is considered that there are some difficulties such as the synchronized flight control and collision with each other. Then, we focus on the transformable aerial robot with two-dimensional multilinks proposed in our previous works, which can transform to the suitable form for the target object and grasp it. However the transformable aerial robot with the serial-link structure could not achieve stable flight in terms of horizontal position and yaw control due to the low rigidity and large inertia in the case of more than 4 links. Thus, first we construct a novel type of multilinks with closed-loop structure to avoid the deformation and a new link module with a tilted propeller for fully-actuated control. Second, we describe transformation method with closed-loop multilinks. Third, we present the optimization planning method for the multilinks form to be adaptive to the two-dimensional shape of the target object. Finally, we present experimental results to demonstrate the feasibility of closed-loop aerial transformation and aerial grasping for the large size object.

To enable robots to perform human-level tasks flexibly in varying conditions, we need a mechanism that allows them to exchange knowledge between themselves for crowd-sourcing the knowledge gap problem. One approach to achieve this is to equip a cloud application with a range of encyclopedic knowledge (i.e. ontologies) and execution logs of different robots performing the same tasks in different environments. In this paper, we show how knowledge exchange between robots can be done using OPENEASE as the cloud application. We equipped OPENEASE with ontologies about the kitchen domain, execution logs of three robots operating in two different kitchens, and semantic descriptions of both environments. By addressing two different use cases, we show that two PR2 robots and one Fetch robot can successfully adapt each other's plan parameters and sub symbolic data to the experiments that they are conducting.

In this paper, we propose novel methods suitable for online real time whole body master-slave control with real life-sized humanoid robot. We conducted some dynamic whole body master-slave experiment with life-sized humanoid robot, and we achieved speedier and flexible master-slave operation compared to conventional study. Conventionally, master-slave operations with humanoid robots were available with only the upper body of the humanoid robot, and the COM movement was limited to be static. In our previous study, we introduced LIP model based restrictions to ensure the balance stability. In this study, we extend the safety restrictions by introducing foot landing delay prediction and trajectory smoothing method suitable for real robot. We conducted master-slave tennis swing experiment and high kick motion experiment with life-sized humanoid robot “JAXON”, and we evaluated the effectiveness of our proposed methods and system.

For robots to autonomously achieve manipulation tasks in various scenes, advanced operational skills such as tool use, learning from demonstration, and multi-robot/human-robot cooperation are necessary. In this research, we devise a method for robots to realize such operational skills in a unified manner by evaluating physical consistency (referred to as “physics reasoning”) based on the formulation of the manipulation statics constraints. First, we propose manipulation planning and estimation methods in which the operational feasibility and properties' likelihood are derived by physics reasoning. In addition, we propose a framework to manipulate an object with unknown physical properties by executing planning and estimation both sequentially and in parallel. We demonstrate the effectiveness of the proposed methods by performing experiments in which real humanoid robots achieve various manipulation tasks with advanced operational skills.

In this paper, we introduce a novel robotic prosthetic knee as shown in Fig. 1 (named as SuKnee) with variable transmission mechanism that could vary transmission ratio while knee angle varies during ambulation activities. A slider crank mechanism is utilized to transform linear motion of series elastic actuator to rotary motion of knee joint. And it contributes to variable transmission ratio with knee angle, which help obtain desired speed variation and torque output in different activities in one mechanism. This feature could uniquely give the SuKnee both: the torque necessary to assist with standing up from a chair and the speed necessary to swing the leg forward during walking. The knee has an active mode, where it operates with batteries and is capable of providing external power, and a passive mode, behaving like a passive prosthesis. Preliminary tests have been performed by a transfemoral amputee and SuKnee could provide user with power to assist walking on level ground and standing up from a chair. And a passive mode test shows it could work like passive prosthesis after battery exhaustion.

In this paper, we propose methods for walking on a steep slope using a rope by a humanoid robot. There are two difficulties for walking on a steep slope without a rope. First, range of motion of ankle joints get limited. Second, feet of a robot slip on a steep slope. For these problems, using a rope is effective solution because the robot can receive enough friction force from the slope and walk on a steep slope by pulling a rope with proper tension. In addition, the robot pulling a rope on a slope can relax limitations of ankle joints. Therefore, we propose methods to determine tension of a grasped rope by solving a linear least-square problem considering deformability of a rope. With these methods, a life-size humanoid robot HRP-2 could walk on a steep slope which angle is 40 degree.

For a robot to manipulate an object, it has to understand the functions and the actions that can be subjected to the object. This set of information is known as affordance of the object. Affordances are generally defined by the geometrical structures and physical properties of the objects. In this paper, we present an affordance detection network (ADNet) for detecting object affordances using multimodal input i.e., RGB-D data. The method is based on the state-of-the-art fully convolutional network with two encoding streams and one decoding stream. In the presented formulation, the network learns powerful discriminative features independently from the RGB and depth images, which enables it to abstract rich photometrical and geometrical properties of the objects. The multimodal encoding is combined at multiple stages of the network using the late-fusion strategy and used is for predicting the potential affordances of the objects.

To enable robots to work on real home environments, they have to not only consider common knowledge in the global society, but also be aware of existing rules there. Since such “local rules” are not describable beforehand, robot agents must acquire them through their lives after deployment. To achieve this, we developed a framework that a) lets robots record long-term episodic memories in their deployed environments, b) autonomously builds probabilistic object localization map as structurization of logged data and c) make adapted task plans based on the map. We equipped our framework on PR2 and Fetch robots operating and recording episodic memory for 41 days with semantic common knowledge of the environment. We also conducted demonstrations in which a PR2 robot tidied up a room, showing that the robot agent can successfully plan and execute local-rule-aware home assistive tasks by using our proposed framework.

Robotic picking of folded objects such as books is required for picking various objects. As a folded object is easily unfolded, it is difficult to carry it stably and place it in a desired pose due to its dangling part. For overcoming this difficulty, we propose a trial-and-error picking system using our Suction Pinching Hand, which can push the dangling part up with pinch grasp until the object lifted with suction grasp is folded. That system utilizes proximity sensors on the hand to predict whether folding will succeed with a current hand pose and decide whether to retry with another pose. Also, proximity sensors, flex sensors and an air pressure sensor are used to deal with uncertainty of the image recognition, the hand hardware and suction grasp. We evaluate our proposed system with experiments of picking and placing folded objects. It is confirmed that our proposed system realizes picking with the ability of our Suction Pinching Hand to carry folded objects stably and place them in desired poses. It is also proved that our proposed system is robust against the uncertainty.

Tendon-driven musculoskeletal humanoids have many benefits in terms of the flexible spine, multiple degrees of freedom, and variable stiffness. At the same time, because of its body complexity, there are problems in controllability. First, due to the large difference between the actual robot and its geometric model, it cannot move as intended and large internal muscle tension may emerge. Second, movements which do not appear as changes in muscle lengths may emerge, because of the muscle route changes caused by softness of body tissue. To solve these problems, we construct two models: ideal joint-muscle model and muscle-route change model, using a neural network. We initialize these models by a man-made geometric model and update them online using the sensor information of the actual robot. We validate that the tendon-driven musculoskeletal humanoid Kengoro is able to obtain a correct self-body image through several experiments.

The sequential online tuning for controller gains is required for the continuous action of the riding into parallel two-wheeled scooter and the speed governing after riding by humanoid robot. The implemented controllers are different between the riding and the speed governing, and these tuning strategies are also different. In particular, the riding requires the immediate tuning in the short riding phase and the speed governing requires the accurate tuning to regulate the speed of humanoid robot. To the above requirements, this paper proposes the Sequential Online Learning Control (SOLC)method composed of the cascade connection of SGD-based open-loop Learning Control (SLC)and Mini-batch-based closed-loop Learning Control (MLC). SLC contributes the damping gain online tuning for the foot torque control during execution of riding, and MLC contributes the PID gains online tuning for the speed governing control. Finally, we show the validity of SOLC through the sequential experiment of riding and speed governing for parallel two-wheeled scooter by life-sized humanoid robot HRP2-JSK.

Recently, self-supervised approach is common for robot grasping. Although this approach improves success rate, it requires a long time to execute a number of grasp trials, and single-arm grasping is only considered. However, robots can grasp more various objects with two arms, and dual-arm robots such as humanoid robots are expected to execute dual-arm manipulation and overcome the single-arm limitation. In this paper, we introduce dual-arm grasping as another possible strategy and propose a multi-stage learning method for selective dual-arm grasping using Convolutional Neural Networks (CNN)for grasping point prediction and semantic segmentation. In the first stage, the network learns grasping points with the automatic annotation. Although a robot learns both single-arm and dual-arm grasping efficiently with the annotation, the robot may not be able to grasp it because the annotation algorithm is designed by human. Therefore, for the second stage, the robot samples various grasping points with both grasping strategies and learns how to grasp in the real world. In this stage, the robot obtains new possible grasping points and prunes unsuccessful ones for both grasping strategies through the robot experience. In the experiments in the real world, the adapted network achieved high success rate 76.7% in 90 trials. Since the network trained with no adaptation stage resulted in lower success rate 56.7%, this result also shows the network was refined with less than 250 times of grasp sampling. As an application of our method, we demonstrated that our system worked well in warehouse picking task.

Human hands can not only grasp objects of various shape and size and manipulate them in hands but also exert such a large gripping force that they can support the body in the situations such as dangling a bar and climbing a ladder. On the other hand, it is difficult for most robot hands to manage both. Therefore in this paper we developed the hand which can grasp various objects and exert large gripping force. To develop such hand, we focused on the thumb CM joint with wide range of motion and the MP joints of four fingers with the DOF of abduction and adduction. Based on the hand with large gripping force and flexibility using machined spring, we applied above mentioned joint mechanism to the hand. The thumb CM joint has wide range of motion because of the combination of three machined springs and MP joints of four fingers have variable rigidity mechanism instead of driving each joint independently in order to move joint in limited space and by limited actuators. Using the developed hand, we achieved the grasping of various objects, supporting a large load and several motions with an arm.

We propose a method for a robot to manipulate an unknown door based on a single user instruction. The primary contributions of this paper are (i) to reduce the user instruction to a single click and (ii) to develop an efficient method to estimate an appropriate shape and manipulation model for a target door by integrating visual and touch information obtained by a robot. The proposed method first detects door candidates using a 3-D camera and then estimates the manipulation model of each candidate based on prior learning results. During door manipulation, the system integrates visual and touch information to estimate the shape and manipulation model to generate an appropriate motion. We evaluated the proposed method experimentally, and the results prove that the proposed method is effective.

Comparing to IK (Inverse Kinematics) based motion control, joint-space motion control is more advantageous in terms of not being restricted by kinematics singularity problem. In this paper, we start with SIMBICON (Simple Biped Locomotion Control) based controller, a joint-space motion control method, extend it for enhancing walking's robustness and versatility. We propose a motion optimization method considering walking robustness, desired walking velocity and energy efficient minimization for walking motion generation. This method enables us to achieve human-like walking motion, which has stretched-knee posture and robust to large push disturbances. We also apply our proposed method to a life-sized biped robot and validate its effectiveness with push recovery and walking on unknown debris experiments.

In this paper, we develop a bipedal robot control system that has an ability to perform instantaneous high power and flexibility to absorb an impulsive disturbance. We utilize a sensor-less whole body torque control method executed in a high responsive realtime distributed system. This system also includes a robust online walking controller that can avoid fall over caused by a strong collision with the robot's legs. We evaluated the proposed control system by hitting a rubber ball or adding a leg sweep disturbance and verified the functionality of the absorbing motion and the balance restoring motion.

Currently, most of commercially available ankle foot prosthesis are passive, which don't exhibit appropriate biomechanics during walking and could not adapt to dynamic property of able-bodied walking. In this paper, we present a novel robotic ankle foot prosthesis with variable transmission series elastic actuator (SEA). Slider crank mechanism is applied to transform linear motion of series elastic actuator to rotary motion of ankle foot joint. And this could contribute to variable transmission ratio while ankle angle varies. Because of variable transmission ratio, ankle joint torque is increasing while ankle angle is flexed from plantar flexion to dorsiflexion, whose feature has similar increase trend with human's ankle joint torque-angle relationship, and exhibits an appropriate characteristic for developing robotic ankle foot prosthesis. Larger torque could be obtained in powered plantar flexion, and this indicates that variable transmission mechanism would help reduce required motor torque compared with traditional mechanism. Energy stored in springs of series elastic actuator contribute a torque to powered plantar flexion. Preliminary experiments with a transtibial amputee and a transferomal amputee have been performed to test the prototype.

We present a robotic system for picking a target from a pile of objects that is capable of finding and grasping the target object by removing obstacles in the appropriate order. The fundamental idea is to segment instances with both visible and occluded masks, which we call `instance occlusion segmentation'. To achieve this, we extend an existing instance segmentation model with a novel `relook' architecture, in which the model explicitly learns the inter-instance relationship. Also, by using image synthesis, we make the system capable of handling new objects without human annotations. The experimental results show the effectiveness of the relook architecture when compared with a conventional model and of the image synthesis when compared to a human-annotated dataset. We also demonstrate the capability of our system to achieve picking a target in a cluttered environment with a real robot.

Robots need to adapt themselves to various surroundings in order to achieve robust object search and grasp in unknown environments. For this adaptation, robot motions should be implemented as combination of primitive motions which are based on sensor reaction. Among various sensing methods, non contact sensing is required as a means of preventing operation failures such as pushing objects. Especially, proximity sensors have been proved effective in avoiding occlusion problems. In this paper, we first develop a gripper on which proximity sensors are mounted all around, and then calculate distance between the gripper and objects using proposed calibration method. This enables robots to recognize detailed shapes of objects surrounding the gripper. We also propose primitive motions for object search and grasp, and describe the contents of each motion. The motions are based on sensor information obtained from the gripper. We verify the effectiveness of our system through an experiment in which a real robot performs complex tasks by combination of the primitive motions.

In this paper, we introduce the achievement of the flight motion to pass through small opening by the multilinked and transformable aerial robot. Previous works about such motion are based on under-actuated multirotors, indicating that aggressive maneuvering is necessary condition. This involves two crucial problems: i) enough free space for deceleration is necessary, otherwise the robot would collide with unknown obstacle after exiting opening; ii) the multirotor can not traverse the openings that are smaller than the robot body. The proposed transformable aerial robot in our work can solve these problems, since the multilinked model can not only guarantee the near-hover condition during the whole motion sequence, but also slowly traverse relative small openings by changing its form like a snake. We first propose an improved dynamics derivation and flight control method for this multilinked aerial robot based on our previous work. Then, we present the path planning method which takes the flight stability in the near-hover condition into account. Finally we demonstrate the experimental results of the motion to pass through a horizontal and small opening which also involves the borders (the floor and the ceiling).

To promote psychosocial development in children, a number of efforts are being made to improve child-child interactions. In this paper, we determine whether walking with a robot has a positive effect on child-child interactions. We developed a walking-together system consisting of two methods of walking: (1) walking hand in hand and (2) following a person by tracking the face. After applying the system to Pepper, a humanoid robot, we conducted the field experiment in a science museum in two phases. The first phase was in an open laboratory, setting in which people (including children) freely interacted with the robot. The second phase was the experimental setting in which 2-3 children simultaneously interacted with the robot. We observed the kinds of child-child interactions that occurred in the two phases of the experiment. We also determined what would be important to generate longer child-child interactions by walking together.

lack of social relationships has become a significant issue in modern society. A robot is expected to enhance communication between people by introducing each person using knowledge of the person which the robot has obtained through interactions on their daily life. Therefore, we need to clarify what kind of behavior is required when a robot interacts with a group of people in such a situation. The goal of this paper is to reveal the kind of communication that occurs between a group of people new to each other if a robot introduces each person. We conducted an experiment in which two robots communicated with 14 people (7 children and 7 adults) from 5 families in the following situation. First, the robots asked each person a simple question about their person or daily life. Then, all the people and the two robots gathered together. The two robots started chatting, including an introduction of each person. Based on the video coding and questionnaires, we found that the robots generated an opportunity for people new to each other to learn about each other. Furthermore, we discuss the design requirements of the robot behavior to enhance communication between people new to each other through the introduction of each person by a robot.

Whole-body manipulation is necessary for a humanoid robot to achieve tasks such as carrying large objects. One difficulty for achieving a whole-body manipulation is that the robot needs to select the appropriate operation from a list of candidates, such as lifting, pushing, and tilting. The appropriate operation depends upon the target object's physical properties, including its mass, center of mass, and friction coefficient, which are difficult to measure directly. In order to select the appropriate manipulation motion online, we propose a method of estimating the object's physical properties and evaluating the feasibility of the object operation. We calculate the likelihood of the object's physical properties from sensor information during manipulation and update these properties' probabifity distribution periodically based on Bayesian methods. The operational feasibility probability is evaluated by physics-based stability determination, allowing the robot to perform manipulation tasks by selecting the appropriate operation. We show the effectiveness of the proposed method by an experiment in which a life-sized humanoid robot carries a large object.

This paper shows a method for achieving multi-contact motion for a humanoid robot on a movable object, such as climbing of a stepladder. Recent research has developed methods for achieving multi-contact motion that considers various constraints, such as joint limits, torques, balance constraints, reachability, and collision avoidance. In addition to these constraints, Motion On a Movable Object (MOMO) has the following features: it has to consider an object's balance during the changing of contact points; and it has to handle scenarios where the mass properties of an object are unknown. In this paper, in order to achieve a humanoid robot having MOMO, we propose balance constraints that consider the constraints imposed by an object as well as an online estimation of object's constraints. First, we use object-environment constraints as the robot's constraints, and then we show a method for estimating them based on information provided by the robot's sensors. Next, we show a method for applying the balance constraints to a humanoid motion planner and for executing planned motion with real-time sensor feedback controller. Finally, we evaluate our proposed method through experiments in which a life-sized humanoid robot climbs stepladders that have unknown mass properties.

In this paper, we introduce the achievement of the aerial manipulation by using the whole body of a transformable aerial robot, instead of attaching an additional manipulator. The aerial robot in our work is composed by two-dimensional multilinks which enable a stable aerial transformation and can be employed as an entire gripper. We propose a planning method to find the optimized grasping form for the multilinks while they are on the air, which is based on the original planar enveloping algorithm, along with the optimization of the internal force and joint torque for the force-closure. We then propose the aerial approach and grasp motion strategy, which is devoted to the determination of the form and position of the aerial robot to approach and grasp effectively the object from the air. Finally we present the experimental results of the aerial manipulation which involves grasping, carrying and dropping different types of object. These results validate the performance of aerial grasping based on our proposed whole-body grasp planning and motion control method.

In this paper, we show the achievement of a transformable aerial robot with internal communication system for multiple objects transportation. As it is not easy to make the flight endurance of an aerial robot longer, we study the problem to transport multiple objects at the same time to improve the efficiency of transportation. However, for conventional aerial robots, multiple objects transportation is difficult because the CoG position changes when the number of grasped objects changes, resulting in the instability of the flight. Therefore, to solve this problem, we focus on the multirotor with two-dimensional multilinks proposed in our previous work, which possesses the ability to modify the CoG position actively and can keep the flight stable. First, we introduce the hardware platform including the structure of link module and internal communication system to achieve the extensibility in terms of the link number. We then propose a method to find the optimal form for the multilinks based on the flight stability. Finally, we present experimental results which include aerial transformation and multiple objects transportation.

In recent years, applications of visual tracking algorithms has seen a substantial growth with deployments in intelligent robots such as drones for human tracking. The algorithms for such tasks has to be efficient in terms of computational cost while been robust, accurate and fast. Object tracking algorithms based on handcrafted heuristics and constraints are widely used in uav applications. The handcrafted heuristics are mostly implemented for task-oriented applications which limits the extensions in uav's capability beyond the predefined functions. This paper considers the challenges of tracking and landing an autonomous uav on a speed high moving target, and presents a visual tracking algorithm that integrates correlation filters with deep comparison network for real-time tracking with state-of-the-art accuracy. The method first tracks the target upto translation using an online learnt model via local search technique. The changes in scale is estimated by a deep comparison network (DCN) instead of the commonly used pyramidal approach. In a single network evaluation, DCN can estimate the changes in scale as well as compensate the drifting of the tracker by refining the object region estimated by the correlation filters. The network is end-to-end trained which attempts to learn a powerful matching function for object localization using a known template. Generally, the integrated framework can be viewed as coarse-to-fine level motion estimation. Moreover, the framework can redetect the lost target without a need for a separate detector.

Picking various objects in cluttered narrow space automatically is required for warehouse automation. In this space, multi-fingered robot hands have difficulty in grasping objects as objects are surrounded by obstacles. On the other hand, vacuum grippers have difficulty in grasping various objects stably. In this paper, we propose the Suction Pinching Hand, which has two underactuated fingers and one extendable and foldable suction finger whose fingertip has a suction cup. This hand can grasp objects in cluttered narrow space using the suction finger. In addition, it can grasp various objects stably using suction and pinch at the same time. The ability to grasp various objects stably of this hand is confirmed by tabletop experiments. We also propose a picking strategy using suction and pinch simultaneously in cluttered narrow space. We evaluate our proposed methods with shelf bin picking experiments. With our methods, a robot can pick various objects in cluttered narrow space.

In this study, we propose a novel method to operate whole body of a humanoid robot, which also includes both feet, dynamically and safely with the master-slave approach. The conventional whole body master-slave approaches need static balancing assumption or a certain time length of planning after operator's input. Then, we introduce a set of limitations that allows the robot to execute human's daily dynamic bipedal locomotion, but forbid dangerous motions like the COM will be gone outside of the support region. In the limitations, we regulate COM velocity based on a positional relation of the Divergent Component of Motion (Capture Point) and the both feet, and automatically modify the swing foot contact timing with judging the ZMP is inside or outside of the single foot support region. At last, we conducted some experiments of the real time master-slave locomotion with using two life-sized humanoid robots and confirmed the effectiveness of our novel limitation methods.

For humanoid robots, unexpected collision can cause instability of robot balancing and damage to both robots and environment. This paper presents a reactive bipedal walking controller against swing foot collision for humanoid robots. This controller is composed of following three components: 1) Swing Foot Trajectory Regenerator, 2) Swing Foot Collision Detector, and 3) Swing Foot Impact Mitigation Controller. By regenerating swing foot trajectory depending on situations, humanoid robots can avoid falling down. However, although humanoid robots detect collision and regenerate a swing foot, collision impact can cause bad effects such as damage and posture rotation. Therefore, to mitigate strong impact, we propose Swing Foot Impact Mitigation Controller, which is composed of two controllers. The proposed method is validated through the experiments by actual humanoid robot CHIDORI. We confirm that CHIDORI can avoid falling down against collision in two situations: walking on the flat ground, and stepping up a stair.

In this paper, we described a new developed life-size humanoid robot. A purpose of the developed robot is to realize continuous operation for a long time and to improve an action autonomously. we considered three aspects of robustness, mechanical robustness, functional robustness and robustness of an action. Mechanical robustness was confirmed by the experiment that the robot fell down without mechanical failures and continued to work after falling down by using hard points. Functional robustness was designed to use power cable and to wear a suit which can be changed by required functionality. Robustness of an action was achieved as a standing up action using “StateNet”, which realized autonomous error recovery. Finally, we present a methodology to develop a humanoid robot platform which can continue to work in the real world.

The human forearm is composed of two long, thin bones called the radius and the ulna, and rotates using two axle joints. We aimed to develop a forearm based on the body proportion, weight ratio, muscle arrangement, and joint performance of the human body in order to bring out its benefits. For this, we need to miniaturize the muscle modules. To approach this task, we arranged two muscle motors inside one muscle module, and used the space effectively by utilizing common parts. In addition, we enabled the muscle module to also be used as the bone structure. Moreover, we used miniature motors and developed a way to dissipate the motor heat to the bone structure. Through these approaches, we succeeded in developing a forearm with a radioulnar joint based on the body proportion, weight ratio, muscle arrangement, and joint performance of the human body, while keeping maintainability and reliability. Also, we performed some motions such as soldering, opening a book, turning a screw, and badminton swinging using the benefits of the radioulnar structure, which have not been discussed before, and verified that Kengoro can realize skillful motions using the radioulnar joint like a human.

Human can not only support their body during standing or walking, but also support them by hand, so that they can dangle a bar and others. But most humanoid robots support their body only in the foot and they use their hand just to manipulate objects because their hands are too weak to support their body. Strong hands are supposed to enable humanoid robots to act in much broader scene. Therefore, we developed new life-size five-fingered hand that can support the body of life-size humanoid robot. It is tendon-driven and underactuated hand and actuators in forearms produce large gripping force. This hand has flexible joints using machined springs, which can be designed integrally with the attachment. Thus, it has both structural strength and impact resistance in spite of small size. As other characteristics, this hand has force sensors to measure external force and the fingers can be flexed along objects though the number of actuators to flex fingers is less than that of fingers. We installed the developed hand on musculoskeletal humanoid “Kengoro” and achieved two self-weight supporting motions: push-up motion and dangling motion.

This paper explains low-latency joint torque feedback control based on torque estimation on each joint for gear-driven humanoid robots with harmonic drives. Force control of gear-driven robots has an advantage in its fully variable stiffness in comparison with elastic robots. However, feedback latency makes gear-driven robots vulnerable to impact rising in several milliseconds. It would be resolved by low-latency torque feedback loop in a single joint, but torque sensors are too large for life-sized humanoid robots. We estimate joint torque from motor current and rotation observed in each joint, and give artificial elasticity to joints using compliance control and shock absorption control. Our controller performance is demonstrated by landing experiments.

This paper presents a COM trajectory generation method for 3D walking and skating motion by nonlinear optimization. In our method, we solve the following problems: (1) dealing with both walking and skating motion in the same framework, (2) generating center of mass (COM) trajectory faster than execution time, (3) executing motion with large acceleration. For solving (1) and (2), we calculate the COM trajectory at every step and introduce frictional constraints to the Divergent Component of Motion as terminal conditions. By changing the terminal condition, we can generate both skating and walking motion. Besides, the nonlinear constrained optimization using Gauss Pseudospectral Method is introduced for solving (2) and (3). Thanks to this method, we generate the 3D COM trajectory considering contact constraints and kinematic constraints faster than execution time. Finally, the walking and skating experiment were carried out to confirm the effectiveness of our method using life-sized humanoid HRP-2. Applying the proposed method, HRP-2 could successfully walk at 0.4 [m/s] and skate at 1.0 [m/s].

Probabilistic 3D map has been applied to object segmentation with multiple camera viewpoints, however, conventional methods lack of real-time efficiency and functionality of multilabel object mapping. In this paper, we propose a method to generate three-dimensional map with multilabel occupancy in real-time. Extending our previous work [1] in which only target label occupancy is mapped, we achieve multilabel object segmentation in a single looking around action. We evaluate our method by testing segmentation accuracy with 39 different objects, and applying it to a manipulation task of multiple objects in the experiments. Our mapping-based method outperforms the conventional projection-based method by 40-96% relative (12.6 mean IU3d), and robot successfuly recognizes (86.9%) and manipulates multiple objects (60.7%) in an environment with heavy occlusions.

Groping behavior based on contact sensors is necessary for manipulation in an unknown environment. For those situations, it is effective for a robot to accumulate contact information as an environment map, and to plan the motions for executing the safe trial motion. We first propose a method of updating the occupancy grid map of the manipulation region from the contact information by introducing the contact sensor model. Using this map, we propose a method of sampling-based motion planning that enables the execution of the safe trial motion based on the criteria of feasibility and safety. To verify the effectiveness, we show the experimentally obtained results, showing that a real robot plans and executes the manipulation with groping behavior in the occluded environment.

We propose a human mimetic foot structure for musculoskeletal humanoids. We designed the foot structure by inspiring from human foot abilities of the multi-bone connected structure for flexibility and the distributed force sensor system. The foot has multi-DOFs structure including toe DOF that is composed of fingers. The distributed force sensing system is composed of 12 an-axis force sensors. In order to demonstrate those effectiveness, we implement the foot into musculoskeletal humanoid Kengoro and conduct several experiments. As a result, we confirmed effectiveness of the foot from tiptoe motion and balancing behavior by utilizing the foot characteristics.

To enable robotic sensing for tasks with requirements on weight, size, and cost, we develop an ultra-tiny line laser range sensor based on the Time-of-Flight (TOF) principle. With delicate circuit design and optical attachments, we create a sensor as small as 35[mm] × 27[mm] × 30[mm] and as light as 20[g]. The line sensor samples 272 pixels (256 effective pixels) uniformly distributed within the measurement field of view customizable using different laser lenses. The optimal measurement range of the sensor is 0.05[m] ~ 2[m]. Higher sampling rates can be achieved with a shorter range. The sensor can also extend its range to 3[m] with reduced accuracy. We model the overall errors of the sensor and formulate calibration methods, achieving repeatable accuracy and measurement bias both within 2[cm] with our tested ambient lighting conditions and measurement ranges. The sensor is applicable to range sensing tasks including humanoid hand-eye measurement, UAV safe landing, tiny robot range sensing, and object detection.

Humanoid robot has the potential to manipulate wide range of tools in daily life. Arms and legs of humanoid robot contribute this ability. Above all, manipulation tasks for vehicles which are the same size as a life-sized humanoid or larger size than it require the operational motion by both arms and legs of humanoid robot. In addition to the arms and legs cooperative motion control, it is also important for humanoid robot to stabilize self posture during driving vehicle. In this research, we focus on the arms-legs-integrated manipulation task for tricycle controlled by humanoid robot. We propose dual manipulators control law that is defined as active manipulator which works movable objects such as handle and crank, and passive manipulator which follows the movement of this objects. We discuss the self stabilizing strategy for humanoid robot by both active manipulating legs as well as manipulation strategy for objects. Furthermore, this paper contributes the strategy of recognition and planning for outside obstacle situations and configures the tricycle manipulation system. Applying this proposed system, we show the experimental result for tricycle manipulation by life-sized humanoid robot HRP2-JSK on obstacle-mixed situation.

In this paper, we develop a waterproof suit for humanoid robots and propose an underwater walking control method. Although very few life-sized humanoid robots are completely waterproof, we can easily make these humanoid robots watertight by putting a waterproof suit on them. In water, humanoid robots are influenced by the two forces due to the water: buoyancy and drag force. We take buoyancy into account when generating a walking pattern because the force is large and easy to estimate before walking. However, drag force is small and difficult to precisely predict and therefore, we treat the force as an unknown disturbance. In our method, we modify footsteps based on the Capture Point in order to deal with large disturbances. We verify the effectiveness of the proposed methods through an experiment in which a life-sized humanoid robot walks on a floor, stairs and debris in water.

In this paper we propose a novel method to utilize the skeletal structure not only for supporting force but for releasing heat by latent heat.

To achieve tasks in unknown environments with high reliability, highly accurate localization during task execution is necessary for humanoid robots. In this paper, we discuss a localization system which can be applied to a humanoid robot when executing tasks in the real world. During such tasks, humanoid robots typically do not possess a referential to a constant horizontal plane which can in turn be used as part of fast and cost efficient localization methods. We solve this problem by first computing an improved odometry estimate through fusing visual odometry, feedforward commands from gait generator and orientation from inertia sensors. This estimate is used to generate a 3D point cloud from the accumulation of successive laser scans and such point cloud is then properly sliced to create a constant height horizontal virtual scan. Finally, this slice is used as an observation base and fed to a 2D SLAM method. The fusion process uses a velocity error model to achieve greater accuracy, which parameters are measured on the real robot. We evaluate our localization system in a real world task execution experiment using the JAXON robot and show how our system can be used as a practical solution for humanoid robots localization during complex tasks execution processes.

For generating motions of robots, global search in configuration space is time consuming although it is sometimes indispensable (e.g. collision avoidance in complex environment). Our idea is to use global sampling algorithm not in the state space but in the task nullspace, which is the redundant degrees of freedom of the state space with respect to the task space. Because the task nullspace is smaller than the original search space (state space), fast global sampling is possible. For embedding this hidden task nullspace parameters, we propose a new deep-auto-encoder-based neural network structure. Our approach learns the map from task and task nullspace towards robot's state (Task-State Map, TSM). As the demonstration, the relationship between 28-dof joint angles (state) and the end-effector coordinates of all limbs (task) is learned, and egress postures and reaching postures are generated.

In this paper, we propose a real-time control method for skating motion of humanoid robots. There are three problems for skating motion: (1) keeping dynamic balance, (2) adequately controlling foot force to suppress slipping at the foot, (3) controlling full-body motion in real-time. For solving these problems, we propose the Skating Motion Generator and the Skating Motion Stabilizer. In the Skating Motion Generator, we separate the slip suppression from motion generation for (3). The separation enables us to generate skating motions in real-time. In the Skating Motion Stabilizer, we adjust the sole pressure distribution of each foot to solve the contradiction between (1) and (2). We show the effectiveness of the proposed controller through the experiments, in which life-sized humanoid HRP-2 pushes the ground and skates on the skateboard. Applying the proposed controller, HRP-2 could successfully accelerate and skate on the skateboard at 0.5[m/s].

In this paper, we present a design and development an autonomous tomato harvesting robot. We developed a harvesting robot with stereo camera which can measure depth in short range and direct sunlight and plucking gripper using the infinity rotational joint. We also evaluate the developed robot through harvesting in the tomato robot competition and the real farm. In the tomato robot competition, the robot harvested tometoes from tomato clusters and tomato trees, harvesting speed was about 80[s/fruit] and successful rate was about 60%. In the real farm we evaluated the robot with tomato trees in semi-outdoor environment to show the effectiveness and robustness under direct sunlight. According to the result of harvesting with real tomatoes, we improved the robot motion and finally harvesting speed was up to 23[s/fruit], however the gripper may grasp multiple fruits in case of very cluttered cluster and the calyx also may be broken when the stem angle is deep from the rotation axis. To avoid this situation, a grasp state estimation of the gripper and simultaneous recognition fruit and stem positions are next problems to improve the harvesting successful rate.

Humanoid robot is able to execute various behavior to manipulate objects because of high degree-of-freedom around the whole-body. Although hands contact with objects and exert force in ordinary pushing motion by robot, pushing motion contacting with the object at various regions of whole-body has potential for extending the scope of feasible manipulation. We derive the fundamental formulas of humanoid robot in the situation that the external force is applied to the arbitrary region of whole-body, and then propose the method to generate and execute the pushing motion based on the formulas. The proposed method is generalized for enabling to select a contact point with an object from whole-body regions and control the pushing force applied to the sensorless region. In order to verify the effectiveness, we show the experimental result that a lifesized humanoid carries large and heavy objects by pushing with various regions of whole-body.

We propose a sensor-driver integrated muscle module by integrating necessarily components for tendon-driven robot which is likely to complicate. The module has abilities of high-tension measurability and flexible tension control. In order to achieve flexible tension control, we developed the new tension measurement mechanism with high-tension measurability and the new motor driver which enables current based motor control. We demonstrate the tension control ability of the module by several experiments. Furthermore, utilizing the module advantage of design facilitation, we made two types of tendon-driven robots and confirmed effectiveness of the module.

To enable robots to learn handover orientations from observing natural handovers, we conduct a user study to measure and compare natural handover orientations with giver-centered and receiver-centered handover orientations for twenty common objects. We use a distance minimization approach to compute mean handover orientations. We posit that, computed means of receiver-centered orientations could be used by robot givers to achieve more efficient and socially acceptable handovers. Furthermore, we introduce the notion of affordance axes for comparing handover orientations, and offer a definition for computing them. Observable patterns were found in receiver-centered handover orientations. Comparisons show that depending on the object, natural handover orientations may not be receiver-centered; thus, robots may need to distinguish between good and bad handover orientations when learning from natural handovers.

We present a vision cognition framework for tomato harvesting humanoid robot based on geometrical and physical reasoning. Inspired from the natural human harvesting behaviour, our goal is to build a humanoid robot to pick tomatoes autonomously or with minimal human efforts. The proposed vision approach uses fusion of calibrated observation data from two RGB-D sensors installed on the head and the hand of the humanoid. We observe the natural human harvesting behaviour and equip our robot with similar grippers to follow the same picking processes for a specific fruit. In the vision approach, we mainly focus on modelling fruits in one branch and then estimating the pedicel direction of each fruit in a branch. Through pointcloud model segmentation, the primitive shape model of each fruit can be obtained and we consider a simple fact that crops in one branch should remain stable with respect to gravity and interaction forces from neighbouring crops in the branch. According to this assumption, a probabilistic model is created and the picking order in the branch is assigned under the evaluated geometrical structure. In the experiments, we tested harvesting of real tomatoes on actual branches and evaluated the successful harvesting rate.

This paper presents a novel control method to stabilize the whole-body motion of humanoid robots when climbing vertical ladders and transitioning between ladders and catwalks. In such environments, the body of the robot tends to incline and rotate because of the slippery surfaces. The inclination and rotation may cause the robot to fail to grasp and thus collide with the rungs. The proposed method modifies the subsequent contact position in real time based on the error of the current robot posture estimated with inertial measurement units (IMUs) and actual joint angles. This paper also presents a method of generating motion by minimizing the contact wrench. This method satisfies hardware limitations, such as collision avoidance, joint torque limits, and joint limits. Applying these methods to a humanoid robot, we realize the robust climbing and descending of multiple rungs of a vertical ladder and bidirectional transitioning from ladders to catwalks.

We have developed a robot with a new control mechanism in order to collect information on flying robots in multiple fields. We aimed for a function that could rotate the tilt angle continuously and without limit and a function for flying maintaining any desired tilt angle with a structure that could efficiently use the thrust generated by the propellers. We devised a mechanism that connected two bicopter modules, each of which combines two of the four propellers into one set and named this mechanism the “Bi2Copter”. This mechanism provided movements including landing, take-off, and flying with any desired tilt angle. This ability of this mechanism to fly walls with continuously changing surface angles and full 360° spherical coverage makes possible applications in investigation, measurement, etc. This report covers the design concepts of this flying robot, the structure design, basic control and operations verification.

In situations where humanoid robots with constrained posture walk through a narrow space (e.g. manufacturing plants and kitchens), shuffling motions that are stepless and possess wide foot supporting area are effective. One of the difficulties of humanoid's shuffle translations is the load distribution between both feet. If sole loads are not distributed appropriately, the humanoid robot cannot maintain target contact states of each foot, and it will result in slipping both feet or falling down. In this paper, we propose Slide Friction Control (S.F.C.): offline pattern generator and Slide Contact Stabilizer (S.C.S.): online controller. First, Slide Friction Control determines reference foot forces and COM trajectories by adjusting sole loads and considering kinematic friction. The appropriate load distribution of S.F.C. enables humanoid robots to maintain target foot contact states. Second, Slide Contact Stabilizer controls each foot by using damping control to realize reference foot forces determined by S.F.C. S.C.S. enables humanoid robots to slide foot smoothly by suppressing friction vibrations. We also take into consideration the dynamic balance of humanoid robots such as previous waking stabilizers. Finally, we demonstrate that the proposed system enables humanoid robot to slide their feet smoothly using a life-sized humanoid robot, HRP-2.

The main goal of this paper is to design and evaluate a spine structure which withstands various motions. The structure around the neck has a prevailing importance since it is involved in various motions of the upper half of the body. The new design method we introduce essentially shows how to design all 7 cervical vertebrae (the part of spine in the neck) in a limited space, actuated by the wires winded around the motors. Then we show the muscle arrangements around the upper half of the spine. More specifically, we make use of a so called planar muscle mechanism. An abduction experiment which requires great force around the spine is made to show its stability. Finally, we show a variable stiffness system which enables the spine to resist an impulsive force. We have tested the system in the situation of whiplash injury which is a case of extreme external forces which can occur in car crash accidents. As such we have evaluated the strength of the design and the viability of our robot to act as a human body simulator.

Whole-body holding manipulation is effective for carrying the handleless large object. In order to keep the object stability, the dexterous transition motion is necessary. From geometric and physical conditions of object manipulation, we propose the general method of generating the transition graph, which represents the object pose and grasp contact. By searching the path on the graph, the transition motion is planned automatically with considering the object motion and contact switching simultaneously. By generating and modifying the whole-body holding motion, the planned object motion is achieved stably. We show the effectiveness of the proposed method by the experiments, in which robot lifts up a large object with whole-body contact by the planned transition motion.

For generating whole-body behavior involving contacts with environments such as climbing ladder behavior or walking on terrain behavior, “contact-before-motion” approach was used in some previous researches. By separating contact search process and motion search process, whole-body behavior generation was achieved. However, in previous researches, there were few examinations about contact transition strategies while generating behavior. For changing contact states, there are many kinds of contact transition strategies such as walking, sliding, rotating and so on. For example, while generating standing up behavior, it is important to switch contact transition strategies because the lack of degree of freedom makes it difficult to detach limb contacts from ground, and it may be desirable not to detach contacts but to slide them. In this study, we propose a novel whole-body behavior generation algorithm which involves contact transition strategies switching function. Especially, in this paper, we focus on the walk-type and slide-type transition strategies switching. We call walk-type transition as the contact transition process which detaches some contacts, moves them, and attaches them again. Besides, we call slide-type transition as the contact transition process which keeps on attaching contacts, and slides to move them. By using this algorithm, it is possible to generate whole-body behaviors which are difficult or impossible to achieve only with walk-type transition and which are more desirable by comparing multiple transition strategies. Finally, we evaluated this algorithm by generating standing up behavior and sitting on chair behavior.

Recent research in psychology argue the importance of “context” in emotion perception. According to these recent studies, facial expressions do not possess discrete emotional meanings; rather the meaning depends on the social situation of how and when the expressions are used. These research results imply that the emotion expressivity depends on the appropriate combination of context and expression, and not the distinctiveness of the expressions themselves. Therefore, it is inferable that relying on facial expressions may not be essential. Instead, when appropriate pairs of context and expression are applied, emotional internal states perhaps emerge. This paper first discusses how facial expressions of robots limit their head design, and can be hardware costly. Then, the paper proposes a way of expressing context-based emotions as an alternative to facial expressions. The paper introduces the mechanical structure for applying a specific non-facial contextual expression. The expression was originated from Japanese animation, and the mechanism was applied to a real desktop size humanoid robot. Finally, an experiment on whether the contextual expression is capable of linking humanoid motions and its emotional internal states was conducted under a sound-context condition. Although the results are limited in cultural aspects, this paper presents the possibilities of future robotic interface for emotion-expressive and interactive humanoid robots.

Object handover is a basic task in many human-robot interactive scenarios and therefore, it is important for assistive robots to be able to perform proper handovers. We previously designed a human-inspired grip-force-varying handover controller for a robot giver and showed on a Willow Garage PR2 robot that the controller yields human-like and human-preferred handovers. The PR2 robot had a non-compliant fully-actuated gripper. However, recently, compliant underactuated grippers have been gaining more popularity. Although compliant underactuated grippers can provide more flexibility in manipulation, it is generally difficult to accurately measure and control the amount of applied grip force. In this paper, we present an implementation of the human-inspired handover controller on a Kawada Industries HRP4R robot, which has compliant underactuated hands, using joint position error measurement for estimating the amount of applied grip force. Through an experiment, we show that we are able to achieve safe, smooth, and intuitive robot-human handovers despite the lack of accurate grip force control on our robot.

Life-size humanoids which have the same joint arrangement as humans are expected to help in the living environment. In this case, they require high load operations such as gripping and conveyance of heavy load, and holding people at the care spot. However, these operations are difficult for existing humanoids because of their low joint output. Therefore, the purpose of this study is to develop the highoutput life-size humanoid robot. We first designed a motor driver for humanoid with featuring small, water-cooled, and high output, and it performed higher joint output than existing humanoids utilizing. In this paper, we describe designed humanoid arm and leg with this motor driver. The arm is featuring the designed 2-axis unit and the leg is featuring the water-cooled double motor system. We demonstrated the arm's high torque and high velocity experiment and the leg's high performance experiment based on water-cooled double motor compared with air-cooled and single motor. Then we designed and developed a life-size humanoid with these arms and legs. We demonstrated some humanoid's experiment operating high load to find out the arm and leg's validity.

This paper proposes a humanoid dance motion generation system that deals with a huge variety of leg motions. While previous research only tackled on a few kinds of leg motions, original human dance leg motions contain various foot touch states such as slide, turn, and heel contact, as well as complex motions such as kick and twist. According to the dance literature, we found that there are seven major foot touch states that make dance motion more “dance-like”. Thus we present a method to classify the seven kinds of foot touch state from human dance motion data, and describe the various dance leg motions by using combinations of the foot touch states and key-frames. Based on these methods, we designed the humanoid dance motion generation system that enables humanoid robots not only to satisfy the geometric condition but also to imitate various human dance leg motions. Finally we show an experiment using a life-sized humanoid, HRP-2.

In case that a robot carries large and heavy objects with unknown physical parameters such as mass automatically, the autonomous decision and execution of the manipulation strategy are necessary. The method to decide the proper strategy from the various candidates depending on the object is a difficult problem and not researched widely. We consider the operation as the mapping from the physical parameter space to the object motion space. Based on the concept of mapping, we define the strategy proving operation (SPO) for determination of strategy feasibility. We introduce two examples of SPO and construct the system for deciding strategy from lifting, pushing, and pivoting. Executing the strategy in the situation that physical parameters are not known is also necessary. We construct the generator and controller for the full-body manipulation, which can be employed regardless of strategy. The controller enables the robot to exert adequate force while keeping balance. We clarify the applicable scope of the proposed method and show that a life-sized humanoid decides the strategy and carries various large and heavy objects autonomously through the experiment.

For humanoid robots to perform whole-body motions, a motion planner should generate feasible motions satisfying various constraints including joint torque limitation, friction, balancing, collision, and so on. Furthermore, for life-size humanoid robots to perform higher-load motions, such as climbing ladders, safely, it is important to generate motions which requirements are not too close to the limitations. In this paper, we propose a humanoid motion planner based on Body Retention Load Vector (BRLV), which is a novel index for representing severity of physical constraints: limitation of joint Torque, contact Force, and contact Moment (TFM limitations). By minimizing the norm of BRLV, we obtain humanoid motions that are farthest from TFM limitations. Finally, we evaluate the proposed motion planner in simulation and confirm the effectiveness of the planner through experiments in which a life-size humanoid robot climbs a ladder and a car.

We present a method for enabling robots to determine appropriate grasp configurations for handovers - i.e., where to grasp, and how to orient an object when handing it over. In our method, a robot first builds a knowledge base by observing demonstrations of how certain objects are used and their proper handover grasp configurations. Objects in the knowledge base are then organized based on their movements and inter-object interaction features. The key point in this process is that similarity in affordances should be recognized. When subsequently asked to handover an object, the robot then computes an appropriate grasp configuration based on the object's recognized affordances. Experimental results show that our method was able to differentiate and group together objects according to their affordances. Furthermore, when given a new object, our method was able to generalize data in the knowledge base and determine an appropriate grasp configuration.

This paper presents a method for daily assistive robots to manipulate objects whose contact relationships are important. In our approach, contact relationships between multiple objects such as “on” and “into” are estimated using the hierarchical relationships between the objects' regions in the images. The hierarchical states are used for failure recovery processes so that the target contact state is achieved. We tested our method in two daily tasks by HRP-2VZ humanoid robot: filing, and serving an egg with a spatula. These results suggested the effectiveness of our approach.

Human knee joint has a yaw-axis rotational DOF and a locking mechanism called screw-home mechanism. We focus on this mechanism and implement it to a musculoskeletal humanoid through hardware design. The importance of developing a knee joint with screw-home mechanism is that such a joint is capable of working yaw-axis properly and generating enough pitch joint torque for supporting whole body motion. In this paper, as an evaluation of our developed knee joint, we first checked the moment arm of the yaw rotational axis of the knee. Moreover, we also checked the yaw angle displacement during squat motion. From these results, we confirmed that the mechanism worked properly. Second, in order to check whether enough pitch joint torque is generated during movement, we conducted several experiments with whole body motions such as squatting. Lastly, as unique and integrated motions that involve the use of yaw DOF derived from the mechanism, we tested knee joint Open-Close, Right-to-Left and whole body twist squat motion. Our results demonstrated the feasibility of musculoskeletal humanoids with screw-home mechanism and showed that we have achieved humanlike twisting motion.

This paper presents a multi-field universal vehicle that is able to work at land, sea and air. The vehicle consists of a quad-copter with variable-pitch propellers that enable the vehicle to stand on the ground at a given tilt angle, roll on the ground like a wheel, and float and move on the water, in addition to flying like a conventional quad-copter. This article clarifies the behavioral objectives, structural design, basic control mechanism of the ring-shaped robot, and examples of 3D measurements.

This paper presents a design methodology for humanoid upper limb based on human anatomy. Kenshiro is a full body tendon driven humanoid robot and is designed from the data of average 14 year old Japanese boy. The design of his upper limb is realizing detail features of muscles, bones and the adhesive relation of the two. Human mimetic design is realized by focusing on the fact that joints are being stabled by muscles winding around the bones, and by accurately mimicking the bone shape this was enabled. In this paper we also introduce details of mechanical specifications of the upper limb. By having muscles, bones, and joint structures based on human anatomy, Kenshiro can move flexibly. The use as human body simulator can be expected by measuring sensor data which can correspond to biological data.

In the case of object manipulation by a humanoid robot, it is important to deal with contact states between objects, a robot, and an environment both to avoid falling down and to achieve objective manipulations. We propose a method to describe and uniformly execute various object manipulations by a humanoid robot. In description, we focus on the contact states and define manipulation phases according to the contact states. In execution, the humanoid's controller autonomously switches manipulation phases and substantiates the contact-force controller. According to switching of the manipulation phases, the humanoid's manipulation system switches the inputs for the contact-force controller, which includes the estimation of object's information and motion generation. We evaluated our proposed system through experiments in which the HRP-2 robot manipulates four objects without information about the objects' masses and necessary operational forces.

This paper describes about a method of clothing classification using a single image. The method assumes to be used for building autonomous systems, with the purpose of recognizing day-to-day clothing thrown casually. A set of Gabor filters is applied to an input image, and then several image features that are invariant to translation, rotation and scale are generated. In this paper, we propose the descriptions of the features with focusing on clothing fabrics, wrinkles and cloth overlaps. Experiments of state description and classification using real clothing show the effectiveness of the proposed method.

Many of today's mobile robots are supposed to perform everyday manipulation tasks autonomously. However, in large-scale environments, a task-related object might be out of the robot's reach. Hence, the robot first has to search for the object in its environment before it can perform the task. In this paper, we present a decision-theoretic approach for searching objects in large-scale environments using probabilistic environment models and utilities associated with object locations. We demonstrate the feasibility of our approach by integrating it into a robot system and by conducting experiments where the robot is supposed to search different objects with various strategies in the context of fetch-and-delivery tasks within a multi-level building.

Pushing heavy and large objects in a plane requires generating correct operational forces that compensate for unpredictable ground-object friction forces. This is a challenge because the reaction forces from the heavy object can easily cause a humanoid robot to slip at its feet or lose balance and fall down. Although previous research has addressed humanoid robot balancing problems to prevent falling down while pushing an object, there has been little discussion about the problem of avoiding slipping due to the reaction forces from the object. We extend a full-body balancing controller by simultaneously controlling the reaction forces of both hands using dual-arm force control. The main contribution of this paper is a method to calculate dual-arm reference forces considering the moments around the vertical axis of the humanoid robot and objects. This method involves estimating friction forces based on force measurements and controlling reaction forces to follow the reference forces. We show experimental results on the HRP-2 humanoid robot pushing a 90[kg] wheelchair.

Humanoid robots working in a household environment need 3D geometric shape models of objects for recognizing and managing them properly. In this paper, we make humanoid robots creating models by themselves with dual-arm re-grasping (Fig.1). When robots create models by themselves, they should know how and where they can grasp objects, how their hands occlude object surfaces, and when they have seen every surface on an object. In addition, to execute efficient observation with less failure, it is important to reduce the number of re-grasping. Of course when the shape of objects is unknown, it is difficult to get a sequence of grasp positions which fulfills these conditions. This determination problem of a sequence of grasp positions can be expressed through a graph search problem. To solve this graph, we propose a heuristic method for selecting the next grasp position. This proposed method can be used for creating object models when 3D shape information is updated on-line. To evaluate it, we compare the result of the re-grasping sequence from this method with the optimal sequence coming out of breadth first search which use 3D shape information. Also, we propose an observation system with dual-arm re-grasping considering the points when humanoid robots execute observation in the real world. Finally, we show the experiment results of construction of 3D shape models in the real world using the heuristic method and the observation system.

In order to know human dynamics, humanoid as a human body simulator is increasing its importance. Such humanoid is expected to have human musculoskeletal structure as close as possible. From this viewpoint, we are trying to create new musculoskeletal humanoid which has detailed human imitating structure, such as bi-articular muscle, muscle arrangement, joint structure and so on. In this paper, we address the design of lower thigh. The concepts of the thigh include leg configuration, new knee joint and link, and artificial muscle arrangements, especially knee joint structure imitating human flexible motion. The knee joint has yaw axis DOF and its locking mechanism which is usually simplified in robotics. Finally, we conduct extension, flexion and rotation as basic experiment to confirm the joint characteristics. Also, we conduct rotation experiment in the ground state to confirm the contribution of yaw axis DOF for human-like motion.

To design a robot with humanlike body structure, this paper presents a design methodology for a humanoid upper limb by tendon driven system. We newly designed an upper limb and rib cage like thorax for a musculoskeletal humanoid robot, based on the knowledge of anatomy. The robot consists of muscle, bone, and joint structure based on human and is expected to move flexibly and dynamically. This paper describes how to design such an upper limb and proposes the key mechanical design points, which is “rib surface thorax”, “muscle cushion”, “planar muscle”, and “open type ball joint”. To show that these mechanisms is effective in making a musculoskeletal humanoid robot, we examine the motion range of the robot. One of our goals is to enable robots to do the same movements as humans do through mimicking the human body structure, finding some important elements of human nature. This robots explained in this paper is the prototype for a new life size robot “Kenshiro” project.

We have been developing and studying musculoskeletal humanoids. Our goal is to realize more human-like humanoids which can do natural and dynamic motions as well as humans. Especially motions with complex contact with environments, which is jumping, running, catching a ball, and landing on one's hands, etc, are difficult to be achieved by humanoids. To achieve that motions, robots have to absorb shock force so as not to break ones' body structures, such as gears, motors, body links and so on. Musculo-skeletal humanoids are suited for these situations, because they can easily have mechanical flexibilit for shock absorption by adding elastic units, which is nonlinear spring units, to its own tendons. In this paper, we propose shock absorption methods by musculoskeletal humanoids, which uses its own mechanical flexibilit and simple refle of each muscles based on tension sensor. This strategy is inspired by human's motion control, and it can achieve shock absorption tasks without any fast sensor feedback controls and any prediction ocntrols used by conventional robots with rigid bodies. We chose a catching a ball task as an example of complex contact motions, implemented the proposed strategy to musculoskeletal humanoid Kenzoh and confirme the feasibility of proposed method by actually catching a ball demonstration.

Manipulation of structured objects connected to the environment by a kinematics chain involves two problems: (a) The objects have movable directions and unmovable directions. An undesired reaction force in the unmovable directions prevents a robot from successful manipulation; (b) The reaction forces from the objects could fluctuate during manipulation. Related works have enabled robots to manipulate objects by integrating position control in movable directions and force control in unmovable directions at the hands. However, in the case of a humanoid robot, too large undesired reaction forces in movable directions cause the robot's falling down and slipping. In this paper, we propose a controller system controlling reaction forces at the hands and successively updating reference forces based on reaction forces. For problem (a), we apply force control both to the movable and unmovable directions in order to satisfy both maintaining full-body balance and achieving manipulation. For problem (b), the update of the reference forces enables the humanoid robot to adapt to fluctuation of the reaction forces. We show experimental results on the cmanipulating four doors and a drawer.

A walking biped robot is required to change its walking direction and speed with minimal delay and not to tip off even when subjected to an unexpected external force. A major drawback of Zero Moment Point (ZMP) based online walking pattern generation methods is that arbitrary ZMPs cannot be achieved without divergence of the Conter of Mass (CoM). In this paper, we propose a new online walking trajectory generation method that utilizes nondivergence conditions of ZMP-CoM in a successive manner. This method enables changes in the walking direction and speed of a robot and push recovery under an unknown external force in unifie form. Experiments confir walking change of up to 4.05 km/h in the speed and changes in the walking direction with minimum delay and successful push recovery under an implse of 22 Ns.

In this paper we propose a “Direct-Mapping Method” and describe a wearable device which has a similar muscle arrangement as a tendon-driven humanoid robot. This device has linear-encoders as many as the muscles of the robot to control, and they are arranged like the robot's muscle alignment. Tendon-driven humanoids have advantages of their human size, multiple Degrees of freedoms and their musclo-skeletal structure, therefore they are able to make more human-like poses than shaft-driven humanoid robots. If we can teach human poses to them, they become more human-like so that we feel a sense of familiarity with robots. However, because of their complex structure, sometimes they make incorrect poses. There must be a feedback system to control their bones in detail. To make human-like pose naturally and to make precise motion by feed back to the user, the best way may be a wearable device. With this device, we can control tendon-driven humanoids without making their simulation model in the computer. We discuss the hardware settings of the device and the controlling system in this paper. Experiments of controlling a tendon-driven humanoid were conducted to demonstrate the effectiveness of the device. When controlling the robot, a situation that some muscles are loose or too tight occured. However, this device's behavior-teaching framework contributes to detailed robotic movement in the future.

Many countries around the world face three major issues associated with their aging societies: a declining population, an increasing proportion of seniors, and an increasing number of single-person households. To explore assistive technologies that can help solve the problems faced by aging societies, we have tested several information and robot technologies. This paper introduces research on a home-assistant robot, which improves the ease and productivity of home activities. For people who work hard outside the home, the assistant robot performs chores in their home environment while they are away. A case study of a life-sized robot with a humanlike functional body performing daily chores is presented. An integrated software system incorporating modeling, recognition, and manipulation skills, as well as a motion generation approach based on the software system, is explained. Moreover, because housekeepers perform chores one after another in their daily environment, we also aim to develop a system for continuously performing a series of tasks by including failure detection and recovery.

A humanoid robot working in a household environment with people needs to localize and continuously update the locations of obstacles and manipulable objects. Achieving such system, requires strong perception method to efficiently update the frequently changing environment. We propose a method for mapping a household environment using multiple stereo and depth cameras located on the humanoid head and the environment. The method relies on colored 3D point cloud data computed from the sensors. We achieve robot localization by matching the point clouds from the robot sensor data directly with the environment sensor data. Object detection is performed using Iterative Closest Point (ICP) with a database of known point cloud models. In order to guarantee accurate object detection results, objects are only detected within the robot sensor data. Furthermore, we utilize the environment sensor data to map out of the obstacles as bounding convex hulls. We show experimental results creating a household environment map with known object labels and estimate the robot position in this map.

Many humanoids have been developed, but more complicated and flexible humanoids must be developed, in order to realize more natural and various motions like humans. However, it is difficult to measure directly joint posture in the multi-dofs joint of such robots (e.g. a hip spherical joint) because of its complicated structure. This paper describes an estimation method for tendon-driven joint postures of these complicated multi-dofs joints, only based on information of muscles' length relative displacement data during joints movement. We regard this posture estimation problem as a pattern matching problem in the mapping space from joint posture to muscle lengths and solve this problem by using very simple searching algorithms. Furthermore, this paper describes a strategy for deciding muscles motor command to acquire its joint proprioception by the proposed joint posture estimation method only based on information of muscles' length relative displacement data. Finally, we confirmed the feasibility of the proposed estimation method by applying this algorithm to the real tendon-driven musculoskeletal humanoid Kojiro.

This paper describes a 3D textured map generation method for autonomous vehicle in urban outdoor environment, where GPS signals can not be reached. Constructed map will be used for short cycle and accurate localization and for obstacle detection using onbody laser scanner. In order to build dense 3D polygon map, planar feature of laser scanner input is extracted. They are associated and transformation matrices in between each scan point were iteratively solved. Aligned points were converted into texture mapped 3D polygons. 400×350[m] area in Univ. of Tokyo were scanned at 59 scan points, and 3D polygon map consists of 14M polygon were obtained. Experimental results of localization and autonomous path following two-wheeled inverted mobile robot PMR are shown.

We propose a strategy for a robot to operate in an environment with movable obstacles using only onboard sensors, with no previous knowledge of the objects in that environment. Movable obstacles are detected using active sensing and a color range sensor, and when an obstacle is moved, the perception of the environment is reconstructed. Active sensing is defined as the classification of an object as either movable or static after the robot tries to push the object using its arm. This classification is collectively based on force sensor inputs, joint angles, and color range sensor inputs. In order to gather information from the environment, we use a color range sensor consisting of a TOF (Time of Flight) range sensor and conventional stereo cameras. Finally, we show experimental result in the environment with movable obstacles such as a table and chairs. Humanoid robot HRP-2 detects that a chair is a movable obstacle, moves the chair to clear a path to its goal, and then reaches the goal.

In this paper we propose a new method to manipulate heavy objects for a humanoid robot. In this method the manipulation strategy is determined based on on-line estimation of the operational force. We integrate these functions with a real-time controller that controls the external force and maintains full-body balance. The feature point of our work is that since a full-body control system includes switching of the manipulation strategy based on the operational force estimated on-line the system enables a humanoid robot to manipulate heavy objects as well as light objects. The effectiveness of our whole system is confirmed in our experiments, in which a humanoid robot manipulates up to 12[kg] while estimating the object's weight.

The high power ability of humanoid is desired for application of nursing or running or jumping motions. Achievement of the actuator of light and powerful equivalent to humans is required. In this paper, we propose a method to extract inherent performance from motors by an active temperature control. The method safely improves the output of motors. The active temperature control is achieved by combining the estimation of an internal temperature of the motor with the forced cooling by liquid. We also developed high power motor drivers for the proposed method. An experiment of a high power joint test bench is shown. In this paper, we show a high power prototype biped robot for application of nursing, running or jumping motions. High power actuator system and robust internal body network are developed for high power robot. Basic demonstration experiments of high power motion are shown.

This paper describes a software system integration of daily assistive robots. Several tasks related to cleaning and tidying up rooms are focused on. Recognition and motion generation functions needed to perform daily assistance are developed, and these functions are used to design various behaviors involved in daily assistance. In our approach, the robot behaviours are divided into simple units which consist of 3 functions as check/plan/do, it provides us with high reusable and flexible development environment. Because sequential task execution can be achieved only after functions about failure detection and recovery, we also try to implement such functions in keeping with this approach. In addition to using simple behavior unit, multilayer error handling is effective. Experiments doing several daily tasks with handling daily tools showed the effectiveness of our system.

Model-based approaches in recognition and planning of robots work effectively, and these approaches can apply to model-less situation using autonomous model construction by an agent. There are problems about segmentation or shape fitting of various objects with different scales or shapes. In this paper, we construct a Head-mounted 3D multi sensor for 3D environment modeling and propose a method of 3D reconstruction for various objects using intentional behavior of human.

This paper describes a novel method of path following for wheeled mobile robots. A number of virtual manipulators mounted on the robot are used to make the robot poses which satisfy both of path following and collision avoidance. This method can calculate robot poses by using a single jacobi matrix despite of coping with several manipulators. We also present about the method of motion stabilization by using a singularity-avoiding criteria. Effectiveness of the proposed method is proven by means of simulation in both cases of omnidirectional wheelbase and 2 wheeled mobile platform.

In order for robots to be able to assist humans at a very close distance, robots should allow contacts occurred at many places and deal with them. For realizing such functions, robots should have whole-body soft sensor exterior for preparing contacts against almost every body parts since it is difficult to limit where to be touched from humans. Although several groups have developed humanoid type robots which have soft tactile sensors, most of them detects distributed 1-axis forces and that is not sufficient when the robot and a human have contacts very closely. In addition to that, there is no consideration in the previous studies about humans' movement during close contact with the robots. In order to solve these problems, we have newly developed a humanoid with soft 3D-deformable sensor flesh and automatic recoverable mechanical overload protection mechanism. Soft 3D-deformable sensors are implemented by molding the infrared light receiving devices into the urethane cube and by detecting the changes of the output voltage of the devices during the deformation. On the other hand, automatic recoverable mechanical overload protection mechanism is implemented by small mechanical torque limiters and monitoring system for embedded potentiometers in the torque limiters. Also detail implementation of the embedded electric system and the overall software structure is described in this paper.

In this paper we present a new method for picking up dishes based on active groping. Though a bird's eye view is commonly used to recognize dishes, this method tends to produce errors in the presence of large occlusion, A multisensory robot hand can be used in a kitchen environment to probe and grasp dishes that are placed close to each other. Though sensing is an effective method, it is difficult to recognize a dish globally by this method because of the locality of measurement. To solve this difficulty, we propose a sensing method to aquire the geometric information about dishes by tracing their surface. We demonstrate that a robot can pick up three types of dishes on the basis of this active groping strategy by conducting an experiment.

This paper describes a navigation system in a dynamic environment for a two-wheeled inverted pendulum mobile robot, PMR. Our system is organized by localization, detection and tracking of pedestrians, and trajectory planner. The localization is robust to effects of moving obstacles and pitching movements of the robot, and the trajectory planner creates a path with a certain smoothness considering movements of pedestrians. In addition, the planner introduces strategies to avoid pedestrians to be friendly to pedestrians around the robot. Besides, our system can run on two laptop PCs in real time. Finally, we show experimental results as well.

A method of giving intelligence to control robots by human navigations is one of appropriate solution to respond to a lot of unforeseeable situations, and making movement of complicated robots in contact condition with surroundings is not easy. In this research, a bilateral wearable device using stiffness adjustable muscle actuator modules is developed in order to control robots, feel external force of robots and teach motion that contact on surroundings. For an experiment it was realized maneuvering a musculo-skeletal humanoid, it could feedback humanoid's external force to manipulator and its force feedback by bilateral control leads manipulator to do safety contact motion.

In some important robotic human daily life support applications, such as housework or nursing care, soft sensor exterior is needed for allowing close contacts against humans or environments. Although there are many studies on tactile sensing elements themselves and some of them realize the mechanical softness of the element, most of their studies just propose their detection theory with their sensor element prototypes and it is difficult to use them for developing soft tactile exterior. Therefore a soft sensor exterior embedded with multi-axis deformation sensor is developed in this paper. The most important feature for this sensor is that it can detect multi-axis deformation, and the sensor element itself has enough softness to deform with the outer exterior. The developed sensor is constructed by molding a soft urethane foam with infrared LEDs and corresponding phototransistors. It detects 3D deformation of the sensor as voltage changes at each phototransistors. Furthermore, a prototype of soft sensor exterior embedded with the developed sensing elements is realized and we confirmed that the 3D deformation, such as stroking, pinching, and pushing, can be detected using the developed sensor exterior prototype.

Humanoid robots for home daily assistance need to have an autonomous behavior selection system. To realize this, situation recognition capability is important. In this paper, we propose a situation recognition system where the use of daily life sounds enables to recognize situations difficult to understand using only visual sensor data and where the use of time series of information enables robust situation recognition. We apply cepstrum feature for recognition of daily life sounds and Dynamic Bayesian Networks(DBN) for robust situation recognition. As an example of situation recognition, we show some experiments and results targeting some situations that a human is in a kitchen.

Methods for a daily assistive humanoid robot to manipulate and recognize the objects incorporating joints and learn the manipulation knowledge are presented. It is necessary for humanoid robots to use the objects incorporating joints such as some furniture and tools to provide daily assistance. We have been tried to make an integrated humanoid robots recognition and manipulation system of the objects and tools in the real world. We extend the system for the objects incorporating joints. In this paper, a recognition system in which the robots recognizes the objects incorporating joints by the visual 3D object recognition method with multi-cue integration using particle filter technique and a manipulation system of them are shown. The search areas of the joints are automatically generated based on the manipulation knowledge. We present three key techniques to recognize and manipulate the objects incorporating rotational and linear joints. 1) Knowledge description for manipulation and recognition of these objects; 2) Motion planning method to manipulate them; and 3) Recognition method of them closely related to the manipulation knowledge. Moreover, a method for a person to teach the handle, one of manipulation knowledge, visually to the robot is shown. Finally, a daily assistive task experiment in the real world using these elements is shown.

We are developing the novel musculoskeletal humanoid that is able to do coordinated motion with arms, legs and spine. In this research, we present how to realize largely bending/extending motion while standing using spine and hip joints of a fullbody musculoskeletal humanoid, which has a complicated body like a human, such as over 90 muscles and about 60 DOFs. And we propose how to generate coordinated motions of spine and hip joints and how to control such a complicated humanoid in order to keep standing during its motion. Furthermore, we confirmed that this robot could bend/extend its spine and hip joints coordinately while standing by some experiments.

In this paper, we treat with wheelchair support by a life-sized humanoid robot. It is quite essential to integrate whole-body motion, recognition of environment and human-interface behind the user in order to achieve this task. Contributions of this paper is whole-body control including pushing motion using the offset of the ZMP and observation of the attitude outlier, recognition of the wheelchair using particle filter and human-interface behind the person using face detection and recognition of gesture.

This paper describes daily assistive task experiments that conducting on the HRP2JSK humanoid robot. We present overall action and recognition integrated system design to realize daily assistive behaviors autonomously and robustly, along with the demonstration that the HRP2JSK pours tea from a bottle to a cup and wash it after human drink it. To obtain autonomy and robustness, visual recognition and behavior control through perception information are important.

Humanoid robots are still in a process of the evolution. We are trying to develop a next-generation which have safety, flexibility and multiplicity of uses. A shoulder structure is one of the keystone, because it controls arms and end effectors and it have direct contact with humans and daily life environments. We have designed new shoulder structure following the scapula-collarbone structure for a muscle-driven humanoid robot named ldquoKojirordquo (Fig. 1)[1]. We consider that the scapula-collarbone structures have the following merits, 1)wide range of movement, 2)flexible shoulder like humans, 3)safety from the driving system, 4)allowing space for a lot of parts inside the thorax. The shoulder structure is consisted of scapula, collarbone, upper-arm and thorax. We have devised the spherical shape thorax and scapula structure which realizes desired scapula motions. In this paper, we describe the design process and operation test of the shoulder structure.

High physical ability of humanoid robots is desired for application to nursing care. Light and powerful actuators are required to realize the high-power performance. In this paper, we propose a method to bring out maximum performance of electric motors aggressively. The technique of motor core temperature estimation and control improves the motor power performance dramatically but safely without motor burnout. We have developed high power motor driver modules for the proposed method and equipped them into our humanoid robot and prototype jumping robot. High-power performance experiments with the robots demonstrate our method.

This paper proposes the new robot programming environment in which robot motion programming environment and dynamics simulator are integrated. This allows robot motion programs to include simulation descriptions. Additionally, a new implementation of simulation that is composed by simulation modules is presented, on the other hand, conventional simulators are monolithic and implanted every function. This makes it difficult to add new simulation functions such as new sensors on a simulator by its users. In the new method, the users of the environment can add new modules easily. The simulation function of this system is evaluated by showing new robot motion simulations like brooming, seesaw and so on. The experiment that shows how the simulation embedded brain changes the motion planning of block moving problem is illustrated in the end this paper.

In this paper, the development of a robot which has "flesh" made of soft polyurethane foam is presented. In order for a robot to behave in contact with an environment, people and itself, it is necessary to have not only rigid bone but also soft flesh. We developed a small humanoid robot "macra" which has thick soft flesh with distributed tactile sensors. A comparative discussion about possible materials of soft cover is described, and then thermal design was explored. A problem about movability of joints and implementation of tactile sensing was described. In the last part of this paper, several examples of advantages which are derived from soft polyurethane foam exterior parts with distributed three-axis force/torque sensors are shown.

This paper presents how to develop a lower body with both redundant and high-powered actuator systems toward the next version of a fully tendon-driven musculoskeletal humanoid, which will be able to realize various flexible motions using whole body. The newly developed lower body has 44 actuators and each actuator can generate 25[kgf] muscle force continually. We propose the new design method of hollowing out the parts assembly areas, which realizes easy design of the skeleton with massive actuators. Finally we presented an experiment on the hip muscles tension and redundancy performances during the hip joint rotation motion.

A vision based object recognition subsystem on knowledge-based humanoid robot system is presented. Humanoid robot system for real world service application must integrate an object recognition subsystem and a motion planning subsystem in both mobility and manipulation tasks. These requirements involve the vision system capable of self-localization for navigation tasks and object recognition for manipulation tasks, while communicating with the motion planning subsystem. In this paper, we describe a design and implementation of knowledge based visual 3D object recognition system with multi-cue integration using particle filter technique. The particle filter provides very robust object recognition performance and knowledge based approach enables robot to perform both object localization and self localization with movable/fixed information. Since this object recognition subsystem share knowledge with a motion planning subsystem, we are able to generate vision-guided humanoid behaviors without considering visual processing functions. Finally, in order to demonstrate the generality of the system, we demonstrated several vision-based humanoid behavior experiments in a daily life environment.

We are trying to develop an innovative humanoid which learns from the structure of a human shoulder, especially the bladebone and collarbone. We consider that the bladebone-collarbone structures have the following merits, l)wide range of movement, 2)having the shoulder structure built on to the outside of the chest, allowing space for a lot of parts inside the chest, 3) flexible shoulder like humans. In contrast, it is difficult to control the shoulder structure because it have multiple degrees of freedom and extraordinary physical constraints. An aim of our research is to create a repeatable motion generation system of the bladebone-collarbone structures for real musculo-skeletal humanoid robots. In this paper, we describe the design process of the bladebone-collarbone structures for a muscle-driven humanoid robot, and the simulation system considering geometrical constraints of the structures. Then, we show a motion experiment using the shoulder structures of the humanoid, based on the simulation system we presented.

Shape-memory alloy(SMA) is a suitable actuator for a micro mobile robot due to its size and weight. However, length of SMA actuators is known to be nonlinear and hysteretic. We can control SMA’s shape analogously to its electrical resistance. This paper presents a platform to construct high degree of freedom robots with tendons of SMA actuators. We incorporate coil shape SMA as variable length tendons of tendon-driven robots. By controlling electrical resistance of SMA tendons, we can control posture of the robot structure in model-based method. A method to resolve electrical resistance for control from given posture of a high degree of freedom robot by using model of SMA and the robot is proposed. A small electrical system for high D.O.F. robots is also developed. As an application of our system, we show a four-legged robot with 16 D.O.F. joints and 24 SMA tendons.

This paper presents a work toward the old dream of the housekeeping robot. One humanoid robot will cooperate with the user to cook simple dishes. The system combines predefined tasks and dialogues to find a plan in which both robot and user help each other in the kitchen. The kitchen problem allows the demonstration of a large variety of actions, and then the necessity to find and to plan those actions. With this problem the task planner can be fully used to enhance the robot reasoning capacity. Furthermore the robot must also use motion planning to have general procedures to cope with the action planned. We focus on the planning problems and the interactions of these two planning methods

This paper presents the concepts, issues on development, and future perspectives of a novel musculoskeletal humanoid `Kotaro'. The specifications and advantages of Kotaro include muscle-driven endoskeletal structure, multiple degrees of freedom, variable physical softness, the multiple-joint spine, easily configurable muscles, distributed onbody controllers, various sorts of many sensors including eyes, ears, and whole-body tactile sensors, and so on. These characteristics would achieve a flexible body, litheness of motions, safety, and adaptability and applicability to diverse tasks which would very often appear in human's daily lives

Memorization, abstraction, and generation of a time-series of sensors and motion patterns are some of the most important functions for intelligent robots, because these memories are useful for situation recognition and behavior decision making. In conventional research, recurrent neural networks are often used for such memory functions. However, they cannot memorize a lot of patterns and its learning algorithm is unreliable. In this paper, we propose a method for the induction of behavior and situational estimation based on hidden Markov models, which is currently one of the most useful stochastic models. With the proposed method, we show the feasibility of: (1) Both recognition and association are executed at the same time, and (2) A multiple degrees of freedom and multiple sensorimotor patterns are acceptable

Online self collision detection system for humanoid robots is an essentially important function for developing sensor based behaviors without worrying about breaking hardware. In this paper, we propose a practical and real-time self collision detection system for humanoid robots that satisfy both enough a range of movement and a safety margin. Previous researches usually add a safety margin around each link to cope with errors in both modeling and control. However this margin significantly decreases a range of movement of a joint, especially a compound joint, a joint between adjacent links and composed of 2 or 3 revolute joints whose axis intersect in a same point. In order to gain enough a range of movement and safety margin, we developed hybrid approach that uses both table based collision checking for compound joints and online geometrical model checking with a simplified link shape for other joints. We have experimentally evaluated our self collision detection system using a HRP2-JSK humanoid robot. Our demonstration shows that the robot automatically stops its motion when self collision occurs

In this paper, we describe the real-time and precise self collision detection system that does not reduce the number of polygons and checks more than 100 collision pairs in real-time by using AABB based collision detection libraries. Previous researches on collision detection of humanoid robots which reduce collision pairs or simplify a shape of a robot has disadvantages such as increasing the dangerousness or decreasing range of movement. However our self collision detection system uses detailed geometric model and collision pairs as many as possible. We have experimentally evaluated collision detection libraries on a real-time self collision detection application of a humanoid robot. This experiment suggests that AABBs based method is much faster than conventional OBBs based method. Finally, we demonstrated real-time collision detection and avoidance function that automatically stops entire motion if self collision occurs using HRP2 humanoid robot.

This paper describes vision-based 3D walking system of a humanoid robot by combining a precise 3D planar surface detection method and a practical 3D footstep planner method. The walking control system requires vision system with 10[mm] accuracy. Then we developed the precise 3D planar surface recognition system by combining the 3D Hough transformation method and the robust estimation method. We also developed practical 3D foot step planner by considering kinematics and dynamics restriction of robot hardware. Finally, we realized vision based 3D walking experiments that a humanoid robot steps upon an unknown obstacle are shown.

We have been studying about muscle-tendon spined humanoid robots for aiming a novel humanoid robot that has modifiable mechanical flexibility, safety and wide variety of postures. This paper describes a new concept of reinforceable-muscle humanoid and the development of a new musculo-skeletal humanoid robot named "Kenji" that is a prototype muscle-driven humanoid next to previous developed humanoid "Kenta". Kenji has 140 actuators in total and numerous multi-modal sensors. This paper also presents some experiments that show the feasibility of the robot.

We are trying to create an innovative humanoid which has increased flexibility by imitating the structure of a human shoulder, especially the shoulder blade. We designed a flexible shoulder structure for a muscle-driven humanoid robot which has the following merits, 1) wide range of movement and flexible shoulder like humans, 2) having the shoulder structure built on to the outside of the chest, allowing space for a lot of parts inside the chest. We developed a prototype humanoid robot named "Blade" which has a musculo-skeletal humanoid torso with the shoulder blade, driven by motor-actuated tendons. This paper describes the advantages of the structure of the shoulder blade, the development of a humanoid with the shoulder blade, and some experiments showing its range of movement using the prototype humanoid robot we presented.

This paper describes the design and development of system software for humanoid robots such that researchers who specialize not only in biped walking but also in various fields are able to use humanoid robots as a research tool. For this purpose, the system for a humanoid must integrate and organize each subsystem such as control, recognition, dialogue, planning and so on, and it must provide efficient full-body motion control by specifying fewer degrees of freedom than all joints. Our system design provides a common interface among subsystems by implementing each function as a method call through a three-dimensional model of the robot for good integration, and it also provides a motion planning technique based full-body posture sequence and walking pattern generation. Finally, we show integrated behavior experiments with vision, planning and motion control using the developed system software for a life-sized humanoid robot, HRP2.

In this paper we present "hand-centered" motion generation and control of whole-body motion for a humanoid robot, which enable the robot to carry out reaching and walking simultaneously. Hand-centered whole-body motion control is defined as a real-time control scheme in which the hand motion (input) leads to the whole-body motion (output). The hand trajectory is described in the absolute coordinate system (world frame) and the torso trajectory follows the hand trajectory. This method enables operation of the hand of a walking humanoid robot, and autonomous generation and control of the whole-body motion based on the hand velocity vector using sensor feedback. We have implemented the method as a hand-centered whole-body motion control system on the existing humanoid robot system "H7". Some experimental results including both reaching and walking motions are shown to demonstrate the effectiveness of the proposed method.

In this paper, we focused on concepts of behavior modifier. We aim to build a system, which acquires concepts of behavior modifier and applies it to motion generation of robots. For a form of motion representation, an existing research achievement, which statistically abstracts motion with known sets of motion examples and represents motions as points in a space, was adopted. Our system uses tendency of motions with adverbial modifier in the space to represent concepts of behavior modifier. The system has a simple representation form and has interactiveness what has potential advantage of ability to be applied to various motion concepts.

In this paper, we propose an approach where robots store shared experiences between human and the robots, and also show that the shared experiences act as an important role not only for the coexistence of the human and robots but also for the realization of the supporting ability of the robots. Enormous amount of shared experiences are effective for the understanding of users under uncertain and incomplete conditions, as well as communication among human. We propose the importance of the shared experiences for real world intelligence, and show a development research of the infrastructure for the storage of the share experiences and application method.

We propose a concept of reinforceable-muscle humanoid with hyper parallel muscle-tendon systems. By the facilitation of changing the assignment of actuators, we can easily strengthen a specific part of robot body, while changing the arrangement of actuators of current humanoid robots is quite difficult. By developing muscle units in each of which a motor and sensors are integrated, and by adopting musculo-skeletal structure, the rearrangeability of muscles has been realized. This paper describes the concept, prototype design and implementation of the reinforceable muscle humanoid, and the evaluating experiment by a musculo-skeletal humanoid is presented.

We propose the portable situation reporting system by a small robot for daily life, which demands the rapidly system reconstitution for daily life support. In this paper, we established a system by an attachment mechanism assisted the robot system based on device-distributed approach and the patterned processing by the tree structuring of sensor information processing. The attachment mechanism facilitates the hardware system reconstitution and the patterned processing simplifies the software system restructuring. We evaluated the reconfigurable system not to re-write programs and judged the effectiveness of the system by the experiments.

In this paper, we describe a planner for a humanoid robot that is capable of finding a path in an environment with movable objects, whereas previous motion planner only deals with an environment with fixed objects. We address an environment manipulation problem for a humanoid robot that finds a walking path from the given start location to the goal location while displacing obstructing objects on the walking path. This problem requires more complex configuration space than previous researches using a mobile robot especially in a manipulation phase, since a humanoid robot has many degrees of freedom in its arm than a forklift type robot. Our approach is to build environment manipulation task graph that decompose the given task into subtasks which are solved using navigation path planner or whole body motion planner. We also propose a standing location search and a displacing obstacle location search for connecting subtasks. Efficient method to solve manipulation planning that relies on whole body inverse kinematics and motion planning technology is also shown. Finally, we show experimental results in an environment with movable objects such as chairs and trash boxes. The planner finds an action sequence consists of walking paths and manipulating obstructing objects to walk from the start position to the goal position.

Since an environment or body states of robots are very changeable, robotic imitation systems should have ability to develop their behaviors by themselves. For the development of the imitation behaviors, we assume that attention during imitation is the key information. In order to realize such imitation systems with evolving ability, the idea of reinforcement learning system based on the attention structure during imitation has been presented in this paper. First, for describing the attention during imitation behaviors, we define the term 'sensor-action attention pair' as the pair of the most important sensor information and the focused body parts during that behavior. Second, we introduce R-learning, the reinforcement learning method for continual tasks such as imitation behaviors. Third, the method to design the state-action space and the reward function based on the sensor-action attention pair is proposed. At last, for the confirmation of the function of the proposed imitation behavior system, we have done some experiments using actual humanoid Kenta. In those experiments, Kenta can develop imitation behavior that imitates the hand position of the human.

We present an integrated humanoid locomotion and online terrain modeling system using stereo vision. From a 3D depth map, a 2.5D probabilistic description of the nearby terrain is generated. The depth map is calculated from a pair of stereo camera images, correlation-based localization is performed, and candidate planar walking surfaces are extracted. The results are used to update a probabilistic map of the terrain, which is input to an online footstep planning system. Experimental results are shown using the humanoid robot H7, which was designed as a research platform for intelligent humanoid robotics.

We present an online algorithm for planning sequences of footstep locations that encode goal-directed navigation strategies for humanoid robots. Planning footsteps is more general than most existing navigation methods designed for wheeled robots, since the options of stepping over or upon obstacles in a cluttered terrain are available. Given a discrete set of plausible footstep locations, a forward dynamic programming approach is used to compute a footstep sequence to a specified goal location in the environment. Heuristics designed to minimize the number and complexity of the step motions are used to encode cost functions used for searching a footstep transition graph. If successful, the planner returns an optimal sequence of footstep locations according to the cost functions and plausible sets of footstep locations defined. We show results from an experimental implementation of the algorithm running on the H7 humanoid robot. Using a stereo vision system to sense obstacles in the immediate environment and identify a target goal location, the robot updates the current optimal footstep sequence to the goal from its present location.

Behavior imitation ability will be a key technology for future human friendly robots. In order to understand the principles and mechanisms of imitation, we take a synthetic cognitive developmental approach, starting with minimum components and create a system that can learn to imitate others. We developed a visuo-motor neural learning system which consists of orientation selective visual movement representation, distributed arm movement representation, and a high-dimensional temporal sequence learning mechanism. The vision and the movement representations model the findings in primate brain, i.e. macaque area MT(or human area V5) and the primary motor area. The learning mechanism is inspired by the finding that there are excessive connections in neonate brain. As our robot explores the visuo-motor self movement patterns, it learns coherent patterns as high-dimensional trajectory attractors. After the learning, a human comes in front of the robot showing arm movements which are similar to the ones in self learning. Although the robot has never seen or programmed to interpret human arm movement, and the detail of visual stimuli are very different, the robot identifies some of the patterns as similar to those in self learning, and responded by generating the previously learned arm movement. In other words, the robot exhibits early imitation ability based on self exploratory learning.

It is very important and hard to develop motions and behaviors for robots that have complex body and numerous actuators and sensors. The more complex a humanoid's body becomes, the more important the developing environment of the motions and behaviors will be. We have built a fully muscle-driven flexible-spine humanoid named "Kenta", which has 96 actuators and more than 400 sensors including tension sensors, posture sensors, vision sensors, and so on. This paper describes the developing environment of motions and behaviors for the robots like Kenta. A visualizable geometric model, a simulation environment transparent to the real environment, sample procedures of the motion development, and, as an example, realization of swing behavior by Kenta are presented.

We present a humanoid walking control system that generates body trajectories to follow a given desired motion online. A layered software and control architecture is used to aggregate system components and provide a framework for high-level autonomous locomotion behaviors. Walking characteristics such as desired torso movements, upper body posture and step cycles, can be specified and used to generate stable whole-body walking trajectories online. The basic architecture consists of 4 layers: footstep planning, trajectory generation, trajectory modification from sensor feedback, and joint servo control. In order to test the complete system, we implemented an experimental top-level behavior that enables the robot to track and follow a moving target based on stereo vision feedback. We explain the overall system design and components, and present results based on our current implementation using the humanoid robot 'H7'.

This paper formulates and solves the inverse dynamics problem of deformable robots made entirely of electro-active polymer gel. One of the primary difficulties with deformable robots is that they have conceptually infinite degrees of freedom. We solve this problem through the selection of an essential point to generate a desired motion. The problem is then reduced to trajectory control of a point on the robot. We have proposed dynamic models of electro-active polymers system and derived a variety of motions by applying either spatially or time varying electric fields. However, the motion control problem has not yet been investigated. We show a procedure to realize an inversion (turning over) motion of a starfish-shaped gel robots by applying both spatially varying and time alternating electric fields. This work takes the first step towards motion control of deformable robots.

This paper describes an experimental stereo vision based motion planning system for humanoid robots. The goal is to automatically generate arm trajectories that avoid obstacles in unknown environments from high-level task commands. Our system consists of three components: 1) environment sensing using stereo vision with disparity map generation and on-line consistency checking, 2) probabilistic mesh modeling in order to accumulate continuous vision input, and 3) motion planning for the robot arm using RRTs (rapidly exploring random trees). We demonstrate results from experiments using an implementation designed for the humanoid robot H7.

In this paper, the method of motion planning for the metamorphic robot which can change the rigidity and the shape of its body is described. When making motions of the metamorphic robot, we can use not only the degrees of freedom in the joints but also the deformation of the links. In making use of the deformation by the length of the shortest path, this connects both ends of the deformed link. Balance constraints which depend on the friction between the body and the environment need to be considered in order to maintain the stability of the motions. In this time, we check the stability by computing contact forces considering the moving direction of the contacts. The motion planning is presented using Linkrobot which is made of metamorphic links and can deform its whole body.

To realize humanoid robots in unknown environment, sensor based navigation system is required as one of an essential function. This paper describes vision-based navigation system for humanoid robots, which has following features: 1) To recognize floor regions from a view of vision of a humanoid robot in unknown environment, we utilized existing technique called Plane Segment Finder, which is able to extract arbitrary planner surface regions from depth image. 2) Path planning for wheeled robots usually models a robot as a 2D circle, however path planning system for humanoid robot requires capable of modeling a robot as a 3D cylinder model, convex hull model, rigid model and so on, according to a situation such as a robot carries a large object or a robot opens its arms. Finally, we show a humanoid robot HOAP-1 with enhanced stereo vision system for navigation task and a result of path planning using generated local map through stereo vision system which uses real images as an input.

We present an approach to self-collision detection suitable for complex articulated robots such as humanoids. Preventing self-collisions is vital for the safe operation of robots that generate body trajectories online. Our approach uses a fast distance determination method for convex polyhedra in order to conservatively guarantee that a given trajectory is free of self-collision. Experimental results using an online joystick control application for the humanoid robot "H7" demonstrate the feasibility and effectiveness of the method.

Addresses the extension of a humanoid's action capability by attaching toe joints. The effectiveness of toe joints is discussed in three aspects. One is utilizing it to speed up the walking, another is using it to enable a humanoid to go up higher steps, and the other is using it to whole-body action in which knees are contacting the ground. Feet with toe joints are developed for humanoid 'H6'. An experiment of the wholebody action in which knees are contacting the ground is carried out to show the usefulness of toe joints for such actions. Then the walking pattern generation system is extended to use toe joints. Using this extended system maximum speed of knee joints can be reduced at the same walking speed, and 80% faster walking speed is achieved on humanoid 'H6'.

This paper describes a design of six-axis force sensor that measures ground reaction force of human or humanoid robot. The key concept is parallel support mechanisms, that allow large torques and forces which are caused when foot is hitting to the environment. Basic concept and design of parallel support mechanisms are denoted. Finally ground reaction force measurement system for human walking, and application to humanoid robot walking are described.

This paper proposes an inverse kinematic model for deformable robots made entirely of electro-active polymer gel. The required method is to control a higher degrees of freedom than numbers of input. We (2000, 2001) have been proposed a kinematic and dynamic model of electro-active polymer system and derived a variety of motions of gel robots by applying spatially varying electric fields. However, inverse kinematic model and the method of applying time alternating electric fields have not been investigated. We challenge the tip control of gel manipulator by applying spatially uniform but time varying electric field. We show the procedure to control the tip position of a gel manipulator by dynamically and slightly changing its whole configuration. Our work is a first step towards the shape control of gel robots.

This paper describes the design of an integrated system for humanoid robotics that consists of three key components, 3D vision, motion planning and bipedal control. Layered system design is adopted to achieve concurrency as well as small latency. Implementation using our humanoid type robot H7 and experiments with this architecture are described. The H7 is expected to be a common test-bed in experiments and discussion for various aspects of intelligent humanoid robotics.

We are trying to realize a humanoid which has flexibility. If humanoids have a flexible structure, safety and posture variety can be achieved. We focus on the role of the human spine and muscle-driven system. By having a flexible spine, a humanoid will have safety and many degrees of freedom to realize a variety of postures. By driving joints by tension-controllable tendons, flexibility of the joints can be controlled. We developed a whole-body tendon-driven flexible-spine humanoid named "Kenta". This paper describes the design and control of Kenta, focusing on the design of the spine. The spine consists of ten joints, vertebrae and rubber disks, ribs, and forty muscles equipped with tension sensors. We also propose control methods of the spine. One uses a geometric virtual robot model and another is based on direct teaching. Using these methods, some whole-body motions are presented.

In this paper, a robot that can change the rigidity and shape of its body is called "metamorphic robot". So far robots have rigid bodies and have their degrees of freedom only in the joints. The metamorphic robot can change their bodies fit to the environment by deforming the limbs between the joints. To realize this robot, the softening deformable structure is developed. The structure is made of a low melting point alloy and becomes soft and deformable through the phase change induced by heating and becomes rigid by cooling. In the soft-deformable state, the robot can change the shape by pressing against objects. On the other hand, in the rigid state it is rigid enough to support the robot's body weight keeping the deformation.

This paper presents an efficient online method to generate humanoid walking motions that satisfy desired upper body trajectories while simultaneously carrying objects. A fast motion pattern generation technique that follows the desired ZMP is adopted. In order to satisfy the control input given online, subsequent motion patterns are updated and connected in a stable manner to the old ones while executing. During the creation of motion trajectories online, the commanded motion parameters are checked and modified automatically considering the performance limitations of the hardware. As an example application, we have implemented a one step cycle control system on the Humanoid H7. Experiments controlling the upper body motion and walking direction using a joystick interface are explained to demonstrate the validity of the proposed method.

This paper describes a rapid development system for humanoid vision-based behavior consisting of both real and virtual robots in a simulation environment. Previous robotics simulators were limited to verify dynamic motions such as walking, or to plan or learn in a simple environment. However, our system is able to simulate vision based behavior, i.e. motion and perception behavior, of a robot as a whole, since it can simulate both dynamics and collisions in the environment. We regard the real-time aspect of the simulation as important so that so that users can develop vision based behavior rapidly and efficiently. We employ a Common Interface API to share behavior software between real and virtual robots. Moreover, visual processing functions such as color extraction and depth map generation are available. As a result, vision based behavior consisting of local map generation, planning and navigation of the humanoid in simulation is presented. We also show that software developed in the simulation environment is applicable to real robots.

This paper describes a design of remote operation interface for humanoid robot with following three functions: (1) Body DOFs control interface through 3D robot model in a virtual environment, and it has two types of online stabilizing software for maintaining the body balance. (2) Environmental recognition display interface for sensors, sound and 3D vision. (3) Voice and face recognition interface for interacting with operator and a human in front of the robot body. Then, implementation and experiments on our humanoid robot H6 are denoted.

The H6 humanoid robot has been developed as a platform for the research on perception-action coupling in intelligent behaviour of humanoid type robots. The H6 features: 1) a body which has enough DOFs and each joint has enough torque for full body motion; 2) a PC/AT compatible high-performance on-board computer which is controlled by RT-Linux so that from low-level to high-level control is achieved simultaneously; 3) self-contained and connected to a network via radio Ethernet; and 4) dynamic walking trajectory generation, motion planning and 3D vision functions are available. The H6 can be used as a common test-bed in experiment for various aspects of intelligent humanoid robotics.

In this paper, the developmental software environment for humanoids is introduced. This environment has the following features: 1) a hardware concealment function which saves the trouble with hardware improvements; 2) an incremental expansion function which supports a long-term development by many developers; and 3) a centralised management of body information which enables the robots to share software among different bodies. These features are realized by the plug-in architecture, the universal robot model and its compilers. Using plug-ins enables one to accumulate functions developed by different developers and to switch computer interfaces of the robot body. The universal robot model and compilers promote the use of external software and share software modules among different bodies.

We present an approach to path planning for humanoid robots that computes dynamically-stable, collision-free trajectories from full-body posture goals. Given a geometric model of the environment and a statically-stable desired posture, we search the configuration space of the robot for a collision-free path that simultaneously satisfies dynamic balance constraints. We adapt existing randomized path planning techniques by imposing balance constraints on incremental search motions in order to maintain the overall dynamic stability of the final path. A dynamics filtering function that constrains the ZMP (zero moment point) trajectory is used as a post-processing step to transform statically-stable, collision-free paths into dynamically-stable, collision-free trajectories for the entire body. Although we have focused our experiments on biped robots with a humanoid shape, the method generally applies to any robot subject to balance constraints (legged or not). The algorithm is presented along with computed examples using the humanoid robot "H6".

This paper introduces and describes a novel method which enables the online generation of humanoid walking patterns that follow desired footprint locations. Online generation is realized by the dynamically stable mixture and connection of pre-designed motions. Characteristics of zero moment point (ZMP) are utilized in order to maintain the overall dynamic stability of "mixed" motions. Experiments using an online pattern generation software system and online walking control joystick interface for the humanoid robot H6 are introduced to show the validity of the method.

This paper describes the development of a plane segment finder, which is able to detect three-dimensional planar surfaces from input images in real-time. We propose an algorithm for detecting plane segments, that includes: 1) plane segment candidate extraction using the 3D Hough transform from depth map information; and 2) fitting the plane segment candidates to the depth map in order to detect the partial plane segment, since the extracted plane segment candidates are general planes, with no boundary. To achieve real-time plane segment finding system, we apply: 1) the recursive correlation method for depth map generation; and 2) the randomized Hough transform for plane segment extraction. Finally, experimental results using an implementation of our system along with a humanoid robot are shown.

This paper shows walking human avoidance and detection behavior of a mobile robot using the 3D depth flow. The 3D depth flow proposed is able to measure the 3D motion vector of every pixels between two time sequential images. First, a definition of the 3D depth flow and a simple 3D depth flow calculation method are presented. Then an implementation of the real-time 3D depth flow generation system using a standard PC is described, and the experimental results are given. Finally, as an application, the walking human detection and avoidance task using a mobile robot in real environments is shown.

This paper proposes a model for predicting the dynamic motions of elastic robots made entirely of electro-active polymer gel. The requirement is that both the active and passive deformations be described simultaneously. Active deformations arise from a generated stress, while passive deformations are due to external forces, such as gravity and friction. We represent both in the same framework, considering them as the result of either electrochemical or mechanical interaction between the gel robot and the environment. We evaluated the model by comparing the results obtained through simulations and experiments using prototype gel robots. One experiment involved hitting an external object, and the other involved bending while resting upon a surface. We present an overview of the model, with a discussion of the experimental results and future work.

We present an algorithm for planning safe navigation strategies for biped robots moving in obstacle-cluttered environments. From a discrete set of plausible statically-stable, single-step motions, a forward dynamic programming approach is used to compute a sequence of feasible footstep locations. In contrast to existing navigation strategies for mobile robots, our method is a global method that takes into account the unique ability of legged robots such as bipedal humanoids to traverse obstacles by stepping over them. Heuristics designed to minimize the number and complexity of the step motions are used to encode cost functions used for searching a footstep transition graph. We show preliminary results of an experimental implementation of the algorithm using a model of the H6 humanoid navigating on an office floor littered with obstacles.

The development of a robot which has a flexible spine is presented. By embedding a multi-DOF (degree of freedom) soft structure into a robot body as a spine, the robot can increase its ability to absorb shock and the ability to work in various environments such as narrow places or human fields, and its motion could be more natural. The developed full-body human-form robot has a five-jointed flexible spine. Each joint (vertebra) has 3 DOFs. Between each two vertebrae is a "disk" made of silicone rubber. The spine is controlled by eight tendons, whose tensions can be controlled using tension-sensors and locally distributed microcontrollers. The paper describes the development of the flexible spine and the control of the posture of the spine and body.

This paper proposes a 3D depth flow generation method which measures 3D motion vector of every pixels between two time sequential images. First, the definition of 3D depth flow and a simple method in order to generate the 3D depth flow are described. Then the implementation of a real time 3D depth flow generation system using only a PC is presented, and experimental results are given. Finally, as an application, a walking human tracking task using mobile robot is included.

This paper describes the design and implementation of real-time depth map generation system with four key issues: 1) recursive (normalized) correlation technique; 2) cache optimization; 3) online consistency checking method; and 4) applying the MMX/SSE/sup R/ multimedia instructions set. The system is implemented on a standard PC/AT hardware with simple image capture board. Implementation details and system evaluations are also denoted. Furthermore, experiments with robots in real world are shown.

Introduces and describes a kinematic model based on a chemical reaction to control robots made of soft materials, poly (2-acrylamido-2-methylpropane sulfonic acid) gel (PAMPS gel). Experiments were conducted using a prototype mechanism whose energy is supplied by applying an electric field in a surfactant solution. We have verified the validity of our model by comparing the experimental and simulation results obtained by bending a flexible strip of PAMPS gel under a uniform electric field.

We propose a behavior learning method based on Bayesian networks and experience of interaction between human and robots, which does not need a priori knowledge and can be applied to human-robot interaction models. In this method, the behavior learning based on interaction experience was established. However, developers must adjust initial sensor state of the Bayesian network according to the user preference. In this paper, we propose a new method of state space construction for user adaptation based on introspection of interaction experience using genetic algorithms. We also give two examples: 1) obstacle avoidance tasks for mobile robots; and 2) symbol grounding for natural language instruction, for realization of user's adaptation of human-robot interaction.

Physically-based simulation software is commonly used for developing and testing low-level robot control algorithms. In order to facilitate the development and evaluation of higher-level robot behaviors, broader-based simulations are needed. Examples include software for simulating 3D vision, motion planning for obstacle avoidance, and integrating vision and planning. In addition to modeling the general interaction between the robot and its environment, the software can be used as a graphical user interface for directly controlling or interacting with a robot operating in the real world. This paper describes our current efforts toward building a large-scale software simulation framework for the development and testing of high-level behaviors for humanoid robots. We view this as a potential useful tool for the visualization and development of robotic systems, as well as an interactive, task-level programming interface for robots.

A map is a description which represents the structure of an environment. In order for a robot to move from a point to another efficiently in an environment, a map of the environment is the essential information. "Omni-View Sequence" is a kind of route representation which we have developed for robot navigation in a corridor environment. It contains a sequence of omnidirectional views as the information of a route. However it cannot be regarded as a map, because the start and the goal points of the route are both fixed. In this paper, we first propose a map named "view-sequenced map" which can represents a whole corridor environment on a floor in a building. Then we describe the method for acquiring a view-sequenced map automatically based on the exploration of a robot in a corridor using both stereo and omnidirectional vision. Finally experimental results of the map acquisition and the autonomous navigation are presented to show the feasibility of the proposed map representation.

Recently, view-based or appearance-based approaches have been attracting the interests of computer vision research. Based on a similar idea, we have proposed a view-based navigation method using a model of the route called the "view sequence." It contains a sequence of frontal views along a route memorized in the teaching run, and the recognition of the environment is realized based on the matching of the current view and memorized view sequence. In this paper, we discuss the required characteristics of the view for the view sequence, and evaluate our former method of generating views. Then we confirm that the stereo disparity satisfies the requirement of the view sequence through an experiment, and the disparity view sequence is applied for outdoor navigation. The experimental results indicate such views other than normal camera images can be utilized for our view-based navigation method.

A humanoid robot "H6" is developed as a platform for the research on perception-action coupling in intelligent behaviour of humanoid type robots. The H6 has the features as follows: 1) a body which has enough DOFs and each joint has enough torque for full body motion, 2) a PC/AT compatible on-board computer which is controlled by RT-linux so that low-level to high-level control is achieved simultaneously, 3) is self-contained and connected to a network via radio ethernet, 4) a 3D vision function can be applied. The H6 is expected to be a common test-bed for experiment and discussion for various aspects of intelligent humanoid robotics.

This paper describes a vision system which recognizes 3D objects in real-time by modeling the shapes of objects and matching the generated models. We develop the following methods for practically solving important problems of integration such as the estimation of sensor accuracy as well as real-time processing: 1) we reduce the computation of signed-distance, which is necessary to apply the marching cubes algorithm, and select the optimal resolution of models to be generated using an octree, thereby enabling us to generate hierarchical mesh models in real-time; 2) we apply spin-image matching by selecting the resolution of generated models and the coarse to-fine algorithm, consequently, we are able to efficiently match multiple objects of different sizes.

It is not simple for humanoid to acquire its actions automatically because: 1) it has very wide search space due to many DOF and is not fixed on the ground; 2) complex actions are difficult to describe using indirect evaluation functions; and 3) a humanoid is required to behave naturally like a human does. Kinematics adaptation generates an action outline which is a rough action description from teacher's action (motion capture data) by absorbing a difference of structures between the human and robot. Then a target action is acquired through dynamics adaptation by searching an action which is executable dynamically. Advantages of this framework are that: 1) the target action is described in an intuitive representation, 2) the search space is reduced in the neighborhood space, and 3) the acquired action will be a natural motion.

Describes what a friendly artifact is and the development of the first prototype robot system of a friendly artifact for multiple sensored interaction. The multiple sensored interaction is indispensable to realize the recognition of scene context, natural communication, self-expression and intuitive teaching which enables the robot to work besides humans in daily life. We developed a full-body prototype robot which works on a table. It has cameras, microphones, and a tactile sensor suit which covers its entire body. It also has a speaker to speak with, and has 15 DOF to realize gestures. A system with multi-threaded structure was built to realize multiple sensored interaction based on context and was implemented as a context based interaction system, the plausibility was confirmed in an experiment.

In this paper, a developmental construction methodology for a humanoid robot system is proposed. "Developmental" means that a versatile robot system is inherently a target to be developed incrementally, repetitively with inheriting results of former generations. Based on this methodology, the developmental system consists of (1) the hardware independent robot model for a skill development, (2) an action space representation for a functional development and (3) a robot body which has an onbody network for a physical development. On this system, five humanoid robots have been developed and locomotion behaviors have been realized.

View-based or appearance-based approaches have been attracting the interests of computer vision research. We have already proposed a visual view-based navigation method using a model of the route called the "view sequence", which contains a sequence of front views along a route memorized in the teaching run. In this paper, we apply an omnidirectional vision sensor to our view-based navigation and propose an extended model of a route called "omni-view sequence". The omnidirectional vision sensor is a desirable sensor for real-time view-based recognition of a mobile robot because all the information around the robot can be acquired simultaneously. The use of omnidirectional images contribute to a better navigation method in three ways: (1) it enables the robot to come back along the route using the same view sequence, (2) it improves the accuracy of the navigation and (3) it improves the robustness of the matching. These improved points are shown through experiments, and indoor navigation, including getting on and off an elevator, is demonstrated.

Simulation is an important issue in robotics research because it is essential for evaluating and predicting the behavior of a robot. However conventional simulators don't have the function of simulating the visual perception process. This indicates that it is impossible to simulate the vision-based behavior of a robot. In this paper, we propose a new simulation system "View Simulation System (VSS)", which is designed to simulate the behavior of a mobile robot equipped with vision. VSS has vision processing hardware, and can run any programs including vision processing for our real mobile robots. To show the feasibility of the simulator, we applied it to the evaluation of our visual navigation method.

The function that enables a robot to react when touched by a human is indispensable for a robot which enters our daily life. This paper describes the model and processing of full-body humanoid's tactile sensor suit for the human-robot interaction. The tactile sensor suit is first modeled to make a symbolized tactile data structure. Then conversion and compensation process is proposed to enable the robot to obtain accurate tactile data no matter what posture it takes. Then we explain the robot system and two experiments using a 36 DOF full-body humanoid: the first is the acquiring maps to process the tactile data, and second is the conformation to a human's shape as a human-robot contact interaction with proposed sensor suit model and processing.

A legged robot "JROB-1" is developed for a robotics research platform as a result of inter-university research program on intelligent robotics supported by the Ministry of Education Grant-in-Aid for Scientific Research on Priority Areas in Japan. The JROB-1 features: 1) self-contained, 2) RT-Linux running on PG/AT processes vision and sensor processing, motion planning and control, 3) connected to a network via radio Ethernet as to utilize networked resources, 4) Fujitsu color tracking vision board and Hitachi general purpose vision processing board, 5) all parts are commercially available, and 6) it is extensible with respect to sensor, sensor processing hardware and software. JROB-1 is expected to be a common testbed for experiment and intelligent robotics research by integrating perception and motion.

In this paper, a second generation remote-brained humanoid robot is presented which is developed for research on whole body action. Humanoid robots are important as a platform for the integration of techniques and algorithms acquired from research on manipulators, legged robots and so on. And they have new problems such as how to acquire, memorize, select and carry out various motions which use their whole bodies efficiently. In order to do research on these problems, a good robot body, which has whole body and enough performance for walking and getting up when it falls down is necessary. At the same time, a powerful brain is necessary which can be evolved through the body. As a solution to these demands, remote-brained approach was proposed and several humanoid robots was developed. Using these robots, several researches were done. For example, a brain framework called BeNet, an action acquisition using GA and NN and so on. They were done using a simple wireless connection. Interface between robot brain and its body to concentrate on a high-level problem. However this interface limited actions the robot can do simultaneously. In this paper, this old interface is taken to the next step. New interface has multiple actuator control methods which are switched on demand, and an onbody microprocessor network which controls actuators, measures sensors and interacts with a brain. Finally a new humanoid robot is developed on this interface.

In this paper, we propose an environmental description for robots to memorize temporality and spatiality of wide-range environment. The description is region-based and the environment is described as a mosaic of regions. The description consists of three components: panoramic labelled image that is a segmented color panoramic image, region database that has spatial and temporal information about each region, and regions relation network that describes adjacency state between regions. The description is constantly updated in the background of other visual processing. The description enables robots to detect changes in the environment, to grasp their characteristics, and to memorize them. Moreover, we implement the environmental description on the humanoid robot Saika and show how it expands the robot's behavior.

We propose an approach "online visual feature determination through discourse", which realizes the finding and the following task in a real environment. To segment the human's image from the complex background, it is possible to prepare many finds of basic visual features, and to combine them. The proposed approach is a solution for the problem of how to select and combine these visual features for each situation. Namely, a robot and a human make a conversation to search for the suitable visual features for the current situation. This idea enables the robot to find and follow a human successfully in a changeable background environment.

View changing because of vibration while walking is one of the most fundamental problem for a vision based legged robot. To overcome this difficulty, three key issues are denoted: a) integration of color segmentation, optical flow and stereo, which is able to apply to vibrating view using correlation hardware, b) software servo loop implemented as a kernel module for the purpose of soft actuation, and c) smooth walking pattern generation using solid model and dynamics simulator. Furthermore, a quadruped legged robot "JROB-1" is developed as a platform for the research on perception-action coupling in intelligent behavior of robots.

A robot body is not only an active unit but also a sensing unit. This paper describes a humanoid that integrates stereo vision, full-body tactile sensor, sound sensors and wrist force sensors into a sensor image. The sensor image reflects the current state of the environment and is regarded as all the input to the robot brain in this system. The brain program processes a series of sensor images and controls its body using them. The robot is remote-brained and designed to be a testbed to do research on sensor based behaviors of a full-body humanoid. This paper describes the design concept and the details of the system.

This paper discusses a stereo vision processing system which processes color and monochrome stereo video signals on a single vision processing board. Here, we propose a "field mixing" technique for multiplexing multiple video signals. A compact color stereo vision system based on this technique is developed for a mobile robot. This system can process multiple video signals simultaneously, and realizes flexible color stereo vision processing. In order to show the feasibility of this vision system, we installed it on our mobile robot, and implemented a correlation-based EZDF method for stereo tracking of an object. The experimental result of the tracking is shown.

We describe the method in which a visually guided swing motion for a 16 DOF two-armed bipedal robot is acquired by applying a GA (genetic algorithm) to a NN (neural network) controller. The evolutionary approach to the acquisition of various motions for robots has been successfully used by many researchers, but most studies have been carried out only through computer simulations. In this research, we adopt a real robot with a complicated body used in a noisy environment. The evolutionary processes are examined in. A virtual world constructed on a CRS-CS6400 parallel computer which simulates such factors as swing dynamics, visual processes noise reduction processes, and time lags in a control system. It took about and hours for an artificial evolution to create a successfully individual after 50 generations from an initial population of 200 unsuccessful genes. Using the NN decoded from the most successful individual of the last generation, a real two-armed bipedal robot that could swing successfully was obtained.

In this paper, we describe the design implementation of this real-time layer for remote-brained robot. "Remote-brained approach" is a paradigm for software research by separating the robot brain from the robot body. The interface from the body to the brain top-level software is also denoted. Furthermore, an application of the reactive motion of humanoid-type robot is described. The real-time facility is fundamentally important for a robot that behaves in the read-world. To accomplish both large scale intelligent software and real-time functionality, we design the system by dividing the brain into a low-level real-time layer and a high-level action decision layer. The action decision layer is put on a WS, and the real-time layer is put on a transputer network.

This article addresses the development of a light-weight, human-size and low-cost developing humanoid robot named Saika. Saika has a 2-DOF neck, two 5-DOF upper arms, a torso and a head. Several types of hands and forearms are developed. They are chosen depending upon the tasks to perform. The features of Saika are: (a) Saika as modularized to reduce the developing cost and to make maintenance easy, (b) the total weight of the head, the neck, the two upper arms and the torso is only eight kilograms and (c) most of the motors are installed inside the arms and the torso.

We present an approach to realizing a complex motion system by describing an asynchronous network of real time computational modules that represent dynamical systems with input, output and internal state completely. In our approach, the system is decomposed into parallel modules that calculate their assigned parts of the internal state at regular intervals. We stipulate the roles of the modules so that we can design each module independently and modify or extend the system easily. We have developed programming environments in which we can describe each module and the network briefly in programming languages and test it immediately in the real world. We describe how to design an understandable hierarchical network for desirable actions of an autonomous legged robot. The system has modules that run in parallel for generating motion patterns and actions. The described network was simulated in real time on a multiprocessor system and it generated desirable actions.

Previous work in vision-based mobile robotics have lacked models of the route which can be utilized for (1) localization, (2) steering angle determination, and (3) obstacle detection, simultaneously. In this paper, the authors propose a new visual representation of the route, the "view-sequenced route representation (VSRR)." The VSRR is a non-metrical model of the route, which contains a sequence of front view images along a route memorized in the recording run. In the autonomous run, the three types of recognition described above are achieved in real-time by matching between the current view image and the memorized view sequence using a correlation technique. the authors also developed an easy procedure for acquiring VSRRs, and a quick control procedure using VSRRs. VSRRs are especially useful for representing routes in corridors. Results of autonomous navigation using a two-wheeled robot in a real corridor are also presented.

This paper describes a method for tracking moving objects with multiple cooperative visual attention regions. Each attention region is tracked using a sum of absolute difference (SAD) correlation technique. The method is based on an implementation of a transputer-based multi-processor vision system with a correlation processor. The vision system provides a large number of attention regions, each of which can be tracked using fast SAD correlation performed in an LSI MEP. We call these regions windows since the region resembles a clipped section of a whole view. On the top of the system, we built a software manager which facilitates program image processing, motion control of a window, and cooperation control of windows. The cooperation is achieved using constraints, which are tree-structured, between windows. Cooperation of multiple attention regions enables one to track route predictable objects, deformable objects, and structured objects in a natural scene. As examples, passing car tracking, human arm tracking, and finger action recognition are shown.

We present design and implementation of a tactile sensor system, sensor suit, that covers the entire body of a robot. The sensor suit is designed to be soft and flexible and to have a large number of sensing regions. We have built the sensor suit using electrically conductive fabric and string. The current version of the sensor suit has 192 sensing regions. Each sensing region works as a binary switch in the current version. All of the signals from the sensor suit are gathered and superimposed on a visual image of the robot. The video multiplexer for the sensor signals is built on a field programmable gate array set. The construction of sensor suit system and its application with a full-body humanoid are presented.

We present a 35 DOF full-body humanoid is designed to be a testbed to integrate research tools for humanoid full-body behaviors. Each limb has 6 DOF. The neck has 3 DOF. Each hand has 4 DOF. The key idea of the system architecture is the remote-brained approach. This paper describes the design concept and the details of the system and introduces experiments to stand up using table for support and to perform action selection in reaching and grasping an object based on vision.

We present a vision-based real-time control system of a human-form robot in swing motion. The robot has 16 DOF joints and a camera which is designed to do research on human-form robot behaviors. The key idea to develop the vision based system for the robots are the remote-brained approach and real-time visual tracking. In this approach, the robot system is designed to have the brain and the body separate and to connect them by a radio link. The body sends the visual image to the brain and the body receives the motion, references. The basic algorithm to observe the swing motion by vision uses the optical flow generation. The control algorithm for flow-based visual feedback is described.

Constructing an environment for robot software research, which can program and experiment in various robot behaviors, design of a software development platform becomes important problem. This paper describes a design of "Software Platform" for real-time large scale robot software, and its implementation in the "Remote-Brained Robot Project". The Software Platform is designed as three layers, "MOTHER, BRAIN, SENSOR-MOTOR". MOTHER consists of tools to produce and evolve BRAIN programs. For tools and libraries in such platform, there are two major problems. One is, there is tradeoff relationship between "Extension" and "Share". A way of overcoming this problem is described. The other is, the methodology is needed between low-level real-time parallel environment monitoring program and high-level software that takes much time. A system developed for remote-brained robots and its application are described. This system aims at developing many sorts of brain architecture and high level software that consists of various flexible multiprocess network.

We have focused on actions of a humanlike robot which has the whole body and realized actions such as "walk", "roll over" and "stand up". In this research, we design and implement the whole body humanlike robot that can perform a "carry objects" action. Keys to realizing a carrying action are (1) the robot system which has a lightweight whole body robot and a powerful brain which can process a multi joint model of the body, (2) the model environment which can generate walking patterns on the horizontal ground, stairs and a slope by moving its centroid and (3) gravity compensation based on modeling of the actuator.

We propose a computational model, BeNet, for describing the brains of autonomous robots and an approach to integrating functions of autonomous robots. BeNet is a network of asynchronously communicating processes that change their state and output vectors and their own frequencies determined by the designer. One can describe a modular system for a desirable behavior as a BeNet that interacts with the real world in real time. We design modules of a BeNet that calculate a part of the internal representation of the brain that is necessary for the desired behavior. The portions of internal representation such as action command, attention command, perceptual/sensational information are realized as messages among the modules. Building such a modular system helps you to develop a brain for a desirable behavior, modifying and extending the described system, since the information flow and the modules' roles are clear. We have implemented environments for developing BeNet on parallel computers. In these environments, you can describe BeNets that consist of processes of wide range in complexity and frequency of calculation, and test their behavior in the real world immediately. They are effective for building desirable behaviors of autonomous robots. They are demonstated for a ballboy robot, with subsystems for vision, motion and decision.

Presents a new type of robot which has two arms and two legs like an ape and is aimed at studying a variety of vision-based behaviors. The robot does not bring its own brain within the body. It leaves the brain in the mother environment and talks with it by radio links. The brain is raised in the mother environment inherited over generations. The key idea of the remote-brained approach is that of interfacing intelligent software systems with real robot bodies through wireless technology. In this framework the robot system can have a powerful vision system in the brain environment. The authors have applied this approach to formation of vision-based dynamic and intelligent behaviors of a multi-limbed mobile robot. In this paper the authors present an apelike robot with the remote-brained environment and describe vision-based experiments carried out with the apelike robot.

Presents a practical mobile robot system, called Hyper Scooter, which a human can ride on, and share access to the environment through visual information. The advantage of these capabilities is significant. A user can give his/her instructions to the robot without skillful programming operations. This is supported by the authors' high speed tracking vision system. To show the advantage, a complex task is presented as an example. After being instructed the way to accomplish the task by a human, Hyper Scooter does it without explicit environmental models.

Focusing attention on flexibility and intelligent reactivity in the real world, it is more important to build, not a robot that won't fall down, but a robot that can get up if it does fall down. This paper presents research on a two-armed bipedal robot, an apelike robot, which can perform biped walking, rolling over and standing up. The robot consists of a head, two arms, and two legs. The control system of the biped robot is designed based on the remote-brained approach in which a robot does not bring its own brain within the body and talks with it by radio links. This remote-brained approach enables a robot to have both a heavy brain with powerful computation and a lightweight body with multiple joints. The robot can keep balance while standing using tracking vision, detect whether it falls down or not by a set of vertical sensors, and perform a getting up motion by coordinating two arms and two legs. The developed system and experimental results are described with illustrated real examples.

We present a variety of vision-based adaptive and interactive behaviors in mechanical animals. The mechanical animal is a multi-legged robot designed as a remote-brained robots which does not bring its own brain within the body. It leaves the brain in the mother environment and talks with it by radio links. The brain is raised in the mother environment inherited over generations. The key idea of the remote-brained approach is that of interfacing intelligent software systems with real robot bodies through wireless technology. In this framework the robot system can have a powerful vision system in the brain environment. We have applied this approach toward formation of vision-based dynamic and intelligent behaviors of mechanical animals such as doglike robots and apelike robots. In this paper we introduce the remote-brained approach and describe some remote-brained robots and visual processes for adaptive and interactive behaviors with them.>

A novel task instruction method for future intelligent robots is presented, In our method, a robot learns reusable task plans by watching a human perform assembly tasks. Functional units and working algorithms for visual recognition and analysis of human action sequences are presented. The overall system is model based and integrated at the symbolic level. Temporal segmentation of a continuous task performance into meaningful units and identification of each operation is processed in real time by concurrent recognition processes under active attention control. Dependency among assembly operations in the recognized action sequence is analyzed, which results in a hierarchical task plan describing the higher level structure of the task. In another workspace with a different initial state, the system re-instantiates and executes the task plan to accomplish an equivalent goal. The effectiveness of our method is supported by experimental results with block assembly tasks.>

Graspless manipulation refers to operations in which a robot attempts to change the position and posture of an object without grasping it. Typical examples of graspless manipulation which have been studied so far are pushing and tumbling. This paper proposes a new method of graspless manipulation, pivoting, in which robot fingers maneuver an object in the same way that person moving furniture, as if an object walks on the floor by using appropriate vertices as virtual feet. This paper introduces the concept of this pivoting operation, describes its characteristics and analyzes it mathematically, points out the need for visual guidance for the precise performance of pivoting, and presents experiments with pivoting manipulation in connection with visual guidance.

A stereo vision system which consists of a stereo viewer and a processing system with viewing control mechanisms is presented. The stereo viewer is an adapter which, attached to a single camera, provides it with a stereo image by means of a set of four mirrors. Since the system can control the angle of mirrors, it can adjust them to operate during zooming, panning, and focusing in stereo viewing. The vision system uses a high-speed block matching processor to provide the essential functionality needed for motion tracking and stereo matching. With this system visual searching and stereo tracking can be implemented in real time. The authors analyze the optical structure of the stereo viewer, describe the vision and control system, and present some experimental examples of its use.

The authors describe a high-performance robot vision system that performs real-time tracking of moving objects, real-time optical flow computation, and high-speed depth map generation. The system was implemented as a transputer-based vision system augmented with a high-speed correlation processor. The transputer vision board was equipped with three image frame memories, each of which could be used simultaneously for image input, image processing, and image display. Thus, the system could devote all its computation power to image processing without waiting for image input or display. The vision board was also equipped with a standard image compression chip, used as a correlation processor. Using the chip, a very fast correlation-based robot vision system was developed. This system can also be used as a multiprocessor configuration to greatly improve performance.>

Functional units and working algorithms for real-time visual recognition of human pick-and-place action sequences are presented. The action recognizer consists of visual feature detectors, an action/environment model, and an attention stack. It generates a symbolic description of the observed action sequence. Given a different initial state, the system reinstantiates the recognized action sequence to carry out an equivalent assembly task. Experimental results on several assembly tasks support the effectiveness of the method.>

This paper describes a method to observe a moving object with multiple visual tracking agents. When a robot vision watches an object, it employs several tracking agents to monitor key features of the object. Each tracking agent is controlled by its visual feedback. In order to perform robust tracking, the authors introduce constraint control between agents. This method is effective in observing a motion of not only a single solid object but also a structured object such as a human hand. Several examples applied to 3D motion are shown.>