In imitation learning for robotic manipulation, decomposing object manipulation tasks into sub-tasks enables the reuse of learned skills and the combination of learned behaviors to perform novel tasks, rather than simply replicating demonstrated motions. Human gaze is closely linked to hand movements during object manipulation. We hypothesize that an imitating agent’s gaze control—fixating on specific landmarks and transitioning between them—simultaneously segments demonstrated manipulations into sub-tasks. This study proposes a simple yet robust task decomposition method based on gaze transitions. Using teleoperation, a common modality in robotic manipulation for collecting demonstrations, in which a human operator’s gaze is measured and used for task decomposition as a substitute for an imitating agent’s gaze. Our approach ensures consistent task decomposition across all demonstrations for each task, which is desirable in contexts such as machine learning. We evaluated the method across demonstrations of various tasks, assessing the characteristics and consistency of the resulting sub-tasks. Furthermore, extensive testing across different hyperparameter settings confirmed its robustness, making it adaptable to diverse robotic systems. Our code is available at https://github.com/crumbyRobotics/GazeTaskDecomp.

Deep imitation learning is a promising approach in robotic manipulation, enabling robots to acquire versatile and adaptable skills. In such research, by learning various tasks, robots achieved generality across multiple objects. However, such multi-task robot datasets have mainly focused on single-arm tasks that are relatively imprecise and not addressed the fine-grained object manipulation that robots are expected to perform in the real world. In this study, we introduce a dataset for diverse object manipulation that includes dual-arm tasks and/or tasks that require fine manipulation. We generated a dataset containing 224k episodes (150 hours, 1,104 language instructions) that includes dual-arm fine tasks, such as bowl-moving, pencil-case opening, and banana-peeling. This dataset is publicly available1. Additionally, this dataset includes visual attention signals, dual-action labels that separate actions into robust reaching trajectories or precise interactions with objects, and language instructions, all aimed at achieving robust and precise object manipulation. We applied the dataset to our Dual-Action and Attention, which is a model that we designed for fine-grained dual-arm manipulation tasks that is robust to covariate shift. We tested the model in over 7k trials for real robot manipulation tasks, which demonstrated its capability to perform fine manipulation.

Long-horizon dexterous robot manipulation of deformable objects, such as banana peeling, is a problematic task because of the difficulties in object modeling and a lack of knowledge about stable and dexterous manipulation skills. This paper presents a goal-conditioned dual-action (GC-DA) deep imitation learning (DIL) approach that can learn dexterous manipulation skills using human demonstration data. Previous DIL methods map the current sensory input and reactive action, which often fails because of compounding errors in imitation learning caused by the recurrent computation of actions. The method predicts reactive action only when the precise manipulation of the target object is required (local action) and generates the entire trajectory when precise manipulation is not required (global action). This dual-action formulation effectively prevents compounding error in the imitation learning using the trajectory-based global action while responding to unexpected changes in the target object during the reactive local action. The proposed method was tested in a real dual-arm robot and successfully accomplished the banana-peeling task.

Deep imitation learning is a promising approach that does not require hard-coded control rules in autonomous robot manipulation. The current applications of deep imitation learning to robot manipulation have been limited to reactive control based on the states at the current time step. However, future robots will also be required to solve tasks utilizing their memory obtained by experience in complicated environments (e.g., when the robot is asked to find a previously used object on a shelf). In such a situation, simple deep imitation learning may fail because of distractions caused by complicated environments. We propose that gaze prediction from sequential visual input enables the robot to perform a manipulation task that requires memory. The proposed algorithm uses a Transformer-based self-attention architecture for the gaze estimation based on sequential data to implement memory. The proposed method was evaluated with a real robot multi-object manipulation task that requires memory of the previous states.

Imitation learning has attracted attention as a method for realizing complex robot control without programmed robot behavior. Meta-imitation learning has been proposed to solve the high cost of data collection and low generalizability to new tasks that imitation learning suffers from. Meta-imitation can learn new tasks involving unknown objects from a small amount of data by learning multiple tasks during training. However, meta-imitation learning, especially using images, is still vulnerable to changes in the background, which occupies a large portion of the input image. This study introduces a human gaze into meta-imitation learning-based robot control. We created a model with model-agnostic meta-learning to predict the gaze position from the image by measuring the gaze with an eye tracker in the head-mounted display. Using images around the predicted gaze position as an input makes the model robust to changes in visual information. We experimentally verified the performance of the proposed method through picking tasks using a simulated robot. The results indicate that our proposed method has a greater ability than the conventional method to learn a new task from only 9 demonstrations even if the object's color or the background pattern changes between the training and test.

Deep imitation learning is promising for solving dexterous manipulation tasks because it does not require an environment model and pre-programmed robot behavior. However, its application to dual-arm manipulation tasks remains challenging. In a dual-arm manipulation setup, the increased number of state dimensions caused by the additional robot manipulators causes distractions and results in poor performance of the neural networks. We address this issue using a self-attention mechanism that computes dependencies between elements in a sequential input and focuses on important elements. A Transformer, a variant of self-attention architecture, is applied to deep imitation learning to solve dual-arm manipulation tasks in the real world. The proposed method has been tested on dual-arm manipulation tasks using a real robot. The experimental results demonstrated that the Transformer-based deep imitation learning architecture can attend to the important features among the sensory inputs, therefore reducing distractions and improving manipulation performance when compared with the baseline architecture without the self-attention mechanisms.

Segmentation of a compound task with multiple subtasks is crucial for imitation learning. Conventional unsupervised segmentation methods focused on only reproducibility of demonstrations and did not use the property that goal-directed actions rarely occur without intention. In this paper, we propose a novel method to segment demonstrations into goal-directed actions by self-supervised learning. We use the discriminator between demonstrations and self-generated unintentional actions performed by the same body in behavioral cloning paradigm because goal-directed actions rarely occur without intention, and thus can be separated from unintentional actions. And we consider the states that cannot be reached by unintentional actions as subtask changepoints. We evaluated our method on manipulation tasks with multiple subtasks. The results indicate that our method can detect subtask changepoints more accurately than an existing unsupervised segmentation method.

Visual occlusions caused by the environment or by the robot itself can be a problem for object recognition during manipulation by a robot hand. Under such conditions, tactile and somatosensory information are useful for object recognition during manipulation. Humans can visualize the appearance of invisible objects from only the somatosensory information provided by their hands. In this paper, we propose a method to generate an image of an invisible object's posture from the joint angles and touch information provided by robot fingers while touching the object. We show that the object's posture can be estimated from the time-series of the joint angles of the robot hand via regression analysis. In addition, conditional generative adversarial networks can generate an image to show the appearance of the invisible objects from their estimated postures. Our approach enables user-friendly visualization of somatosensory information in remote control applications.

This paper presents a system for capturing human movement and tactile data and methods for analyzing this data. We cannot fully capture the essence of motion without tactile information, and sometimes the lack of such information causes critical problems. To achieve a better understanding of motion behavior, we developed a wearable motion capture suit with full-body tactile sensors. We also developed a motion sensor which can estimate its orientation with its inner CPU. We also built a tactile sensor module which can fit many kinds of body shapes. With this system, we can measure a user's movement and tactile information simultaneously. By integrating tactile data with motion data, we can achieve many kinds of meaningful insights. We demonstrate the effectiveness of this system with experiments. We captured two motions: stretching after sitting on a chair and laying down on a bed. By recognizing the contact point from the tactile data and fitting it into the environment, we were able to estimate the motion trajectories.

To help with care work and rescue operations, it is necessary for humanoid robots to have the ability to transport humans steadily and gently. In this research we consider "piggyback" motions for transporting humans. Most people can perform this motion, allowing us to measure and analyze piggyback motions of human subjects using tactile sensing and whole body movements to design whole body contact control. One interesting result of this investigation is that frictional forces are skillfully controlled by the carrier. In the first experiment, we study a "knack" that allows the carrier to reposition the rider. In the second experiment we verify the effectiveness of the knack in achieving the repositioning result. We also studied the principle of the repositioning motion, and found that it is similar in many ways to a jumping motion. Then we confirmed the validity of our modeling assumptions using a dynamical simulator.

We present realization of a humanoid which can lift a heavy object by whole body contact. Most humanoid motions are limited to the posture of the end-effectors only landing. In principle these humanoids can not do natural motion. If a humanoid robot is allowed arbitrary contact with the surrounding objects, it can improve the performance and operate a heavier object. We propose a "whole body contact motion" of a humanoid robot. It is defined as a control of contact state of a humanoid robot which has the distributed tactile sensors. We develop conformable and scalable tactile skin and an adult-size humanoid with a smooth surfaces for arbitrary contact. We install the skin on the entire surfaces of the humanoid. Finally we describe the humanoid lifting a 30kg box by tactile feedback.

We present the design and realization of a conformable tactile sensor skin (patent pending). The skin is organized as a network of self-contained modules consisting of tiny pressure-sensitive elements which communicate through a serial bus. By adding or removing modules it is possible to adjust the area covered by the skin as well as the number (and density) of tactile elements. The skin is therefore highly modular and thus intrinsically scalable. Moreover, because the substrate on which the modules are mounted is sufficiently pliable to be folded and stiff enough to be cut, it is possible to freely distribute the individual tactile elements. A tactile skin composed of multiple modules can also be installed on curved surfaces. Due to their easy configurability we call our sensors "cut-and-paste tactile sensors." We describe a prototype implementation of the skin on a humanoid robot

We propose that highly dynamic whole-body motions should be analyzed, realized and exploited for extending the capability of humanoid robots in the real world. Such motions are very different from the today's standard humanoid behaviors such as stable ZMP-based biped walking, and upper-body motion assuming the stability of the lower-body. The kind of motions we are interested are not locally stable states. Sometimes they include diverging trajectories. In this paper, we focus on one example of such motions: a roll-and-rise motion, in which the robot stands up in one action from lying state. It first swings up both of its legs high, swings them down, rolling forward and up on both feet, then extends the legs to achieve the standing posture. Analysis of the dynamics governing the motions is carried out, and some boundary conditions for successful motions are presented. Our current goal is to identify essential minimum control laws that assure the success of the task. In search of them, a series of systematic simulation experiments are carried out to plot the parameter regions which define success or failure. Experiments with real adult-size humanoid robot are also presented.