We have developed an autonomous robot motion generation model based on distributed and integrated multimodal learning. Since each modality used as a robot's senses, such as image, joint angle, and torque, has a different physical meaning and time characteristic, the generation of autonomous motions using multimodal learning has sometimes failed due to overlearning in one of the modalities. Inspired by the sensory processing of the human brain, our model is based on the processing of each sense performed in the primary somatosensory cortex and the integrated processing of multiple senses in the association cortex and the primary motor cortex. Specifically, the proposed model utilizes two types of recurrent neural networks: sensory RNNs, which learn each sense in a time series, and a union RNN, which communicates with sensory RNNs and learns sensory integration. The simulation results of multiple tasks showed that our model processes multiple modalities appropriately and generates smoother motions with lower jerk than the conventional model. We also demonstrated a chair assembly task by combining fixed motions and autonomous motions with our model.

We achieved contact-rich flexible object manipulation, which was difficult to control with vision alone. In the unzipping task we chose as a validation task, the gripper grasps the puller, which hides the bag state such as the direction and amount of deformation behind it, making it difficult to obtain information to perform the task by vision alone. Additionally, the flexible fabric bag state constantly changes during operation, so the robot needs to dynamically respond to the change. However, the appropriate robot behavior for all bag states is difficult to prepare in advance. To solve this problem, we developed a model that can perform contact-rich flexible object manipulation by real-time prediction of vision with tactility. We introduced a point-based attention mechanism for extracting image features, softmax transformation for predicting motions, and convolutional neural network for extracting tactile features. The results of experiments using a real robot arm revealed that our method can realize motions responding to the deformation of the bag while reducing the load on the zipper. Furthermore, using tactility improved the success rate from 56.7% to 93.3% compared with vision alone, demonstrating the effectiveness and high performance of our method.

We propose a motion generation model that can achieve robust behavior against environmental changes based on language instructions at a low cost. Conventional robots that communicate with humans use a restricted environment and language to build up a mapping between language and motion, and thus need to prepare a huge training set in order to achieve versatility. Our method trains pairs of language, visual, and motor information of the robot, and generates motions in real-time based on the “attention” of the language instructions. Specifically, the robot generates motions while focusing on the indicated objects by the human when multiple objects are in the field of view. In addition, since position recognition and motion generation of the indicated object are performed in real-time, robust motion generation is possible in response to changes in the object position and lighting conditions. We clarified that features related to the object name and its location are self-organized in the latent (PB: Parametric Bias) space by end-to-end learning of robot motion and sentences. These observations may indicate the importance of integrated learning of robot motion and sentences since such feature representations cannot be obtained by learning motions alone.

Fast and accurate depth estimation is essential for autonomous driving vehicles and autonomous robots. Monocular cameras have been expected to be widely applied due to their lower cost, but since they can't provide depth directly, fast and accurate monocular depth estimation methods are required. This paper proposes a novel method of monocular depth estimation based on flat surface using feature triangulation and two frames per estimation. Experiments under ideal conditions and with the KITTI Vision Benchmark Suite showed that the method achieved smaller errors comparing to existing methods that rely on inertial sensors and many frames.

We are developing a robot that supports people in their daily lives: a human-symbiotic robot. Such robot must share space with its users, be user-friendly, and be able to assist its users. We have developed a prototype autonomous mobile robot that makes use of a self-balancing two-wheeled mobile system and a body swing mechanism to shift its center of gravity. This allows it to move nimbly at up to 6 km per hour. It also has capabilities to avoid collisions with obstacles for moving safely through complex environments. Distant-speech-recognition and high-quality speech-synthesis technologies enable it to communicate with people naturally (i.e., without special tools). These capabilities were demonstrated at the 2005 World Exposition in Aichi, Japan

To monitor multiple moving objects from a robot in motion is an essential technology in robotic application areas including service and security for human daily life. For this, a method to track multiple walking humans using a mobile robot "in motion" with laser range finder (LRF) is investigated in this paper. Geometric characteristics of human legs are considered to detect their position from the LRF data. Frequency and phase of walking motion are extracted using a pendulum model of the angle between two legs and extended Kalman filter. The algorithm with human walking model anticipates the position of moving humans. The effectiveness of the proposed method is also evaluated with some experiments to track multiple walking humans in indoor environment. An experimental testbed which consists of a mobile robot "Yamabico" and a LRF is employed to track people. Resultant experiments and analysis showed that multiple walking people are tracked well from a mobile robot with the proposed method