Human aesthetic appreciation of images is shaped by both the objective properties of the images and the subjective aspects arising from the viewer’s internal process. Traditional AI research on aesthetics assessment has largely focused on objective properties or group level evaluations, often overlooking individual subjectivity. Large Language Models (LLMs), trained on vast text data, offer potential for addressing this gap through their semantic and contextual understanding capabilities. We hypothesized that their capabilities could be leveraged for predicting aesthetic evaluations and explored effective prompting techniques using a proposed list of evaluation factors. We found that the factors related to semantic and contextual understanding enhance LLMs’ performance in predicting beauty. This study shows the potential of a novel approach to image aesthetics assessment by leveraging the semantic and contextual understanding capabilities of LLMs.

In imitation learning for robotic manipulation, decomposing object manipulation tasks into sub-tasks enables the reuse of learned skills and the combination of learned behaviors to perform novel tasks, rather than simply replicating demonstrated motions. Human gaze is closely linked to hand movements during object manipulation. We hypothesize that an imitating agent’s gaze control—fixating on specific landmarks and transitioning between them—simultaneously segments demonstrated manipulations into sub-tasks. This study proposes a simple yet robust task decomposition method based on gaze transitions. Using teleoperation, a common modality in robotic manipulation for collecting demonstrations, in which a human operator’s gaze is measured and used for task decomposition as a substitute for an imitating agent’s gaze. Our approach ensures consistent task decomposition across all demonstrations for each task, which is desirable in contexts such as machine learning. We evaluated the method across demonstrations of various tasks, assessing the characteristics and consistency of the resulting sub-tasks. Furthermore, extensive testing across different hyperparameter settings confirmed its robustness, making it adaptable to diverse robotic systems. Our code is available at https://github.com/crumbyRobotics/GazeTaskDecomp.

Deep imitation learning is a promising approach in robotic manipulation, enabling robots to acquire versatile and adaptable skills. In such research, by learning various tasks, robots achieved generality across multiple objects. However, such multi-task robot datasets have mainly focused on single-arm tasks that are relatively imprecise and not addressed the fine-grained object manipulation that robots are expected to perform in the real world. In this study, we introduce a dataset for diverse object manipulation that includes dual-arm tasks and/or tasks that require fine manipulation. We generated a dataset containing 224k episodes (150 hours, 1,104 language instructions) that includes dual-arm fine tasks, such as bowl-moving, pencil-case opening, and banana-peeling. This dataset is publicly available1. Additionally, this dataset includes visual attention signals, dual-action labels that separate actions into robust reaching trajectories or precise interactions with objects, and language instructions, all aimed at achieving robust and precise object manipulation. We applied the dataset to our Dual-Action and Attention, which is a model that we designed for fine-grained dual-arm manipulation tasks that is robust to covariate shift. We tested the model in over 7k trials for real robot manipulation tasks, which demonstrated its capability to perform fine manipulation.

Homeostasis is a self-regulatory process, wherein an organism maintains a specific internal physiological state. Homeostatic reinforcement learning (RL) is a framework recently proposed in computational neuroscience to explain animal behavior. Homeostatic RL organizes the behaviors of autonomous embodied agents according to the demands of the internal dynamics of their bodies, coupled with the external environment. Thus, it provides a basis for real-world autonomous agents, such as robots, to continually acquire and learn integrated behaviors for survival. However, prior studies have generally explored problems pertaining to limited size, as the agent must handle observations of such coupled dynamics. To overcome this restriction, we developed an advanced method to realize scaled-up homeostatic RL using deep RL. Furthermore, several rewards for homeostasis have been proposed in the literature. We identified that the reward definition that uses the difference in drive function yields the best results. We created two benchmark environments for homeostasis and performed a behavioral analysis. The analysis showed that the trained agents in each environment changed their behavior based on their internal physiological states. Finally, we extended our method to address vision using deep convolutional neural networks. The analysis of a trained agent revealed that it has visual saliency rooted in the survival environment and internal representations resulting from multimodal input.

Long-horizon dexterous robot manipulation of deformable objects, such as banana peeling, is a problematic task because of the difficulties in object modeling and a lack of knowledge about stable and dexterous manipulation skills. This paper presents a goal-conditioned dual-action (GC-DA) deep imitation learning (DIL) approach that can learn dexterous manipulation skills using human demonstration data. Previous DIL methods map the current sensory input and reactive action, which often fails because of compounding errors in imitation learning caused by the recurrent computation of actions. The method predicts reactive action only when the precise manipulation of the target object is required (local action) and generates the entire trajectory when precise manipulation is not required (global action). This dual-action formulation effectively prevents compounding error in the imitation learning using the trajectory-based global action while responding to unexpected changes in the target object during the reactive local action. The proposed method was tested in a real dual-arm robot and successfully accomplished the banana-peeling task.

The embodied autonomous agent, such as a house-hold robot or a pet robot, is required to satisfy multiple requirements simultaneously. One possible approach would be to train the agent to simultaneously solve multiple tasks using a single physical entity (i.e., the body). However, integrating multiple tasks is not a trivial problem in the reward design paradigm of reinforcement learning (RL). Homeostatic RL treats the integration of multiple tasks as a control problem over the agent's internal bodily state. Still, learning in previous studies was extremely slow. In this study, we report novel behavior switching architectures: the Interoceptive Mixture of Experts (IMoE) and the Interoceptive Behavior Switching (IBS) for homeostatic RL agents with continuous motor control. In these architectures, the agent switches between multiple policies using internal body states (interoception). We tested IMoE and IBS in four homeostatic RL environments. We also compared IMoE and IBS with a fully connected model and a query key value switching model with full observation policies. The results indicate that the proposed architectures provide better or competitive results in all four benchmark environments.

This study aimed to develop a counseling robot to improve the quality of life (QOL) of older adults and verify its effectiveness. QOL is a comprehensive indicator that includes physical, mental, and social aspects, and gerontechnology aims to improve the autonomy and QOL of older adults. In recent years, utilizing robots to address the shortage of caregivers has been increasingly studied. We developed a counseling robot that estimates the QOL of older adults in real time, and generates appropriate and empathetic responses according to the estimated QOL. The results obtained from a one-week interaction experiment with a counseling robot targeted toward older adults indicated a significant improvement in the mental aspect of QOL, and the use of cognitive behavioral therapy and empathetic responses was inferred to facilitate self-disclosure and enhance the effectiveness of counseling. Additionally, advice based on the QOL estimation results contributed to organizing the thoughts of older adults and further improved their mental health. This study not only contributes to improving the QOL of older adults but also suggests that robots that understand and appropriately respond to individuals can facilitate continuous relationship building. This study may also serve as a guide for promoting the introduction of information and communication technology into welfare facilities.

Deep imitation learning is a promising approach that does not require hard-coded control rules in autonomous robot manipulation. The current applications of deep imitation learning to robot manipulation have been limited to reactive control based on the states at the current time step. However, future robots will also be required to solve tasks utilizing their memory obtained by experience in complicated environments (e.g., when the robot is asked to find a previously used object on a shelf). In such a situation, simple deep imitation learning may fail because of distractions caused by complicated environments. We propose that gaze prediction from sequential visual input enables the robot to perform a manipulation task that requires memory. The proposed algorithm uses a Transformer-based self-attention architecture for the gaze estimation based on sequential data to implement memory. The proposed method was evaluated with a real robot multi-object manipulation task that requires memory of the previous states.

Imitation learning has attracted attention as a method for realizing complex robot control without programmed robot behavior. Meta-imitation learning has been proposed to solve the high cost of data collection and low generalizability to new tasks that imitation learning suffers from. Meta-imitation can learn new tasks involving unknown objects from a small amount of data by learning multiple tasks during training. However, meta-imitation learning, especially using images, is still vulnerable to changes in the background, which occupies a large portion of the input image. This study introduces a human gaze into meta-imitation learning-based robot control. We created a model with model-agnostic meta-learning to predict the gaze position from the image by measuring the gaze with an eye tracker in the head-mounted display. Using images around the predicted gaze position as an input makes the model robust to changes in visual information. We experimentally verified the performance of the proposed method through picking tasks using a simulated robot. The results indicate that our proposed method has a greater ability than the conventional method to learn a new task from only 9 demonstrations even if the object's color or the background pattern changes between the training and test.

We propose ThermoCaress, a haptic device to create a stroking sensation on the forearm using pressure force and present thermal feedback simultaneously. In our method, based on the phenomenon of thermal referral, by overlapping a stroke of pressure force, users feel as if the thermal stimulation moves although the position of temperature source is static. We designed the device to be compact and soft, using microblowers and inflatable pouches for presenting pressure force and water for presenting thermal feedback. Our user study showed that the device succeeded in generating thermal referrals and creating a moving thermal illusion. The results also suggested that cold temperature enhance the pleasantness of stroking. Our findings contribute to expanding the potential of thermal haptic devices.

Deep imitation learning is promising for solving dexterous manipulation tasks because it does not require an environment model and pre-programmed robot behavior. However, its application to dual-arm manipulation tasks remains challenging. In a dual-arm manipulation setup, the increased number of state dimensions caused by the additional robot manipulators causes distractions and results in poor performance of the neural networks. We address this issue using a self-attention mechanism that computes dependencies between elements in a sequential input and focuses on important elements. A Transformer, a variant of self-attention architecture, is applied to deep imitation learning to solve dual-arm manipulation tasks in the real world. The proposed method has been tested on dual-arm manipulation tasks using a real robot. The experimental results demonstrated that the Transformer-based deep imitation learning architecture can attend to the important features among the sensory inputs, therefore reducing distractions and improving manipulation performance when compared with the baseline architecture without the self-attention mechanisms.

Segmentation of a compound task with multiple subtasks is crucial for imitation learning. Conventional unsupervised segmentation methods focused on only reproducibility of demonstrations and did not use the property that goal-directed actions rarely occur without intention. In this paper, we propose a novel method to segment demonstrations into goal-directed actions by self-supervised learning. We use the discriminator between demonstrations and self-generated unintentional actions performed by the same body in behavioral cloning paradigm because goal-directed actions rarely occur without intention, and thus can be separated from unintentional actions. And we consider the states that cannot be reached by unintentional actions as subtask changepoints. We evaluated our method on manipulation tasks with multiple subtasks. The results indicate that our method can detect subtask changepoints more accurately than an existing unsupervised segmentation method.

The physical human–robot interaction (pHRI) research field is expected to contribute to competitive and cooperative human–robot tasks that involve force interactions. However, compared with human–human interactions, current pHRI approaches lack tactical considerations. Current approaches do not estimate intentions from human behavior and do not select policies that are appropriate for the opponent’s changing policy. For this reason, we propose a reinforcement learning model that estimates the opponent’s changing policy using time-series observations and expresses the agent’s policy in a common latent space, referring to descriptions of tactics in open-skill sports. We verify the performance of the reinforcement learning agent using two novel physical and competitive environments, push-hand game and air-hockey. From this, we confirm that the latent space works properly for policy information because each latent variable that represents the machine agent’s own policy and that of the opponent affects the behavior of the agent. Two latent variables can clearly express how the agent estimates the opponent’s policy and decides its own policy.

Spiking neurons might play a larger role than simply as an efficient signal transmitter. Several studies have demonstrated how movements can be generated using networks of spiking neurons. However, the complexity of spiking neural networks makes their implementation difficult, and the use of spiking neurons in robotics has remained largely impractical. In this paper, we show that the addition of a single layer of spiking neurons can help improve performance on stabilization tasks in dynamically changing environments. In a one-dimensional inverted pendulum stabilization task, the spiking neurons seem to expand the space of usable parameters of the controller. Using a robot arm in 3-D space, the additional layer of spiking neurons suffices to improve performance up to 30% on an inverted pendulum stabilization task. We expect this technique to enhance performance in most stabilization tasks but also tasks that are essentially similar such as reaching tasks and posture control. We also expect the effects of this layer to be greatest when the optimal tuning of control parameters is difficult, such as when the environment is unpredictable and dynamic.

It is essential for a monitoring system or a communication robot that interacts with an elderly person to accurately understand the user's state and generate actions based on their condition. To ensure elderly welfare, quality of life (QOL) is a useful indicator for determining human physical suffering and mental and social activities in a comprehensive manner. In this study, we hypothesize that visual information is useful for extracting high-dimensional information on QOL from data collected by an agent while interacting with a person. We propose a QOL estimation method to integrate facial expressions, head fluctuations, and eye movements that can be extracted as visual information during the interaction with the communication agent. Our goal is to implement a multiple feature vectors learning estimator that incorporates convolutional 3D to learn spatiotemporal features. However, there is no database required for QOL estimation. Therefore, we implement a free communication agent and construct our database based on information collected through interpersonal experiments using the agent. To verify the proposed method, we focus on the estimation of the "mental health" QOL scale, which is the most difficult to estimate among the eight scales that compose QOL based on a previous study. We compare the four estimation accuracies: single-modal learning using each of the three features, i.e., facial expressions, head fluctuations, and eye movements and multiple feature vectors learning integrating all the three features. The experimental results show that multiple feature vectors learning has fewer estimation errors than all the other single-modal learning, which uses each feature separately. The experimental results for evaluating the difference between the estimated QOL score by the proposed method and the actual QOL score calculated by the conventional method also show that the average error is less than 10 points and, thus, the proposed system can estimate the QOL score. Thus, it is clear that the proposed new approach for estimating human conditions can improve the quality of human-robot interactions and personalized monitoring.

Quality of life (QoL) is an effective index of well-being, including physical health, aspect of social activity, and mental state of individuals. A new approach that uses a deep-learning architecture to estimate the score of a user's QoL is presented. This system was built using a combination of a 3D convolutional neural network and a support vector machine for multimodal data. In order to evaluate the accuracy of the estimation system, three experiments were conducted. Before these experiments, ten hours of audio and video data were collected from healthy participants during a natural-language conversation with a conversational agent we implemented. In the first experiment, the QoL question-answer estimation experiment, the accuracy of “Physical functioning,” which is one of the eight scales that constitute QoL, reached 84.0%. In the second experiment, the QoL-score-regression experiment, in which the scores of each scale were directly estimated, the distribution of the difference between the actual score and the estimated results, known as error, was investigated. These results imply that the features necessary for QoL estimation can be extracted from audio and video data, except for the “Mental Health” domain. One of the reasons why it was difficult to estimate the “Mental Health” scale may be that the learning framework could not extract an appropriate feature for estimation. Therefore, we estimated “Mental Health” by focusing on eye movement. From the result, it was proven that estimation is possible, and the proposed system using multimodal data demonstrated its effectiveness for estimation for all eight scales that constitute QoL and for extracting high-dimensional information regarding the QoL of a human, including their satisfaction level towards daily life and social activities. Finally, suggestions and discussions regarding the plausible behavior of the estimation results were made from the viewpoint of human–agent interaction in the field of elderly welfare.

Visual occlusions caused by the environment or by the robot itself can be a problem for object recognition during manipulation by a robot hand. Under such conditions, tactile and somatosensory information are useful for object recognition during manipulation. Humans can visualize the appearance of invisible objects from only the somatosensory information provided by their hands. In this paper, we propose a method to generate an image of an invisible object's posture from the joint angles and touch information provided by robot fingers while touching the object. We show that the object's posture can be estimated from the time-series of the joint angles of the robot hand via regression analysis. In addition, conditional generative adversarial networks can generate an image to show the appearance of the invisible objects from their estimated postures. Our approach enables user-friendly visualization of somatosensory information in remote control applications.

This paper presents novel event-driven forward kinematics algorithms for open kinematic chains with O(log n) complexity. This event-driven algorithm can efficiently update forward kinematics only when new sensory data comes. This will also contribute to localization of computational resources at sensitive joints to the position of the endpoint (e.g. a fingertip), like a root joint. We constructed 3 event-driven FK algorithms. We proved that the algorithms have the complexity of O(logn) for updating 1 joint angle, and O(logn) for obtaining a homogeneous transformation matrix between links. We compared the 3 algorithms with a conventional forward kinematics algorithm in the viewpoint of complexity, computation time, time-variance and algebraic structures. The results showed that the computation time is well adequate for real-time computation. Computation time is less than 2 us per 1 query, for 40,000 kinematic chains.

The dolphin kick is a swimming style characterized by undulation of the body. As a platform for swimming research, we have developed a musculoskeletal humanoid robot called Triton. Triton has a flexible spine with erector spinae muscles and a stiffness adjustment system for lumbar joints. The musculoskeletal body includes biarticular and polyarticular muscles, providing multi-joint coordination. The robot is actuated by pneumatic muscles, yielding lightweight and inherently waterproof properties. The compliance of the joints allows interactions between body and fluid similar to those of human swimming. This study presents the design concept of Triton and experimental results from a water tank test. We compare the results with simulation and human movements reported in literature. The results show that the musculoskeletal swimming robot has similar cycle trends in joint angle and thrust force.

This paper presents a new algorithm for forward kinematics, called Asynchronous Forward Kinematics (AFK). The algorithm has the complexity of O(log n) for updating one joint angle, and O(logn) for obtaining a homogeneous transformation matrix between links. AFK enables computation for efficient forward kinematics under asynchronous sensory data. Moreover, AFK peovides localise computational resources at sensitive joints to the position of the endpoint (e.g. a fingertip), like a root joint. We provide comparative results including computation time, evaluating AFK against the conventional forward kinematics (CFK). The results showed that the computation time is well adequate for real-time computation. Computation time for 100 links takes less than 20 us for 1 query. Moreover, computation time with over 50000 links takes less than 35 us for 1 query.

Systems with changeable mechanical properties show promise for expanding the applications of dynamic robots. We proposed the Torque-Angle Relationship Control System (TARCS) for musculoskeletal robots with changeable output properties. First, we formulated TARCS and examined its static properties. Next, we used TARCS to change the properties and investigated the effect of the same on the jumping ability through simulations. We found that TARCS in a biarticular muscle determined the general shape of the jumpable range. Furthermore, TARCS in a monoarticular muscle could change the amplitude of the jumpable range. Both TARCS also changed the directional property of the reaction to the disturbance. Finally, we developed a musculoskeletal quadruped robot in which TARCS was implemented. This robot could change its jumping direction using TARCS, and it jumped to a height of 0.254 m at the lowest point of its body.

The goal of this paper is to develop a regrasp planning algorithm general enough to perform statistical analysis with thousands of experiments and arbitrary mesh models. We focus on pick-and-place regrasp which reorients an object from one placement to another by using a sequence of pick-ups and place-downs. We improve the pick-and-place regrasp approach developed in 1990s and analyze its performance in robotic assembly with different work surfaces in the workcell. Our algorithm will automatically compute the stable placements of an object, find several force-closure grasps, generate a graph of regrasp actions, and search for regrasp sequences. We demonstrate the advantages of our algorithm with various mesh models and use the algorithm to evaluate the completeness, the cost and the length of regrasp sequences with different mesh models and different assembly tasks in the presence of different work surfaces. Our results show that spare work surfaces are beneficial to assembly. Tilted work surfaces are only sometimes beneficial, depending on the objects.

For robots using elastic devices, pole vault is a particularly interesting task because poles have large differences from previously studied elastic elements in terms of their elastic capacity. The active actuation of the agent in “pole support phase” plays important roles in improving vaulting performance. Investigating this actuation can contribute to the design of novel control strategies during the time when the agent contacts with environment through the elastic device. In this study, we specifically examined an active bending effect performed in the “pole support phase.” We analyzed the active bending effect on reachable height (vaulting height) using the “Transitional Buckling Model.” We applied this active bending theory to a robot and verified the active bending effect to improve vaulting height. Results show that active bending motion in the “pole support phase” improves the pole vault performance and that the timing of the bending direction change is an important factor for defining the vaulting performance. These results will facilitate the application of robots using large elasticity.

We propose an efficient method to train multiple object detectors simultaneously using a large scale image dataset. The one-vs-all approach that optimizes the boundary between positive samples from a target class and negative samples from the others has been the most standard approach for object detection. However, because this approach trains each object detector independently, the scores are not balanced between object classes. The proposed method combines ideas derived from both detection and classification in order to balance the scores across all object classes. We optimized the boundary between target classes and their “hard negative” samples, just as in detection, while simultaneously balancing the detector scores across object classes, as done in multi-class classification. We evaluated the performances on multi-class object detection using a subset of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2011 dataset and showed our method outperformed a de facto standard method.

We propose the adoption of a vector extrapolation technique to accelerate convergence of correspondence problems under the quadratic assignment formulation for attributed graph matching (QAP). In order to capture a broad range of matching scenarios, we provide a class of relaxations of the QAP under elastic net constraints. This allows us to regulate the sparsity/complexity trade-off which is inherent to most instances of the matching problem, thus enabling us to study the application of the acceleration method over a family of problems of varying difficulty. The validity of the approach is assessed by considering three different matching scenarios; namely, rigid and non-rigid three-dimensional shape matching, and image matching for Structure from Motion. As demonstrated on both real and synthetic data, our approach leads to an increase in performance of up to one order of magnitude when compared to the standard methods. 1

We consider a parametrized relaxation of the widely adopted quadratic assignment problem (QAP) formulation for minimum distortion correspondence between deformable shapes. In order to control the accuracy/sparsity trade-off we introduce a weighting parameter on the combination of two existing relaxations, namely spectral and game-theoretic. This leads to the introduction of the elastic net penalty function into shape matching problems. In combination with an efficient algorithm to project onto the elastic net ball, we obtain an approach for deformable shape matching with controllable sparsity. Experiments on a standard benchmark confirm the effectiveness of the approach.

“Grasping by caging” has been considered as a powerful tool to deal with uncertainty. In this paper, we continue to explore into “grasping by caging” and propose a new solution by using eigen-shapes and space mapping. For one thing, eigen-shapes fix dexterous hands into a series of finger formations and help to reduce dimensionality and computational complexity. For the other, space mapping builds a mapping between rasterized grids in 2-D Work space (W space) and rasterized voxels in 3-D Configuration space (C space) and helps to rapidly reconstruct C space so that we can efficiently measure the robustness of caging and find an optimal caging configuration for grasping. Our algorithm can work rapidly and squeezingly cage any 2-D shapes, including objects with either convex boundaries, concave boundaries, 1-order or high-order boundaries and even objects with inner holes. We implement the algorithm with MATLAB and carry out experiments with WEBOTS simulation to test its robustness to uncertainties. The results show that our algorithm can work well with various object shapes and can be robust to noisy control and noisy perception. It is promising in the power grasping tasks of dexterous hands.

Caging can offer robustness to uncertainties in grasping. If a robotic hand is designed based on the idea of caging, it would probably work well with noisy perception devices and low-quality control. This paper takes into account these merits and designs and implements a gripping hand based on the idea of caging. The gripping hand is concise and offers a low-cost alternative to co-operate with noisy data and low-quality control. According to previous work, we need four fingers to cage any 2D objects. That is to say, if each finger has one, two or three degree of freedoms, we will totally need four, eight or twelve actuators. The large number of actuators would be costly. This paper simplify the number of actuators into one by quantitatively analyzing finger formations with caging tests conducted on both random objects and objects from MPEG-7 shape database. It successfully lowers costs while maintains high performance. Following the simplified one-actuator design we implement a gripping hand by modifying a SCHUNK RH707 hand and carried out experiments with a manipulator built on the Neuronics Katana arm. The one-actuator gripping hand could work well with common depth cameras (Swiss Ranger) and pick up various objects. It bridges the gap between caging theories and applications and demonstrates the merits of caging.

We propose a weakly-supervised learning method for object detection using color and depth images of a real environment attached with object labels. The proposed method applies Multiple Instance Learning to find proper instances of the objects in training images. This method is novel in the sense that it learns multiple objects simultaneously in a way to balance the scores of each training sample across all object classes. Moreover, we combine 3D features considering different properties, that is, color texture, grayscale texture, and surface curvature, to improve the performance. We show that our method surpasses a conventional method using color and depth images. Furthermore, we evaluate its performance with our new dataset consisting of color and depth images with weak labels of 100 objects and various backgrounds.

This paper presents a caging approach which deals with planar boundary clouds collected from a laser scanner. Given the boundary clouds of a target object and two fixed finger positions, our aim is to find potential third finger positions that can prevent target from escaping into infinity. The major challenge in working with boundary clouds lies in their uncertainty in geometric model fitting and the failure of critical orientations. In this paper, we track canonical motions according to the rotational intersection of Configuration space fingers and rasterize Work space with grids to compute the third caging positions. Our approach can generate the capture region with max(O(np),O(h2)) ≤ O(n2) cost where n denotes the resolution of grid rasterization, p denotes the resolution of canonical rasterization and h denotes the resolution of boundary rasterization or the number of boundary cloud points. Moreover, we propose a rough approximation which measures a subset of the possible positions by contracting rotations, indicating computational complexity of max(O(n),O(h2)). In the experimental part, our proposal is compared with state-of-the-art works and applied to many other objects. The approach makes caging fast and effective.

This paper presents a novel approach to deal with uncertainty in grasping. The basic idea is to initiate a caging manipulation state and then shrink fingers into immobilization to perform a practical grasping. Thanks to flexibility from caging, this procedure is intrinsically safe and gains tolerance towards uncertainty. Besides, we demonstrate that the minimum caging is immobilization and consequently propose using three or four fingers to manipulate planar convex objects in a grasping-by-caging way. Experimental results with physical simulation show the robustness and efficacy of our approach. We expect its leading benefits in saving finger number, conquering low-friction materials and especially, dealing with pose/shape uncertainty.

We propose a novel method of visual anomaly detection for mobile robots in daily real-life settings. Visual anomaly detection using mobile robots is important for security systems or simply for gathering information. However, this task is challenging for two reasons. First, because the number of observed images sampled at the same location is small, anomaly detection systems cannot use standard statistical methods. Second, anomalies must be detected in the presence of other continuous, ambient changes in the visual scene, such as changes in lighting from morning to night. Regarding the former problem, we develop and apply an analysis-by-synthesis-based anomaly detection method for mobile robots. For the latter, we propose a novel definition of anomaly that uses observed samples at other locations to filter out ambient changes that should be ignored by the system. Experimental results demonstrate that our method can detect anomalies from small samples in the presence of ambient changes, which could not be detected by conventional methods.

Sentence generation from images is an ultimate goal of image recognition. In this paper, we attack a novel problem, the "multi-keyphrase problem", to address this goal. We hypothesize that image contents can be described with multi-keyphrases, and that a natural sentence can be generated by connecting multi-keyphrases with an experimental grammar model. Existing methods require semantic knowledge such as labels of an object, action, or scene. Using these methods, we must strive to prepare a highly organized dataset. Therefore, we propose a novel online learning method for multi-keyphrase estimation. The proposed framework, although simple and scalable, can generate sentences from images with no semantic knowledge. Moreover, the proposed method for multi-keyphrase estimation is applicable to image annotation, and it achieves state-of-the-art performance. Our experiment using only images and texts demonstrates that the proposed framework is useful for sentence generation from images.

This paper proposes a novel image representation called a Graphical Gaussian Vector, which is a counterpart of the codebook and local feature matching approaches. In our method, we model the distribution of local features as a Gaussian Markov Random Field (GMRF) which can efficiently represent the spatial relationship among local features. We consider the parameter of GMRF as a feature vector of the image. Using concepts of information geometry, proper parameters and a metric from the GMRF can be obtained. Finally we define a new image feature by embedding the metric into the parameters, which can be directly applied to scalable linear classifiers. Our method obtains superior performance over the state-of-the-art methods in the standard object recognition datasets and comparable performance in the scene dataset. As the proposed method simply calculates the local auto-correlations of local features, it is able to achieve both high classification accuracy and high efficiency.

For the design of artificial systems with the ability to co-create with humans, it is necessary to consider how robots interpret and apply its human partner's actions in its own actions. Nowadays many robots with the ability to co-create with human appeared but few research succeeded in quantifying the interaction between humans and robots, making it difficult to assess the importance of the robot's contribution to the interaction.

Optical flow is a widely used technique for extracting flow information from video images. While it is useful for estimating temporary movement in video images, it only captures one aspect of extracting dominant flow information from a sequence of video images. In this paper, we propose a novel flow extraction approach called causal flow, which can estimate the dominant causal relationships among nearby pixels. We assume flows in video images as pixel-to-pixel information transfer, whereas the optical flow measures the relative motion of pixels. Causal flow is based on the Granger causality test, which measures causal influence based on prediction via vector autoregression, and is widely used in economics and brain science. The experimental results demonstrate that causal flow can extract dominant flow information which cannot be obtained by current methods.

Spatial Pyramid Representation (SPR) is a widely used method for embedding both global and local spatial information into a feature, and it shows good performance in terms of generic image recognition. In SPR, the image is divided into a sequence of increasingly finer grids on each pyramid level. Features are extracted from all of the grid cells and are concatenated to form one huge feature vector. As a result, expensive computational costs are required for both learning and testing. Moreover, because the strategy for partitioning the image at each pyramid level is designed by hand, there is weak theoretical evidence of the appropriate partitioning strategy for good categorization. In this paper, we propose discriminative SPR, which is a new representation that forms the image feature as a weighted sum of semi-local features over all pyramid levels. The weights are automatically selected to maximize a discriminative power. The resulting feature is compact and preserves high discriminative power, even in low dimension. Furthermore, the discriminative SPR can suggest the distinctive cells and the pyramid levels simultaneously by observing the optimal weights generated from the fine grid cells.

Optical flow is a widely used technique for extracting flow information from video images. While it is useful for estimating temporary movement in video images, it only captures one aspect of extracting dominant flow information from a sequence of video images. In this paper, we propose a novel flow extraction approach called causal flow, which can estimate the dominant causal relationships among nearby pixels. We assume flows in video images as pixel-to-pixel information transfer, whereas the optical flow measures the relative motion of pixels. Causal flow is based on Granger causality test, which measures causal influence based on prediction via vector autoregression, and is widely used in economics and brain science. The experimental results demonstrate that causal flow can extract dominant flow information which cannot be obtained by current methods.

Realizing automatic object search by robots in an indoor environment is one of the most important and challenging topics in mobile robot research. If the target object does not exist in a nearby area, the obvious strategy is to go to the area in which it was last observed. We have developed a robot system that collects 3D-scene data in an indoor environment during automatic routine crawling, and also detects objects quickly through a global search of the collected 3D-scene data. The 3D-scene data can be obtained automatically by transforming color images and range images into a set of color voxel data using self-location information. To detect an object, the system moves the bounding box of the target object by a certain step in the color voxel data, extracts 3D features in each box region, and computes the similarity between these features and the target object's features, using an appropriate feature projection learned beforehand. Taking advantage of the additive property of our 3D features, both feature extraction and similarity calculation are considerably accelerated. In the object learning process, the system obtains the feature-projection matrix by weighting unique features of the target object rather than its common features, resulting in reducing object detection errors.

Infants present impressive developmental changes during the first year in almost all domains marked by memory categorization and variability. We propose that one important actor for this developmental shift is the cholinergic innervation of the cortico-hippocampal circuits. Based on neurological observations and developmental studies done in infants, we model how the neuromodulator acetylcholine could be gradually released from the fetal period till the first year in the hippocampal system to support the detection and the sustaining of novel signals. By doing so, the cholinergic system realizes the functional reorganization of the cortico-hippocampal system which can progressively operate then as a working memory for novelty.

In this paper, we propose a news-gathering mobile robot system, and the novel visual anomaly detection method as the core function of news detection in the real world. Visual anomaly detection is important and widely applicable not only to the news-gathering robot but also to the security systems. However, visual anomaly detection from the mobile robot is highly challenging, because the appearances of images captured by the moving robot are dynamically changing. In consequence, the number of observed images at the same location becomes small, and the sampling interval of those images is not constant. To tackle this problem, we developed a new method to incorporate many samples observed at different locations as previous knowledge, which implicitly represent semantically similar to the intended location. Also, we developed a new statistical model, which explicitly considers sampling interval of input images, whereas conventional methods ignore correlation among samples. Experimental results demonstrate that our method outperforms conventional methods, and our mobile robot system including the proposed method finds, investigates, and publishes news of a local community of the real world.

To gain a synthetic understanding of how the body and nervous system co-create animal locomotion, we propose an investigation into a quadruped musculoskeletal robot with biologically realistic morphology and a nervous system. The muscle configuration and sensory feedback of our robot are compatible with the mono- and bi-articular muscles of a quadruped animal and with its muscle spindles and Golgi tendon organs. The nervous system is designed with a biologically plausible model of the spinobulbar system with no pre-defined gait patterns such that mutual entrainment is dynamically created by exploiting the physics of the body. In computer simulations, we found that designing the body and the nervous system of the robot with the characteristics of biological systems increases information regularities in sensorimotor flows by generating complex and coordinated motor patterns. Furthermore, we found similar results in robot experiments with the generation of various coordinated locomotion patterns created in a self-organized manner. Our results demonstrate that the dynamical interaction between the physics of the body with the neural dynamics can shape behavioral patterns for adaptive locomotion in an autonomous fashion.

In this paper, we propose a new, efficient retrieval system for large-scale multi-modal data including video tracks. With large-scale multi-modal data, the huge data size and various contents cause degradation of efficiency and precision of retrieval results. Recent research on image annotation and retrieval shows that image features based on the Bag-of-Visual Words approach with local descriptors such as SIFT perform surprisingly well with large-scale image datasets. Those powerful descriptors tend to be high-dimensional, imposing a high computational cost for approximate nearest neighbor searching in raw feature space. Our video retrieval method is focused on the correlation between image, sound, and location information recorded simultaneously, and to learn conceptual space describing the contents of the data to realize efficient searching. Experiments show good performance of our retrieval system with low memory usage and temporal complexity.

We propose a novel system which generates sentential captions for general images. For people to use numerous images effectively on the web, technologies must be able to explain image contents and must be capable of searching for data that users need. Moreover, images must be described with natural sentences based not only on the names of objects contained in an image but also on their mutual relations. The proposed system uses general images and captions available on the web as training data to generate captions for new images. Furthermore, because the learning cost is independent from the amount of data, the system has scalability, which makes it useful with large-scale data.

For the overwhelming amounts of multimedia used on the Web, methods of search and understanding with sentences are necessary. Representing the contents not only using labels but also using sentences including labels' relations enables users to search with a story and to understand multimedia deeply. However, few existing works describe such sentences because obtaining objects' relations and grammar is difficult. We specifically examine captions of images that are similar to an input image. They are expected to explain the input image to some degree. Therefore, we propose a novel approach to generate a sentential caption for the input image by summarizing those captions. Our experiment using a dataset consisting of images and text demonstrates that the proposed method can generate sentential captions.

There are many robots which have communication skills with human but most of their actions and responses are preprogrammed and few researches inspect psychological aspects of behavior in human-robot co-creation. In this paper, we focus on action-similarity awareness as a key for building up co-creative relationship with humans. Here, action-similarity awareness means that one recognizes affinity to one's own actions derived from other's actions. As a simple task involving action-similarity awareness, we focused on rhythm ensemble. The ensemble is a type of drumming session in which even people who have little musical experience easily take part. In this research we expect to realize the awareness in the rhythm ensemble session by employing the other's past action and mixing it to metrical structure, a basic beat structure in music. To make sure of this idea in real session, we developed a robot system which can actually play a drumming session with humans and control action-similarity awareness. We conducted a subjective experiment and analyzed questionnaire and interaction record. From the questionnaire results, it was confirmed that the sense of prediction increases with action contributing rate and that it is effective to imitate humans actions and add 10% to 30% originality. From the interaction record it was confirmed that phrase structure was shared by making consent on which part to be varied or fixed.

Image auto-annotation is a challenging task in computer vision. The goal of this task is to predict multiple words for generic images automatically. Recent state-of-theart methods are based on a non-parametric approach that uses several visual features to calculate distances between image samples. While this approach is successful from the viewpoint of annotation accuracy, the computational costs, in terms of both complexity and memory use, tend to be high, since non-parametric methods require many training instances to be stored in memory to compute distances from a query. In this paper, we investigate several linear dimensionality reduction methods for efficient image annotation. Using the additional information provided by multiple labels, we can obtain a small representation preserving (and hopefully improving) the semantic distance of a visual feature. Linear methods are computationally reasonable and are suitable for practical large-scale systems, although only limited comparison of such methods is available in this research field. Extensive experiments and analyses on various datasets and visual features show how these simple methods can be applied effectively to image annotation.

Local features provide powerful cues for generic image recognition. An image is represented by a “bag” of local features, which form a probabilistic distribution in the feature space. The problem is how to exploit the distributions efficiently. One of the most successful approaches is the bag-of-keypoints scheme, which can be interpreted as sparse sampling of high-level statistics, in the sense that it describes a complex structure of a local feature distribution using a relatively small number of parameters. In this paper, we propose the opposite approach, dense sampling of low-level statistics. A distribution is represented by a Gaussian in the entire feature space. We define some similarity measures of the distributions based on an information geometry framework and show how this conceptually simple approach can provide a satisfactory performance, comparable to the bag-of-keypoints for scene classification tasks. Furthermore, because our method and bag-of-keypoints illustrate different statistical points, we can further improve classification performance by using both of them in kernels.

The amount of multimedia data on personal devices and the Web is increasing daily. Image browsing and retrieval systems in a low-dimensional space have been widely studied to manage and view large numbers of images. It is essential for such systems to exploit an efficient similarity measure of the images when searching for them. Existing methods use the distance in a low-level image feature space as the similarity measure, and therefore, images with different content may be treated as similar images. In this paper, we propose a novel method to improve the similarity measures for images by considering the text surrounding the images. If there is text describing the images, similarities can be measured more effectively by taking into account the text streams. The proposed method improves the image similarity measures based on the latent semantics obtained from the combination of image and text. It should be noted that the text does not need to be clear tags; indeed, any generic Web text is applicable. Moreover, our method can effectively improve the similarities even if only a small portion of the images include textual descriptions. Additionally, the proposed method is scalable as it has linear computational complexity based on the number of images. In the experiments, we compare our method with previous methods using an original dataset in which a portion of the images are annotated by long text. We show that the proposed method can retrieve semantically similar images more precisely than existing methods.

In this paper we propose a method of high-speed 3D object recognition using linear subspace method and our 3D features. This method can be applied to partial models with any size in any posture. Although it is becoming easy to obtain textured 3D models by a 3D scanner, there are few methods for 3D object recognition which take into account both shape and textures of objects. Moreover, it is difficult to achieve high-speed processing of large 3D data. Our 3D features consider the co-occurrence of shape and colors of an object's surface. The additive property of these features makes it possible to calculate the similarity between a query part and the subspace of each object in a database without division, and therefore the time for recognition is quite short. In the experiments, we compare our method with conventional methods using Spin-Images and Textured Spin-Images. We show that our method is appropriate for 3D object recognition.

Image annotation and retrieval are extremely difficult because of the generic nature of the target images. Generic images contain various miscellaneous objects and scenes. Therefore, desirable annotation results are subjective and underspecified. To overcome this problem, it is important to assume "weak labeling" framework, where images are weakly related to multiple words without region information. In this paper, we propose a high speed and high accuracy image annotation and retrieval method based on efficient learning of the contextual latent space. A distance between samples can be defined in the intrinsic feature space for annotation using latent space learning between images and labels. The proposed method is shown to be faster and more accurate than previously published methods.

This paper presents a system for capturing human movement and tactile data and methods for analyzing this data. We cannot fully capture the essence of motion without tactile information, and sometimes the lack of such information causes critical problems. To achieve a better understanding of motion behavior, we developed a wearable motion capture suit with full-body tactile sensors. We also developed a motion sensor which can estimate its orientation with its inner CPU. We also built a tactile sensor module which can fit many kinds of body shapes. With this system, we can measure a user's movement and tactile information simultaneously. By integrating tactile data with motion data, we can achieve many kinds of meaningful insights. We demonstrate the effectiveness of this system with experiments. We captured two motions: stretching after sitting on a chair and laying down on a bed. By recognizing the contact point from the tactile data and fitting it into the environment, we were able to estimate the motion trajectories.

A joint learning approach is described that meets a major challenge with social robots - developing a methodology for learning communicative behaviors. We focus on interaction rule that is relationship between a robot's action and a partner's response. In this approach a robot is simultaneously a learner and proposer of interaction rules. The human partner and robot continuously search for and co-create new rules as inspired by the social games played between an infant and a caregiver. A simple and universal scheme with response prediction and habituation/dishabituation was developed, and a robot model was built using the scheme. The robot generates actions, observes the partner's response, and get to predict them. It identifies relationships between its actions and the responses, and generates actions designed to elicit particular responses from the partner. After it is habituated to the responses, it generates other actions to search for other rules. In experiments of human-robot interaction based on this model and using a ball, different patterns of interaction emerged, such as passing the ball back and forth, rolling and catching, and feint passing. Response prediction and appropriate habituation supported the emergence of interactions, indicating that the scheme and the model are effective. This joint learning should lead to natural communication between human partners and social robots beyond teach/taught relationship.

To help with care work and rescue operations, it is necessary for humanoid robots to have the ability to transport humans steadily and gently. In this research we consider "piggyback" motions for transporting humans. Most people can perform this motion, allowing us to measure and analyze piggyback motions of human subjects using tactile sensing and whole body movements to design whole body contact control. One interesting result of this investigation is that frictional forces are skillfully controlled by the carrier. In the first experiment, we study a "knack" that allows the carrier to reposition the rider. In the second experiment we verify the effectiveness of the knack in achieving the repositioning result. We also studied the principle of the repositioning motion, and found that it is similar in many ways to a jumping motion. Then we confirmed the validity of our modeling assumptions using a dynamical simulator.

Time series prediction is an important issue in a wide range of areas. There are various real world processes whose states vary continuously, and those processes may have influences on each other. If the past information of one process X improves the predictability of another process Y, X is said to have a causal influence on Y. In order to make good predictions, it is necessary to identify the appropriate causal relationships. In addition, the processes to be modeled may include symbolic data as well as numerical data. Therefore, it is important to deal with symbolic and numerical time series seamlessly when attempting to detect causality.

Image segmentation is an important problem because it is required for many different applications. In particular, visual extraction of an object that is the target of attention or manipulation, is an increasingly important issue in robot vision. In real-world applications, a robot needs to extract an object designated by a human in a complicated environment. There is a large literature on the problem of image segmentation, but most previous methods have a limited ability to extract desired objects from a cluttered scene. Moreover, from the perspective of human-robot interfaces, it is desirable to make it as easy as possible for the user to indicate an object. In this paper, we propose a segmentation method, CD-matting, which can correctly extract a target object in complicated real-world visual situations. This method exploits color and distance information in an integrated way. The system requires only a simple input to designate the target object. We verify the proposed system by real-world experiments. The results show the effectiveness of our method in complicated situations.

Jumping and landing movements are characterized by large instantaneous forces, short duration, and a high uncertainty concerning take off and landing points. Such characteristics make conventional types of control and robot design inadequate. Here we present an approach to realize motor control of jumping and landing which exploits the synergy between control and mechanical structure. Our experimental system is a pneumatically actuated bipedal robot called "Mowgli". Mowgli's artificial musculoskeletal system consists of six McKibben pneumatic muscle actuators including bi-articular muscle and two legs with hip, knee, and ankle joints. Mowgli can reach jump heights of more than 50% of its body height and can land softly. Our results show a proximo-distal sequence of joint extensions during jumping despite simultaneous motor activity. Extensions in the whole body motion are caused by the compliance and the natural dynamics of the legs. In addition to the experiments with the real robot, we also simulated two types of open loop controllers for vertical jumping with disturbance. We found that the model controlled by open loop motor command through a muscle-tendon mechanism could jump robustly. The simulation results demonstrate the contribution of the artificial musculoskeletal system as a physical feedback loop in explosive movements.

In this paper, we developed a tiny orientation device equipped with a wireless network function for a wearable motion capture. The wearable motion capture is defined that it not only measures the posture of the human body but also collects environmental information and the human’s internal state simultaneously and easily. Because the realized device automatically configures wireless networks and is small enough to attach it anywhere, it is easy to gather any sensor information. The feature of the orientation estimation method is that models are switched according to the environment to exclude the effect of motion disturbances. In experiments, by integrating of sole sensors and the orientation sensors, walking around, walking up and down, and jumping tasks were successfully measured. Because it is difficult to measure these motions with only the inertial sensors, it showed the importance of integrating various sensors for acquiring human motion.

We describe the development of a journalist robot system, which generates articles by searching for news in the real world. Our system repeats the steps: (1) autonomous exploration (2) recording of news, and (3) generation of articles. We characterize events with two values: "anomaly" and "relevance" to the user. During the exploration step, images are evaluated using these values. If an interesting event is detected, the robot approaches it to collect additional information. The system then labels the images, and generates a description from the labels. Experiments show the ability of our system to find news-like phenomena and describe images with words.

We present realization of a humanoid which can lift a heavy object by whole body contact. Most humanoid motions are limited to the posture of the end-effectors only landing. In principle these humanoids can not do natural motion. If a humanoid robot is allowed arbitrary contact with the surrounding objects, it can improve the performance and operate a heavier object. We propose a "whole body contact motion" of a humanoid robot. It is defined as a control of contact state of a humanoid robot which has the distributed tactile sensors. We develop conformable and scalable tactile skin and an adult-size humanoid with a smooth surfaces for arbitrary contact. We install the skin on the entire surfaces of the humanoid. Finally we describe the humanoid lifting a 30kg box by tactile feedback.

We present the design and realization of a conformable tactile sensor skin (patent pending). The skin is organized as a network of self-contained modules consisting of tiny pressure-sensitive elements which communicate through a serial bus. By adding or removing modules it is possible to adjust the area covered by the skin as well as the number (and density) of tactile elements. The skin is therefore highly modular and thus intrinsically scalable. Moreover, because the substrate on which the modules are mounted is sufficiently pliable to be folded and stiff enough to be cut, it is possible to freely distribute the individual tactile elements. A tactile skin composed of multiple modules can also be installed on curved surfaces. Due to their easy configurability we call our sensors "cut-and-paste tactile sensors." We describe a prototype implementation of the skin on a humanoid robot

A novel model for dynamic emergence and adaptation of embodied behavior is proposed. A musculo-skeletal system is controlled by a number of chaotic elements, each of which driving a muscle based on local sensory feedback. Thus, the chaotic elements interact with each other through the physical body and the environment. This overall structure is modelled as a coupled chaotic system, which has been known in the complex systems science for its capability of creating and moving among extremely rich variety of ordered patterns. In our model, body-environment interaction dynamics, or embodiment, serves as the chaos coupling field, which is nonlinear and time-varying. Theoretically very little is known about such cases, but since the coupling field directly reflects the current body-environment dynamics, we believe that the emergent ordered patterns correspond to useful motor coordination patterns which immediately get reorganized in response to dynamically changing environmental situation. We implemented the above model and carried out a series of experiments using a dynamics simulator. The results confirmed the above conjecture. In a "muscle-joint" model and a "multi-legged insect" model, the systems autonomously explored and found meaningful motor behaviors within a few seconds. And when the environmental condition changes they immediately created novel motor patterns, which comply with the new situation. Unlike existing learning methods, our model does not require long training period or well designed reward function. The emergence and adaptation takes place immediately, and yet effective in the current embodied situation. Furthermore, we present a methodology to introduce "goal-directedness" to the system without destroying its emergent property.

In this paper, characteristics of a sensory-motor system driven by coupled bursting neurons are examined. We constructed a sensory-motor system by connecting networks of coupled bursting neurons. Mutually connected networks showed several kinds of oscillation rhythms. When networks oscillate at 5-10 Hz, a robot pushed an object in an environment, and when networks oscillate at 20-50 Hz, a robot avoided an object. Spontaneous transition between these rhythms was observed, and was clarified to be an emergent property based on cooperative phenomena of bursting neurons. These studies may provide a basis for utilization of coupled bursting neurons and its primitive applications.

Behavior imitation ability will be a key technology for future human friendly robots. In order to understand the principles and mechanisms of imitation, we take a synthetic cognitive developmental approach, starting with minimum components and create a system that can learn to imitate others. We developed a visuo-motor neural learning system which consists of orientation selective visual movement representation, distributed arm movement representation, and a high-dimensional temporal sequence learning mechanism. The vision and the movement representations model the findings in primate brain, i.e. macaque area MT(or human area V5) and the primary motor area. The learning mechanism is inspired by the finding that there are excessive connections in neonate brain. As our robot explores the visuo-motor self movement patterns, it learns coherent patterns as high-dimensional trajectory attractors. After the learning, a human comes in front of the robot showing arm movements which are similar to the ones in self learning. Although the robot has never seen or programmed to interpret human arm movement, and the detail of visual stimuli are very different, the robot identifies some of the patterns as similar to those in self learning, and responded by generating the previously learned arm movement. In other words, the robot exhibits early imitation ability based on self exploratory learning.

This paper formulates and solves the inverse dynamics problem of deformable robots made entirely of electro-active polymer gel. One of the primary difficulties with deformable robots is that they have conceptually infinite degrees of freedom. We solve this problem through the selection of an essential point to generate a desired motion. The problem is then reduced to trajectory control of a point on the robot. We have proposed dynamic models of electro-active polymers system and derived a variety of motions by applying either spatially or time varying electric fields. However, the motion control problem has not yet been investigated. We show a procedure to realize an inversion (turning over) motion of a starfish-shaped gel robots by applying both spatially varying and time alternating electric fields. This work takes the first step towards motion control of deformable robots.

We present a neural architecture which learns to recognize object-directed actions by visually observing examples. Our architecture learns to extract spatial (e.g. object relationships) and movement contexts, self-organizes symbolic representations of them, and integrates them in a temporal context, producing self-organized symbolic action classes. Each of the above functions is realized by a self organizing neural network module. A preprocessing module takes a video input and feeds object and movement features to the modules. The System can learn to recognize simple grasp-transfer-place actions performed by a human hand in 2D scenes by simply observing example performances. Intermediate and top level categorical representations are self organized without explicit external supervisory signals.

We propose that highly dynamic whole-body motions should be analyzed, realized and exploited for extending the capability of humanoid robots in the real world. Such motions are very different from the today's standard humanoid behaviors such as stable ZMP-based biped walking, and upper-body motion assuming the stability of the lower-body. The kind of motions we are interested are not locally stable states. Sometimes they include diverging trajectories. In this paper, we focus on one example of such motions: a roll-and-rise motion, in which the robot stands up in one action from lying state. It first swings up both of its legs high, swings them down, rolling forward and up on both feet, then extends the legs to achieve the standing posture. Analysis of the dynamics governing the motions is carried out, and some boundary conditions for successful motions are presented. Our current goal is to identify essential minimum control laws that assure the success of the task. In search of them, a series of systematic simulation experiments are carried out to plot the parameter regions which define success or failure. Experiments with real adult-size humanoid robot are also presented.

We present an approach to self-collision detection suitable for complex articulated robots such as humanoids. Preventing self-collisions is vital for the safe operation of robots that generate body trajectories online. Our approach uses a fast distance determination method for convex polyhedra in order to conservatively guarantee that a given trajectory is free of self-collision. Experimental results using an online joystick control application for the humanoid robot "H7" demonstrate the feasibility and effectiveness of the method.

Addresses the extension of a humanoid's action capability by attaching toe joints. The effectiveness of toe joints is discussed in three aspects. One is utilizing it to speed up the walking, another is using it to enable a humanoid to go up higher steps, and the other is using it to whole-body action in which knees are contacting the ground. Feet with toe joints are developed for humanoid 'H6'. An experiment of the wholebody action in which knees are contacting the ground is carried out to show the usefulness of toe joints for such actions. Then the walking pattern generation system is extended to use toe joints. Using this extended system maximum speed of knee joints can be reduced at the same walking speed, and 80% faster walking speed is achieved on humanoid 'H6'.

This paper describes a design of six-axis force sensor that measures ground reaction force of human or humanoid robot. The key concept is parallel support mechanisms, that allow large torques and forces which are caused when foot is hitting to the environment. Basic concept and design of parallel support mechanisms are denoted. Finally ground reaction force measurement system for human walking, and application to humanoid robot walking are described.

This paper proposes an inverse kinematic model for deformable robots made entirely of electro-active polymer gel. The required method is to control a higher degrees of freedom than numbers of input. We (2000, 2001) have been proposed a kinematic and dynamic model of electro-active polymer system and derived a variety of motions of gel robots by applying spatially varying electric fields. However, inverse kinematic model and the method of applying time alternating electric fields have not been investigated. We challenge the tip control of gel manipulator by applying spatially uniform but time varying electric field. We show the procedure to control the tip position of a gel manipulator by dynamically and slightly changing its whole configuration. Our work is a first step towards the shape control of gel robots.

This paper describes the design of an integrated system for humanoid robotics that consists of three key components, 3D vision, motion planning and bipedal control. Layered system design is adopted to achieve concurrency as well as small latency. Implementation using our humanoid type robot H7 and experiments with this architecture are described. The H7 is expected to be a common test-bed in experiments and discussion for various aspects of intelligent humanoid robotics.

We are trying to realize a humanoid which has flexibility. If humanoids have a flexible structure, safety and posture variety can be achieved. We focus on the role of the human spine and muscle-driven system. By having a flexible spine, a humanoid will have safety and many degrees of freedom to realize a variety of postures. By driving joints by tension-controllable tendons, flexibility of the joints can be controlled. We developed a whole-body tendon-driven flexible-spine humanoid named "Kenta". This paper describes the design and control of Kenta, focusing on the design of the spine. The spine consists of ten joints, vertebrae and rubber disks, ribs, and forty muscles equipped with tension sensors. We also propose control methods of the spine. One uses a geometric virtual robot model and another is based on direct teaching. Using these methods, some whole-body motions are presented.

In this paper, a robot that can change the rigidity and shape of its body is called "metamorphic robot". So far robots have rigid bodies and have their degrees of freedom only in the joints. The metamorphic robot can change their bodies fit to the environment by deforming the limbs between the joints. To realize this robot, the softening deformable structure is developed. The structure is made of a low melting point alloy and becomes soft and deformable through the phase change induced by heating and becomes rigid by cooling. In the soft-deformable state, the robot can change the shape by pressing against objects. On the other hand, in the rigid state it is rigid enough to support the robot's body weight keeping the deformation.

This paper presents an efficient online method to generate humanoid walking motions that satisfy desired upper body trajectories while simultaneously carrying objects. A fast motion pattern generation technique that follows the desired ZMP is adopted. In order to satisfy the control input given online, subsequent motion patterns are updated and connected in a stable manner to the old ones while executing. During the creation of motion trajectories online, the commanded motion parameters are checked and modified automatically considering the performance limitations of the hardware. As an example application, we have implemented a one step cycle control system on the Humanoid H7. Experiments controlling the upper body motion and walking direction using a joystick interface are explained to demonstrate the validity of the proposed method.

This paper describes a rapid development system for humanoid vision-based behavior consisting of both real and virtual robots in a simulation environment. Previous robotics simulators were limited to verify dynamic motions such as walking, or to plan or learn in a simple environment. However, our system is able to simulate vision based behavior, i.e. motion and perception behavior, of a robot as a whole, since it can simulate both dynamics and collisions in the environment. We regard the real-time aspect of the simulation as important so that so that users can develop vision based behavior rapidly and efficiently. We employ a Common Interface API to share behavior software between real and virtual robots. Moreover, visual processing functions such as color extraction and depth map generation are available. As a result, vision based behavior consisting of local map generation, planning and navigation of the humanoid in simulation is presented. We also show that software developed in the simulation environment is applicable to real robots.

A novel concept for controlling a humanoid robot, global dynamics, is investigated by motion capture experiments. This concept generalises human/humanoid body motion as successive transitions of "envelopes", where body dynamics is exploited and high level control input is adopted only at the "nodes", where the body is unstable and control input is necessary. Dynamical rising is chosen for our experiment and full body motion is measured. By evaluating the coordination between joint angles, we have seen variation of motion and corresponding envelope volume (stable region within the phase space). Also, by analysis in the phase space (including variables both of positions and their time derivatives), we have seen variations not only according to boundary conditions for the motion (i.e., adopting physical restriction) but also according to experience. Our results suggest that the body dynamics have rich complexity in phase space. Within an envelope, small nodes may exist to give variation and controllability without damaging stability.

This paper presents the final stage of development of the humanoid system, ETL-Humanoid. It is full-scale humanoid system with 46 degrees of freedom, with the height and weight of an average Japanese person. It was designed as an experimental platform, to explore the general principle of controls of complex embodied systems. The complete system will be presented; the mechanical configuration of the system and the low-level network-based control system will also be presented. The final system possesses properties of compactness, modularity and is light in weight. The mechanical system is high in performance, is backdrivable and compliant, allowing the possibility of a wide range of motions and capabilities. The general capability of being able to support itself is demonstrated. A "Chin Up" experiment showing the physical strength of our system is presented. The system is able to support its own body weight while rising up to a supporting bar. Aside from its physical strength, the system is also capable of performing higher-level perceptions and actions. These capabilities will be briefly presented.

While bipedal walking robots developed recently are based on high-gain servo to planned trajectory, our purpose is to make a robot's behavior more natural: exploiting the useful feature of the dynamics of the body itself. A new concept for controlling humanoid robots, the global dynamics approach is proposed. This is to harness the robot body by applying control input intensively around critical points of the planned motion and for the rest of period, rely on passive stability of its own dynamics. As the first step, we numerically studied motion and stability of an elastic body in standing position. A robot with three joints is used and linear springs are adopted for each joints. We show that the body is more stable for small oscillation, if the body is more elastic (i.e., movable joints are many, each spring is soft). Next, we show a way to change the body's dynamical state by changing its dynamical parameters. Combining these two results, the global dynamics method is being developed.

We have developed a system for detecting and tracking human face and eye in an unstructured environment. We adopt a biologically plausible retinally connected neural network architecture and integrate it with an active vision system. While the active vision system tracks the object moving in real time, the neural network detects the face and eye location from the video stream at a slower rate. The paper provides a systematic way of creating and selecting examples for training the network by exploring the link between theory and practice. Experimental results on a real sequence of images from a space-varying sensor depicts the performance of the system.

Human interaction involves a number of factors. One key and noticeable factor is the mass perceptual problem. Humans are equipped with a large number of receptors, equipped for seeing, hearing and touching, to name just a few. These stimuli bombard us continuously, often not on a singular basis. Typically multiple stimuli are activated at once, and in responding to these stimuli, variations of responses are exhibited. The current aim of our project is to provide an architecture, that will enable a humanoid robot to yield meaningful responses to complex and continuous interactions, similar to that of humans. We present our humanoid, a system which is able to simultaneously detect the spatial orientation of a sound source, and is also able to detect and mimic the motion of the upper body of a person. The motion produced by our system is human like-ballistic motion. The focus of the paper is on how we have come about the integration of these components. A continuous interactive experiment is presented in demonstrating our initial effort. The demonstration is in the context of our humanoid interacting with a person. Through the use of spatial hearing and multiple visual cues, the system is able to track a person, while mimicking the persons upper body motion. The system has shown to be robust and tolerable to failure, in performing experiments for a long duration of time.

The aim of this paper is to present our attempt to create a development platform for complex robotic applications. Such a system needs appropriate tools for handling real-time aspects, distributed architectures, portability through heterogeneous hardware, and code re-usability. We show in this paper that, by constraining the shape of an application, specifying its scheduling and building a separate representation of the hardware, it is possible to realize a system where all those aspects are integrated. The major properties of our system is a strong multithreading architecture, the possibility to handle design patterns, and a powerful model of hardware platforms using a hypergraph.

Presents the hardware of the upper body of the ETL-Humanoid system. It has 24 degrees of freedom with high performance actuators, which gives the system high mobility, such as is required for pushups, lifting itself or another person. The mechanical design is able to achieve human proportions, with smooth outer shape, and is light weight at a very high level, considering the high motor performance embedded inside. Custom compact electronics for AC servo control and fast embedded network for control and measurement was also developed. The system is still evolving, being used as a working testbed for versatile human interaction experiments.

CPG (central pattern generator) and entrainment dynamics together form a promising framework for robust and adaptive behavior generation for a high degree of freedom system in unstructured environment. This paper investigates its possibility in the domain of biped robotic locomotion. We extend a previous work on 2D biped locomotion using neural oscillators to 3D, introducing many more degrees of freedom and complexity in control. While the complexity of the problem has been increased, we have simplified the internal neural mechanism compared to the original 2D work. Our fully dynamic 3D simulation experiments showed that our mechanism can generate 3D stable biped stepping motion as well as tolerance against external perturbations.

Great interest has been shown in the application of the principles of artificial life to physically embedded systems such as mobile robots, computer networks, home devices able continuously and autonomously to adapt their behavior to changes of the environments. At the same time researchers have been working on the development of evolvable hardware, and new integrated circuits that are able to adapt their hardware autonomously and in real time in a changing environment. This article describes the navigation task for a real mobile robot and its implementation on evolvable hardware. The robot must track a colored ball, while avoiding obstacles in an environment that is unknown and dynamic. Although a model-free evolution method is not feasible for real-world applications due to the sheer number of possible interactions with the environment, we show that a model-based evolution can reduce these interactions by two orders of magnitude, even when some of the robot's sensors are blinded, thus allowing us to apply evolutionary processes online to obtain a self-adaptive tracking system in the real world, when the implementation is accelerated by the utilization of evolvable hardware.

We have designed and implemented a real-time binocular tracking system which uses two independent cues commonly found in the primary functions of biological visual systems to robustly track moving targets in complex environments, without a-priori knowledge of the target shape or texture: a fast optical flow segmentation algorithm quickly locates independently moving objects for target acquisition and provides a reliable velocity estimate for smooth tracking. In parallel, target position is generated from the output of a zero-disparity filter where a phase-based disparity estimation technique allows dynamic control of the camera vergence do adapt the horopter geometry to the target location. The system takes advantage of the optical properties of our custom-designed foveated wide-angle lenses, which exhibit a wide field of view along with a high resolution fovea. Methods to cope with the distortions introduced by the space-variant resolution, and a robust real-time implementation on a high performance active vision head are presented.

This paper considers the humanoid research as an approach to understanding and realizing complex real world interactions among the robot, environment, and human. As a first step towards extracting a common principle over the three term interactions, the concept of action oriented control has been investigated with simulation example. The complex interaction view casts unique constraints on the design of a humanoid, such as the whole body, smooth shape and non-functional-modular design. A brief description of ongoing design of ETL-humanoid which conforms to the above constraints is presented.

Achieving the first step of a framework for human-robot interaction, we have designed a binocular tracking system which uses disparity and velocity information for the detection and pursuit of moving objects in cluttered environments without a-priori knowledge of the target shape or texture. The implemented system robustly tracks in real-time deformable objects such as human hands and faces, taking advantage of the mechanical and optical properties of ESCHeR, a high performances active vision head equipped with foveated wide-angle lenses.

Inspired by the properties of the human visual system, a new active vision system called ESCHeR (Etl Stereo Compact Head For Robot Vision) has been recently implemented with foveated wide angle lenses. The lenses exhibit a wide field of view along with a space-varying resolution for facilitating both detection and close observation. However, to handle such optical properties and achieve basic eye movement functions, new calibration methods are needed. Therefore, two novel and online techniques are presented that in one case perform a global identification of the optical process through artificial neural techniques and in the other case compute the physical parameters by using environmental feature-tracking and controlled rotations of the cameras. Self-alignment of the cameras is also achieved using a similar technique.

In order to allow robot agents to adapt their behaviour, a new approach-learning by imitation-has been proposed, in which a robot learns novel behaviours through interactions with the environment and other agents. In this paper we describe how our sighted agent, ESCHeR -who is equipped with dual foveated lenses and can control head, neck and eye joints-develops fine oculo-motor control through interaction with the environment. ESCHeR successfully learns to coordinate pan, tilt and vergence such that he can track bright moving objects and saccade rapidly to new objects of interest. In developmental terms, ESCHeR can be considered to have progressed from stage 1 to stage 2 of imitation learning. A novel representation of visual scenes is then introduced, and it is discussed how ESCHeR will use this to progress to stage 3.

We present a novel design and implementation of a foveated wide angle lens. It is strongly motivated towards applications for active vision in complex and dynamic environments. The projection curve is designed so that it facilitates active vision algorithms for motion analysis, object identification, and precise fixation. The implementation achieves small size and light weight so that the lens can be swung at a high speed.

Presents a scheme for specifying and executing purposive navigation tasks for a behaviour-based mobile robot. A user specifies the robot's navigation task in general and qualitative terms using a graphical resource called the purposive map (PM). The robot navigates using the incomplete and approximate information stored in the PM as an aid to achieve the specified mission. The robot is able to augment the knowledge provided in the PM with environment information learnt by the robot. Using the augmented PM, the robot learns how to perform efficient obstacle avoidance. The authors present experimental results using a real robot to show their scheme is robust. The authors' robot can escape from dead-ends, can deduce that goals are unreachable and can withstand disturbances to the environment between missions.

In this paper we describe an extended behavior-based architecture capable of executing purposive tasks. The key to this capability is a novel mechanism for sharing data between behaviors. Simple processing elements called markers ground task related data on sensor data flow and communicate it to the behaviors. A control system based on the architecture controls a mobile robot to help other robots in their transfer tasks. In addition to robot movements the control system controls a stereo gaze platform performing real time stereo tracking. In this paper we concentrate on behavior coordination by markers and realistic behavior-based systems. We present experimental results using a real robot.

A novel framework for multiple robot cooperation called "cooperation by observation" is presented. It introduces many interesting issues such as a viewpoint constraint and role interchange, as well as novel concepts like "attentional structure". The framework has the potential to realize a high level of task coordination by decentralized autonomous robots allowing minimum explicit communication. Its source of power lies in an advanced capability given to each robot for recognizing other agent's actions by (primarily visual) observation. This provides rich information about the current task situation around each robot which facilitates highly-structured task coordination. The basic visuo-motor routines are described. Concrete examples and experiments using real mobile robots are also presented.>

This paper presents some advanced examples of reactive vision-based cooperative behaviors: 1) chasing and posing against another robot among others; 2) unblocking the path of another robot by removing an obstacle; and 3) passing an object from one to another. These behaviors are demonstrated using real mobile robots equipped with CCD cameras, in a complex environment, and with no central controller or explicit communication among the robots. The action observation is based on real time processing of optical flow analysis and stereo tracking. An extended behavior-based architecture for "cooperation by observation" is presented. The core extension consists of a mobile space buffer and an image space buffer with manipulable markers which control the internal flow of information, thereby coordinating parallel behaviors and achieving purposive tasks in complex and dynamic environments.>

This paper presents a stereo active vision system which performs tracking tasks on smoothly moving objects in complex backgrounds. Dynamic control of the vergence angle adapts the horopter geometry to the target position and allows to pick it up easily on the basis of stereoscopic disparity features. We introduce a novel vergence control strategy based on the computation of "virtual horopters" to track a target movement generating rapid changes of disparity. The control strategy is implemented on a binocular head, whose right and left pan angles are controlled independently. Experimental results of gaze holding on a smoothly moving target translating and rotating in a complex surrounding demonstrate the efficiency of the tracking system.>

This paper presents a new scheme of purposive navigation for mobile agents. The new scheme is robust, qualitative and provides a mechanism for combining mapping, planning and mission execution for a mobile agent into a single data structure called the purposive map (PM). The agent can navigate using incomplete and approximate information stored the PM. We present a novel approach to perform obstacle avoidance for a behaviour based robot. Our approach is based on using a physically grounded search while monitoring and co-ordinating behaviours. The physically grounded search exploits stagnation points (local minima) to guide the search for the shortest path to a target. This scheme enables our robot to escape from dead-end situations and allows it to deduce that a target location is unreachable. Simulation results are presented.>

A novel task instruction method for future intelligent robots is presented, In our method, a robot learns reusable task plans by watching a human perform assembly tasks. Functional units and working algorithms for visual recognition and analysis of human action sequences are presented. The overall system is model based and integrated at the symbolic level. Temporal segmentation of a continuous task performance into meaningful units and identification of each operation is processed in real time by concurrent recognition processes under active attention control. Dependency among assembly operations in the recognized action sequence is analyzed, which results in a hierarchical task plan describing the higher level structure of the task. In another workspace with a different initial state, the system re-instantiates and executes the task plan to accomplish an equivalent goal. The effectiveness of our method is supported by experimental results with block assembly tasks.>

Functional units and working algorithms for real-time visual recognition of human pick-and-place action sequences are presented. The action recognizer consists of visual feature detectors, an action/environment model, and an attention stack. It generates a symbolic description of the observed action sequence. Given a different initial state, the system reinstantiates the recognized action sequence to carry out an equivalent assembly task. Experimental results on several assembly tasks support the effectiveness of the method.>

A multi-fingered robot is an artificial organ with many sensors and actuators which require both real-time control and higherlevel motion coordination. A two-layered task architecture for the control of a multi-fingered robot is presented. A control unit, called an agent, is split into two tasks, lower and upper. While a lower task is performing real-time computation, the upper task takes care of communication with other agents. The system is implemented with VxWorks on multi processors and an agent network for the description of a pick-up grasping task is shown.