In human-robot interaction (HRI), the beginning of an interaction is often complex. Whether the robot should communicate with the human is dependent on several situational factors (e.g., the current human’s activity, urgency of the interaction, etc.). We test whether large language models (LLM) and vision language models (VLM) can provide solutions to this problem. We compare four different system-design patterns using LLMs and VLMs, and test on a test set containing 84 human-robot situations. The test set mixes several publicly available datasets and also includes situations where the appropriate action to take is open-ended. Our results using the GPT-4o and Phi-3 Vision model indicate that LLMs and VLMs are capable of handling interaction beginnings when the desired actions are clear. The design using direct image input scored an 89% accuracy on the test set. Of the designs using indirect input, a combined text about human activity and gaze performed best with a 90% accuracy. However, challenges remain in the open-ended situations where the model must choose the priority between the human and robot situation. The design using direct image input mostly prioritized the robot situation, whereas the design with best performance using indirect input mostly prioritized the human situation. Such one-sided behavior could be crucial for practical HRI applications.

Recent large language models (LLMs) are capable of planning robot actions. In this paper, we explore how LLMs can be used for planning actions with tasks involving situational human-robot interaction (HRI). A key problem of applying LLMs in situational HRI is balancing between "respecting the current human’s activity" and "prioritizing the robot’s task," as well as understanding the timing of when to use the LLM to generate an action plan. In this paper, we propose a necessary plan-and-act skill design to solve the above problems. We show that a critical factor for enabling a robot to switch between passive / active interaction behavior is to provide the LLM with an action text about the current robot’s action. We also show that a second-stage question to the LLM (about the next timing to call the LLM) is necessary for planning actions at an appropriate timing. The skill design is applied to an Engage skill and is tested on four distinct interaction scenarios. We show that by using the skill design, LLMs can be leveraged to easily scale to different HRI scenarios with a reasonable success rate reaching 90% on the test scenarios.

In underwater environments, the reflected ultrasonic waves from objects generally provide more than just information about their color and shape for object recognition. Previous studies have overlooked the influence of object pose on these wave components. It is crucial to investigate how these poses affect the reflected wave components because object poses can vary widely and are often unpredictable in real-world scenarios. In this work, we introduce a novel dataset comprising reflected wave components collected from objects made of various materials and observed from various angles. We also show the preliminary evaluations on the performance of machine learning-based material classification on object pose. Our results indicate that the accuracy is consistently high (≥ 91%) for known angles but significantly drops (< 60%) when dealing with unknown angles in most cases. Based on these evaluations, we suggest several directions for future research. Our dataset is available at https://github.com/Nyamotaro/U2R.

Analyzing time-series changes of the contact surface by touching is important to elucidate the touching skills of Humanitude as one of the pervasive multimodal comprehensive care methodologies. For the analysis, there is a frustrated total internal reflection (FTIR) method to capture contact surfaces on a transparent flat plate by a camera. However, this conventional flat plate is far from the actual surfaces of care receivers because the surfaces of humans consist of curved shapes. In this paper, we propose an FTIR sensing system using a transparent curved plate to capture more ideal contact states with a surface shape more similar to the human body. We collect the contact surface data of the beginning of touching motions by Humanitude experts and novices using the FTIR sensing system with the curved and flat plates. Then, we compare the data by the experts and novices in terms of time-series contact areas and the quantitative indices and subjective evaluation and discuss the analysis results. Through these experiments, we confirm that the proposed system has the potential for the novices to perform more correctly the beginning of Humanitude's touching motions.

To achieve recycling of mixed industrial waste toward an advanced sustainable society, waste sorting automation through robots is crucial and urgent. For this purpose, a robot is required to recognize the category, shape, pose, and condition of different waste items and manipulate them according to the category to be sorted. This survey considers three potential difficulties in the sorting automation: 1) End-effector: to robustly grasp and manipulate different waste items with dirt and deformations; 2) Sensor: to recognize the category, shape, and pose of existing objects to be manipulated and the wet and dirty conditions of their surfaces; and 3) Planner: to generate feasible and efficient sequences and trajectories. This survey includes 76 references to studies related to automatic waste sorting and 159 references to worldwide waste recycling attempts. This pioneering investigation reveals the possibility and limitations of conventional systems; thus, providing insights on open issues and potential technologies to achieve a robot-incorporated sorter for the chaotic mixed waste is one of its contributions. This paper further presents a system design policy for readers and discusses future advanced sorters, thereby contributing to the field of robotics and automation.Note to Practitioners—Most automated sorting systems operate for limited target waste items. This study is motivated by the automation of mixed industrial waste treatment facilities using advanced robotic sorters. Emerging advances and increasing functionalities of robot system components will widen system applicability and increase use cases in the chaotic mixed industrial waste domain. This paper surveys the research conducted to date, discusses open issues and potential approaches, and presents user guides that provide practitioners with a system design policy. The user guides created according to the strengths and weaknesses of each system configuration provide future researchers and developers with a useful a priori design policy that has been thus far validated on efficiency, quality, productivity, and reliability. A question-and-answer style guide and a sorting-target-aware previous study reference list allows users to find the desired system configuration, including the investigated components according to their purpose.

High-precision three-dimensional scanning systems have been intensively researched and developed. Recently, for acquisition of large scale scene with high density, simultaneous localisation and mapping (SLAM) technique is preferred because of its simplicity; a single sensor that is moved around freely during 3D scanning. However, to integrate multiple scans, captured data as well as position of each sensor must be highly accurate, making these systems difficult to use in environments not accessible by humans, such as underwater, internal body, or outer space. In this paper, we propose a new, flexible system with multiple line lasers that reconstructs dense and accurate 3D scenes. The advantages of our proposed system are (1) no need of synchronization nor precalibration between lasers and a camera, and (2) the system can reconstruct 3D scenes in extreme conditions, such as underwater. We propose a new self-calibration method leveraging coplanarity and Epipolar constraints is proposed. We also propose a new bundle adjustment (BA) technique that is tailored to the system for a dense integration of multiple line laser scans. Experimental evaluation in both air and underwater environments confirms the advantages of the proposed method.

For assembly tasks, it is essential to fix target parts firmly and accurately estimate their poses. Several rigid jigs for individual parts are frequently used in assembly factories to achieve a precise and time-efficient product assembly. However, providing customized jigs is time-consuming. In this study, to address the lack of versatility in the shapes for which jigs can be used, we developed a flexible jig with a soft membrane including transparent beads and oil with a tuned refractive index. The bead-based jamming transition was accomplished by discharging only the oil, enabling the part to be firmly fixed. Because the two cameras under the jig can capture membrane shape changes, we proposed a sensing method to estimate the orientation of the part based on the behaviors of markers created on the jig's inner surface. Through estimation experiments, the proposed system can estimate the orientation of a cylindrical object with a diameter larger than 50 mm and an RMSE of less than 3°.

To design a general-purpose assembly robot system that can handle objects of various shapes, we propose a soft jig that fits to the shapes of assembly parts. The functionality of the soft jig is based on a jamming gripper developed in the field of soft robotics. The soft jig has a bag covered with a malleable silicone membrane, which has high friction, elongation, and contraction rates for keeping parts fixed. The bag is filled with glass beads to achieve a jamming transition. We propose a method to configure parts-fixing on the soft jig based on contact relations, reachable directions, and the center of gravity of the parts that are fixed on the jig. The usability of the soft jig was evaluated in terms of the fixing performance and versatility for various shapes and postures of parts.

To generate assembly sequences that robots can easily handle, this study tackled assembly sequence generation (ASG) by considering two tradeoff objectives: (1) insertion conditions and (2) degrees of the constraints affecting the assembled parts. We propose a multi-objective genetic algorithm to balance these two objectives. Furthermore, we extend our previously proposed 3D computer-aided design (CAD)-based method for extracting three types of two-part relationship matrices from 3D models that include deformable parts. The interference between deformable and other parts can be determined using scaled part shapes. Our proposed ASG can produce Pareto-optimal sequences for multi-component models with deformable parts such as rubber bands, rubber belts, and roller chains. We further discuss the limitation and applicability of the generated sequences to robotic assembly.

Among various 3D capturing systems, since the system with line lasers based on the light sectioning method is simple and accurate, it has widely attracted many developers and used for many purposes. In addition, there is no need to synchronize the camera and the laser and also the configuration of the camera and the lasers is flexible, and thus, the system can be used for extreme conditions, such as underwater. There are two open problems for the system. The first problem is a low density of the 3D shape obtained from a single image, i.e., just several curves. The second problem is the accuracy of line detection in the wild. In this paper, we propose a self-calibration method using visual odometry (VO) to bundle a large number of frames to increase the density to solve the first problem. We also propose a robust line detection algorithm using CNN to solve the second problem. Comparative experiments prove the effectiveness of our proposed method. In addition, the system was tested in the extreme condition for demonstration.

A robust, easy-to-deploy robot for service tasks in a real environment is difficult to construct. Record-and-playback (R&P) is a method used to teach motor-skills to robots for performing service tasks. However, R&P methods do not scale to challenging tasks where even slight changes in the environment, such as localization errors, would either require trajectory modification or a new demonstration. In this paper, we propose a Sequence-to-Sequence (Seq2Seq) based neural network model to generate robot trajectories in configuration space given a context variable based on real-world measurements in Cartesian space. We use the offset between a target pose and the actual pose after localization as the context variable. The model is trained using a few expert demonstrations collected using teleoperation. We apply our proposed method to the task of toilet cleaning where the robot has to clean the surface of a toilet bowl using a compliant end-effector in a constrained toilet setting. In the experiments, the model is given a novel offset context and it generates a modified robot trajectory for that context. We demonstrate that our proposed model is able to generate trajectories for unseen setups and the executed trajectory results in cleaning of the toilet bowl.

For a robotic gripper to perform as well as the human hand, the performance of the tactile sensing and grasping of the gripper must be improved. In this study, a parallel gripper with a universal fingertip device that combines both object holding using jamming transition and optical sensing is proposed. By using a transparent material along with the jamming transition, both a tactile sense is achieved via optical sensing and the holding stability is improved by tightly restraining the object. Because the hardness of the contact region of the membrane can be adjusted, the gripper can grasp heavy objects as well as light objects. In this paper, the structure of the developed gripper and its grasping characteristics are described, and the experimental results from grasping and sensing are presented. It is experimentally shown that the proposed gripper is capable of both grasping fragile objects (such as an egg and tofu) by using tactile sensing and grasping dumbbells (with weights between 1-3 kg) by using jamming transition. Additionally, grasping a plastic cup, the proposed gripper demonstrated the potential of combining tactile sensing and more-constrained grasping.

The appearance of android robots is very similar to that of human beings. From their appearance, we expect that androids might provide us with high-level communication. The imitation of human behavior gives us the feeling of natural behavior even if we do not know what drives high-level communication. In this paper, we evaluate the imitation of human eye behavior by an android. We consider that the android imitates human eye behavior while explaining some research topic and a person acts as a listener. Then, we construct a method to imitate the eye behavior obtained from eye trackers. For the evaluation, we asked seventeen male subjects for their subjective evaluation and compared the imitation with an android that controlled eye-contact duration and eyeblinks by editing the imitation or programming rule-based behavior. From the results, we found out that 1) the rule-based behaviors kept human-likeness, 2) 3-second eye contact obtained better scores regardless of the imitation-based or rule-based eye behavior, and 3) the subjects might regard the longer eyeblinks as voluntary eyeblinks, with the intention to break eye contacts.

As sensing and image recognition technologies advance, the environments where service robots operate expand into human-centered environments. Since the roles of service robots depend on the user situations, it is important for the robots to understand human intentions. Gaze information, such as gazed objects (i. e., the objects humans are looking at) can help to understand the users' intentions. In this paper, we propose a real-time gazed object identification method from RGBD images captured by a camera mounted on a mobile service robot. First, we search for the candidate gazed objects using state-of-the-art, real-time object detection. Second, we estimate the human face direction using facial landmarks extracted by a real-time face detection tool. Then, by searching for an object along the estimated face direction, we identify the gazed object. If the gazed object identification fails even though a user is looking at an object, i. e., has a fixed gaze direction, the robot can determine whether the object is inside or outside the robot's view based on the face direction, and, then, change its point of view to improve the identification. Finally, through multiple evaluation experiments with the mobile service robot Pepper, we verified the effectiveness of the proposed identification and the improvement of the identification accuracy by changing the robot's point of view.

In this work we propose a novel approach to remove undesired objects from RGB-D sequences captured with freely moving cameras, which enables static 3D reconstruction. Our method jointly uses existing information from multiple frames as well as generates new one via inpainting techniques. We use balanced rules to select source frames; local homography based image warping method for alignment and Markov random field (MRF) based approach for combining existing information. For the left holes, we employ exemplar based multi-view inpainting method to deal with the color image and coherently use it as guidance to complete the depth correspondence. Experiments show that our approach is qualified for removing the undesired objects and inpainting the holes.

We propose a new neural network called Temporal-enhanced Convolutional Network (T-CN) for video-based person reidentification. For each video sequence of a person, a spatial convolutional subnet is first applied to each frame for representing appearance information, and then a temporal convolutional subnet links small ranges of continuous frames to extract local motion information. Such spatial and temporal convolutions together construct our T-CN based representation. Finally, a recurrent network is utilized to further explore global dynamics, followed by temporal pooling to generate an overall feature vector for the whole sequence. In the training stage, a Siamese network architecture is adopted to jointly optimize all the components with losses covering both identification and verification. In the testing stage, our network generates an overall discriminative feature representation for each input video sequence (whose length may vary a lot) in a feed-forward way, and even a simple Euclidean distance based matching can generate good re-identification results. Experiments on the most widely used benchmark datasets demonstrate the superiority of our proposal, in comparison with the state-of-the-art.

The universal gripper has attracted attention due to its simple structure and advanced grasping ability for irregularly shaped objects. In this research, we propose a novel design for a granular-jamming-based gripper which uses a transparent filling and a semi-transparent membrane to allow optical sensing to detect both deformation of the membrane and the object being grasped. By adjusting the refractive index of an oil mixture to the refractive index of the granular bodies, we produced a fully transparent filling that allows the use of a camera inside the universal gripper. In this paper, we present the materials and development of our prototype, and describe the experimental confirmation of the prototype's performance. We showed that our prototype was able to grasp cylindrical and rectangular objects between 10 to 70 mm length while also tracking the deformation of the gripper.

Since the appearance of androids is very similar to the appearance of humans, androids are expected to perform human-like interactions. Among these interactions, telling a lie is especially interesting since such an interaction is not always malicious and is employed due to social politeness and to protect self-esteem of the person who tells a lie. In this paper, we develop and evaluate an android that performs humanlike subconscious behaviors when telling a lie in a simple game. For this purpose, we design a game to evaluate the effectiveness of such behaviors. Based on a literature review, we devise a strategy to implement such behaviors on the android. By observing the game being played between two subjects, we analyze and implement the behaviors obtained. The experimental results indicated that the android exhibiting the human-like subconscious behaviors gave users an impression of higher affinity, activity, and honesty. Finally, we concluded that playing games with the android was more amusing for subjects than simple interaction.

Robot usage in the fields of human support and healthcare is expanding. Robotic devices to assist humans in the self-feeding task have been developed to help patients with limited mobility in the upper limbs but the acceptance of these robots has been limited. In this work, we investigate how to quantitatively evaluate the comfort of an eating assistive device by estimating the interaction forces between the human and the robot when eating. Rather than using expensive or commercially unavailable devices to directly measure the forces involved in feeding, we use an accelerometer to estimate these forces, which are calculated using a previously observed estimation of the system mass and the measured acceleration during the feeding process. We experimentally verify our concept with a commercially-available eating assistive device and a human subject. The evaluation results demonstrate the feasibility of our approach.

Many studies of hand motion recognition using a surface electromyogram (sEMG) have been conducted. However, it is difficult to get the activity of deep layer muscles from an sEMG. The pronation and supination of the forearm are caused by the activities of deep layer muscles. These motions are important in grasping and manipulating daily objects. We think it is possible to accurately recognize hand motions from the activity of the deep layer muscles using the forearm deformation. Forearm deformation is caused by a complex motion of the surface and deep layer muscles, tendons, and bones. In this study, we propose a novel hand motion recognition method based on measuring forearm deformation with a distance sensor array. The distance sensor array is designed based on a 3D model of the forearm. It can measure small deformations because the shape of the array is designed to fit the neutral position of the forearm. A Support Vector Machine (SVM) is used to recognize seven types of hand motion. Two types of features are extracted for the recognition based on the time difference of the forearm deformation. Using the proposed method, we perform hand motion recognition experiments. The experimental results showed that the proposed method correctly recognized hand motions caused by the activity of both surface and deep layer muscles, including the pronation and supination of the forearm. Moreover, the hand opening of small deformation motions was correctly recognized.

Moving to the location designated by the user is the most fundamental task for a mobile robot. Remote control is one of the effective solutions to navigate the robot to the target location, but the user suffers from the burden to continuously concentrate on the remote control. As the result, several users are required in accordance with the number of robots to operate. In this paper, we propose the remote control system that uses touch panel interface, simultaneous localization and mapping (SLAM), and motion planning to achieve autonomy of mobile robots. To navigate the robot in the proposed system, the user only designates the destination and via-points by touching on the map estimated by the SLAM. After receiving the user's input, the Rapidly-exploring Random Tree (RRT) generates the feasible path to the destination using the estimated map. The effectiveness of the proposed system is verified through experiments where multiple mobile robots are remotely controlled.

Unlike conventional portable eye-tracking methods that estimate the position of the mounted camera using 2-D image coordinates, the techniques that are proposed here present richer information about person's gaze when moving over a wide area. They also include visualizing scanpaths when the user with a head-mounted device makes natural head movements. We employ a Visual SLAM technique to estimate the head pose and extract environmental information. When the person's head moves, the proposed method obtains a 3-D point-of-regard. Furthermore, scanpaths can be appropriately overlaid on image sequences to support quantitative analysis. Additionally, a 3-D environment is employed to detect objects of focus and to visualize an attention map.

The goal of this work is to teach a robot to regrasp an object using knowledge obtained from human demonstration. This paper presents a task model that represents a human regrasping movement. The task model is based on the topological information and comprised of four task primitives. Human regrasping movement is recognised and represented as a sequence of these task primitives by the proposed recognition algorithm. The proposed method then maps each task primitive to the target robot hand using knowledge obtained from human demonstration. The experimental result verified the proposed task model by executing the regrasping movement on the real robot hand.

The Human-Robot Interaction gets increasingly closer. In consequence, human safety has become a key issue for the success of the symbiosis between humans and robots. When the minimum distance between a human and a robot is too short, it can be naturally considered that the probability of a collision increases. Therefore, we consider that the robot should increase the distance to the human when the human is getting closer. We propose Withdrawal strategy as a method that aims to increase the distance by moving the end-effector not only away from the human but also to a parking position that can be previously assessed to be safer. To withdraw the end-effector, we use a virtual force model consisting of two virtual forces: a repelling force exerted by the human and an attractive force exerted by the parking position. We carry out experiments using a human-sized humanoid robot and five human subjects, and report the task completion time to evaluate the efficiency of the robot when performing a simple task.

In this paper, we propose a producing method of softness sensation using device vibration. The basic idea of the softness producing method is to provide a softness illusion of stimulating cutaneous mechanoreceptor by vibration. A prototype device is developed to verify effect of parameters of the vibration: frequency, amplitude and wave shape, on softness sensation. We create a database which indicates relationship between softness sensation and the vibration parameters to control the prototype device. Then subjective experiment is conducted for validation of the softness producing method. Experimental result shows that the proposing method can be used for softness sensation producing with valid resolution.

Natural body gesturing and speech dialogue, is crucial for human-robot interaction (HRI) and human-robot symbiosis. Real interaction is not only with one-to-one communication but also among multiple people. We have therefore developed a system that can adjust gestures and facial expressions based on a speaker's location or situation for multi-party communication. By extending our already developed real-time gesture planning method, we propose a gesture adjustment suitable for human demand through motion parameterization and gaze motion planning, which allows communication through eye-to-eye contact. We implemented the proposed motion planning method on an android Actroid-SIT and we proposed to use a Key-Value Store to connect the components of our systems. The Key-Value Store is a high-speed and lightweight dictionary database with parallelism and scalability. We conducted multi-party HRI experiments for 1,662 subjects in total. In our HRI system, over 60% of subjects started speaking to the Actroid, and the residence time of their communication also became longer. In addition, we confirmed our system gave humans a more sophisticated impression of the Actroid.

Natural body gesture, as well as speech dialog, is crucial for human-robot interaction and human-robot symbiosis. We have already proposed a real-time gesture planning method. In this paper, we afford this method more flexibility by adding motion parameterization function. Especially in multi-person HRI, this function becomes more important because of its adaptation to changes of a speaker's and/or object's locations. We implement our method for multi-person HRI system on the android Actroid-SIT, and conduct two experiments for estimating the precision of gestures and the human impressions about the Actroid. Through these experiments, we confirmed our method gives humans a more sophisticated impressions.

In this paper, we propose a method for semantic motion retrieval in large data sets of human motions to classify body gestures automatically. This method extracts spatio-temporal features from the motions by expressing them in frequency domain. And these features are transformed into the Bag-of-words representation to accelerate the calculation and to emphasize the semantic aspect. The method is inspired by techniques of natural language processing or image processing. We conducted experiments for evaluating the performance of the motion classification using data sets captured by a motion capture system. Through the experiments, we confirmed that our method improves the performance of the motion classification and reduces the computational time drastically.

The quantitative evaluation of product usability is important for product design. In this paper, we propose an evaluation system for product usability using a human mimetic robot hand with a tendon skeletal model. The proposed system enables us to evaluate product usability of a physical object, such as a prototype. To realize the system, we developed a robot hand which has a human mimetic structure. The design concepts of the robot hand are to imitate the human finger size, degree of freedom and fingertip structure. Ease of control is also characteristic of the robot hand. We propose to use tendon forces to evaluate the product usability. The joint torques during manipulation of the product are obtained from sensors on the robot hand. The tendon forces are calculated from the joint torques using the tendon skeletal model. Precise imitation of human finger structure makes the sensed joint torques similar to those of humans. The obtained joint torques are expected to be similar to those of humans. To evaluate the effectiveness of the proposed system, we conduct the experiments where the robot finger pushes cell phone button to evaluate its usability. The experimental result is verified by comparing with the existing questionnaire results from Ikeda [1]. The calculation score of the proposed system reflects the questionnaire result. These results indicate that the proposed system can quantitatively evaluate the actual product usability.

Androids are potentially required to show human-like behavior, because their appearance resembles humans' physical features. Therefore, we propose a gaze motion planning method. Within this method, we control the convergence of eyes and the ratio of eye angle to head angle, which leads to a more precise estimation of gaze direction. We implemented our method on the android Actroid-SIT and conducted experiments for evaluation of the effects of our method. Through these experiments, we achieved a common guidance for androids when planning more precise gaze motion.

We present a novel non-rigid surface registration method that achieves high accuracy and matches characteristic features without manual intervention. The key insight is to consider the entire shape as a collection of local structures that individually undergo rigid transformations to collectively deform the global structure. We realize this locally rigid but globally non-rigid surface registration with a newly derived dual-grid Free-form Deformation (FFD) framework. We first represent the source and target shapes with their signed distance fields (SDF). We then superimpose a sampling grid onto a conventional FFD grid that is dual to the control points. Each control point is then iteratively translated by a rigid transformation that minimizes the difference between two SDFs within the corresponding sampling region. The translated control points then interpolate the embedding space within the FFD grid and determine the overall deformation. The experimental results clearly demonstrate that our method is capable of overcoming the difficulty of preserving and matching local features.

We propose a method for estimating inherent surface color robustly against image noises from two registered images taken under different outdoor illuminations. We formulate the estimation based on maximum likelihood manner while considering both inter-pixel and intra-pixel relationships. We define inter-pixel relationship based on stochastic behavior of image noises and properties of outdoor illumination chromaticity. We rely on the spatial continuity of both surface color and illumination to define intra-pixel relationship. We also propose to maximize the estimation function in two step manner. Experimental results demonstrate the significant improvement of the proposed method in estimation accuracy compared to previous methods.

We propose a method for estimating demosaicing algorithms from image noise variance. We show that the noise variance in interpolated pixels becomes smaller than that of directly observed pixels without interpolation. Our method capitalizes on the spatial variation of image noise variance in demosaiced images to estimate the color filter array patterns and demosaicing algorithms. We verify the effectiveness of the proposed method using various images demosaiced with different demosaicing algorithms extensively.

The portability of an eye tracking system encourages us to develop a technique for estimating 3D point-of-regard. Unlike conventional methods, which estimate the position in the 2D image coordinates of the mounted camera, such a technique can represent richer gaze information of the human moving in the larger area. In this paper, we propose a method for estimating the 3D point-of-regard and a visualization technique of gaze trajectories under natural head movements for the head-mounted device. We employ visual SLAM technique to estimate head configuration and extract environmental information. Even in cases where the head moves dynamically, the proposed method could obtain 3D point-of-regard. Additionally, gaze trajectories are appropriately overlaid on the scene camera image.

In this research, a Universal map, which can be converted to individual maps for heterogeneous mobile robots, is proposed. A Universal map can be generated using our developed measurement robot, and it is composed of a textured 3D environment model. Therefore, every robot can use a Universal map as a common map, and it is utilized for various localization technologies such as view-based and LRF-based methods. In LRF-based localization, accurate localization is achieved using a specific map, which is generated from Universal map. In a view-based approach, localization and navigation are achieved using rendered images. The use of a Universal map enables generation of these maps automatically. The effectiveness of this approach is confirmed through experiments.

Generating natural motion of an articulated object with higher DOF (e.g., humanoid robot and robot hand) is a crucial issue in robotics and computer graphics fields. Use of the motion capture data is one of the solutions, but it requires expensive device and time-consuming measurement. In this paper, we propose a method for generating natural hand motion to play a piano from the inputted music score. The proposed method uses inverse kinematics while considering naturalness of hand poses. We revisit background of the inverse kinematics based on the maximum likelihood estimation and use the prior model of the hand pose to achieve the naturalness. We evaluate the effectiveness of the proposed method using voluntary survey.

Representing 2D and 3D data sets with implicit polynomials (IPs) has been attractive because of its applicability to various computer vision issues. Therefore, many IP fitting methods have already been proposed. However, the existing fitting methods can be and need to be improved with respect to computational cost for deciding on the appropriate degree of the IP representation and to fitting accuracy, while still maintaining the stability of the fit. We propose a stable method for accurate fitting that automatically determines the moderate degree required. Our method increases the degree of IP until a satisfactory fitting result is obtained. The incrementability of QR decomposition with Gram-Schmidt orthogonalization gives our method computational efficiency. Furthermore, since the decomposition detects the instability element precisely, our method can selectively apply ridge regression-based constraints to that element only. As a result, our method achieves computational stability while maintaining fitting accuracy. Experimental results demonstrate the effectiveness of our method compared with prior methods.

Continuous action sets are used in many reinforcement learning (RL) applications in robot control since the control input is continuous. However, discrete action sets also have the advantages of ease of implementation and compatibility with some sophisticated RL methods, such as the Dyna [1]. However, one of the problem is the absence of general principles on designing a discrete action set for robot control in higher dimensional input space. In this paper, we propose to construct a discrete action set given a set of basis functions (BFs). We designed the action set so that the size of the set is proportional to the number of the BFs. This method can exploit the function approximator's nature, that is, in practical RL applications, the number of BFs does not increase exponentially with the dimension of the state space (e.g. [2]). Thus, the size of the proposed action set does not increase exponentially with the dimension of the input space. We apply an RL with the proposed action set to a robot navigation task and a crawling and a jumping tasks. The simulation results demonstrate that the proposed action set has the advantages of improved learning speed, and better ability to acquire performance, compared to a conventional discrete action set.

We propose a view-based indoor/outdoor navigation method as an extension of the view-sequence navigation. The original view-sequence navigation method uses the template matching method with normalized correlation for localization. Because the matching method is sensitive to local illumination changes, it is only used for indoor environment. In this paper, we propose to adopt the accumulated block matching method to improve robustness against locally changing illumination, in which a template is split into small patches and matched by maximizing the average of the normalized correlations of all the patches.We also propose a localization criterion which helps the robot decide its motion. Our experimental results demonstrate that the proposed methods can be applied to both indoor and outdoor environments.

This paper presents a new continuous action space for reinforcement learning (RL) with the wire-fitting. The wire-fitting has a desirable feature to be used with action value function based RL algorithms. However, the wire-fitting becomes unstable caused by changing the parameters of actions. Furthermore, the acquired behavior highly depend on the initial values of the parameters. The proposed action space is expanded from the DCOB, proposed by Yamaguchi et al., where the discrete action set is generated from given basis functions. Based on the DCOB, we apply some constraints to the parameters in order to obtain stability. Furthermore, we also describe a proper way to initialize the parameters. The simulation results demonstrate that the proposed method outperforms the wire-fitting. On the other hand, the resulting performance of the proposed method is the same as, or inferior to the DCOB. This paper also discuss about this result.

We propose a method for estimating camera response functions using a probabilistic intensity similarity measure. The similarity measure represents the likelihood of two intensity observations corresponding to the same scene radiance in the presence of noise. We show that the response function and the intensity similarity measure are strongly related. Our method requires several input images of a static scene taken from the same viewing position with fixed camera parameters. Noise causes pixel values at the same pixel coordinate to vary in these images, even though they measure the same scene radiance. We use these fluctuations to estimate the response function by maximizing the intensity similarity function for all pixels. Unlike prior noise-based estimation methods, our method requires only a small number of images, so it works with digital cameras as well as video cameras. Moreover, our method does not rely on any special image processing or statistical prior models. Real-world experiments using different cameras demonstrate the effectiveness of the technique.

The learning from observation (LFO) paradigm has been widely applied in various types of robot systems. It helps reduce the work of the programmer. However, the applications of available systems are limited to manipulation of rigid objects. Manipulation of deformable objects is rarely considered, because it is difficult to design a method for representing states of deformable objects and operations against them. Furthermore, too many operations are possible on them. In this paper, we choose knot tying as a case study for manipulating deformable objects, because the knot theory is available and the types of operations possible in knot tying are limited. We propose a knot planning from observation (KPO) paradigm, a KPO theory, and a KPO system.

Learning from Observation (LFO) has been widely applied in various types of robot system. It helps reduce the work of the programmer. But the available systems have application limited to rigid objects. Deformable objects are not considered because: 1) it is difficult to describe their state, and 2) too many operations are possible on them. In this paper, we choose the knot tying as case study for operating on nonrigid bodies, because a "knot theory" is available and the type of operations is limited. We describe the Knot Planning from Observation (KPO) paradigm, KPO theory and KPO system.

To learn a new everyday task under the "Learning from Observation" framework, the system needs to detect which parts of the demonstration are essential to complete the task without task-dependent knowledge. In the previous research, we proposed a technique to estimate essential interactions in a task by integrating multiple demonstrations which represent virtually the same task. Although, the technique could automatically segment the essential interactions and determine the number of the interactions, the segmentation algorithm depends on some heuristics and only stationary interactions could be obtained. In this paper, a novel technique is proposed, which overcomes this limitation and can estimate almost any types of interactions. In this approach, a demonstrator needs to give a explicit signal once during each essential interaction as a hint on the occurrence of the essential interaction. From visual information and these signals, the system automatically analyzes the essential parts of the task and their periods, and also detects which environmental objects are interacted with the manipulated object. These information is hard to be obtained from a single demonstration, because of the ambiguity in interpreting the interaction especially in cluttered environment. The proposed method is evaluated in a simulation and also in a real world by using a humanoid robot.

Various methods to recognize assembly tasks using possible local displacement of objects have been proposed. To calculate this displacement, the screw theory is employed. It is equivalent to the first order Taylor expansion of the displacement. However, such methods can treat polyhedral objects only. Because the screw theory cannot treat curvature information of objects. In this paper, we propose a method to calculate possible local displacement of curve objects using improved screw theory, which is equivalent to the second order Taylor expansion of the displacement, and verify the validity of the proposed method.

A technique to recognize the shape of a grasping hand during manipulation tasks is proposed; which utilizes 3D articulated hand model and a reconstructed 3D volume from infrared cameras. Vision-based recognition of a grasping hand is a tough problem, because a hand may be partially occluded by a grasped object and the ratio of occlusion changes along the progress of the task. To recognize the shape in a single time frame, a robust recognition method of an articulated object is proposed. In this method, 3D volumetric representation of a hand is reconstructed from multiple silhouette images and 3D articulated object model is fitted to be reconstructed data to estimate the pose and the joint angles. To deal with large occlusion, a technique to simultaneously estimate time series reconstructed volumes with the above method is proposed, which can automatically suppress the effect form badly reconstructed volumes. The proposed techniques are verified in simulation as well as in a real world.

This paper describes a new approach on how to teach a robot everyday manipulation tasks under the "learning from observation" framework. Most of the approaches so far assume that a demonstration can be well understood from a single demonstration. However, a single demonstration contains ambiguity, in that interactions which are essential to complete a task cannot be discerned without prior task dependent knowledge, which should be obtained from observation. To address these issues, we propose a technique to integrate multiple observations of demonstrations. The demonstrations differ, but are virtually the same task. The shared interactions among all the demonstrations are considered to be essential and we form a task model from their symbolic representations. Then the relative trajectories corresponding to each essential interaction are generalized by calculating their mean and variance and are also stored in the task model, which is used to reproduce a skilled behavior. We examine this approach by using a human-form robot, which successfully imitates human demonstrations of everyday tasks.

This paper describes a new approach on how to teach everyday manipulation tasks to a robot under the "Learning from Observation" framework. In our previous work, to acquire low-level action primitives of a task automatically, we proposed a technique to estimate essential interactions to complete a task by integrating multiple observations of similar demonstrations. But after many demonstrations are performed, there may be interactions which are the same in nature. These identical interactions should be grouped so that each action primitive becomes unique. For this purpose, a Hidden Markov Model based clustering algorithm is presented which automatically determines the number of independent interactions. We also show that the obtained interactions can be used as discriminators of human behavior. Finally, simulation and experimental results in which a real humanoid robot learns and recognizes essential actions by observing demonstrations are presented.

The learning-from-demonstration method is considered for a novel robot-programming style. It consists of two parts: 1) to recognize human performance from observation as sequential motion primitives; and 2) to execute the same performance. We (2000) have proposed a method to recognize assembly tasks. However, the execution requires the ability to convert motion primitives to collision free paths. In this paper, we describe a method to calculate collision free paths. Many researchers have proposed to calculate collision free paths using analytical methods, potential fields or probabilistic methods. Potential and probabilistic methods are very powerful tools on a computer, but their solutions are not optimal. We propose a method to calculate optimal collision free paths analytically.

The local displacement of an object is very useful for deciding grasp stability, generating trajectories, recognizing assembly tasks, and so on. To calculate this displacement, the screw theory is employed. It is equivalent to the first order Taylor expansion of the displacement. The screw theory is very convenient, because the displacement is formulated as simultaneous linear inequalities, and a powerful tool to calculate such inequalities, the theory of the polyhedral convex cones, has already been established. However, truncation errors introduced by first order approximations sometimes cause mistaken results. In this paper, we improve the screw theory by using 2nd order terms and verify the validity of the result.

The completion of robot programs requires long development time and much effort. To shorten the programming time and to minimize the effort, we have been developing a system which we refer to as the "assembly-plan-from-observation (APO) system." This system requires assembly task recognition from observing human performance. Observation data by a robot's vision system is usually error contaminated, and, thus, we cannot use those data directly. This paper proposes two methods to clean up those errors by using contact relations and their transitions. The first one corrects the observed configuration from contact relations observed. The second one identifies wrongly determined contact relations from an analysis of configuration space (C-space). We have implemented both methods on our test bed and have verified their effectiveness.

The completion of robot programs requires long development time and much effort. To shorten this programming time and minimize the effort, we have been developing a system which we refer to as "assembly-plan-from-observation (APO) system;" this system provides the ability for a robot to observe a human performing an assembly tasks, understand the tasks, and subsequently generate a program to perform that same task. One of the necessary tasks in APO is to create a trajectory of robot hand movement from observing human performance. The previous system developed a direct observation method based on the trajectory of a human movement. Though simple and handy, the system was susceptible to noise. This paper proposes a method to make the observation robust against noise by using symbolic representations of a trajectory based on contact analysis. The system divides the trajectory into small segments based on the contact analysis, then allocates an operation element referred to as a sub-skill to those segments; the result is a robust trajectory-based APO system.

The completion of robot programs requires a long development time and much effort. To shorten this programming time and to minimize the effort, we have been developing a system which we refer to as the "Assembly Plan from Observation (APO) system". This system provides the ability for a robot to observe a human performing an assembly task, to recognize the task and to then generate a program to perform that same task. One of the necessary tasks in APO is to create the trajectory of a robot hand movement after having observed a human's performance. The previous system developed a direct observation method based on the trajectory of a human's movement. Although it was simple and handy, the system was susceptible to noise. This paper proposes a method to make the observation robust against noise by analyzing topological contact relations. The system divides the trajectory into small segments based on this contact analysis, and then allocates an operation element, referred to as a "sub-skill", to those segments. The result is a robust, trajectory-based APO system.