In this research, we propose a method of estimating object class and orientation given multiple input images assuming the relative camera orientations are known. Input images are transformed to descriptors on 2-D manifolds defined for each class of object through a CNN, and the object class and orientation that minimize the distance between input descriptors and the descriptors associated with the estimated object class and orientation are selected. The object orientation is further optimized by interpolating the viewpoints in the database.The usefulness of the proposed method is demonstrated by comparative evaluation with other methods using publicly available datasets. The usefulness of the proposed method is also demonstrated by recognizing images taken from the cameras on our humanoid robot using our own dataset.

Person recognition from gait images is generally not robust to changes in appearance, such as variations of the walking direction. In general conventional methods have focused on training a model to transform gait features or gait images to those at a different viewpoint, but the performance gets worse in case the model is not trained at a viewpoint of a subject. In this paper we propose a novel gait recognition approach which differs a lot from existing approaches in that the subject's sequential 3D models and his/her motion are directly reconstructed from captured images, and arbitrary viewpoint images are synthesized from the reconstructed 3D models for the purpose of gait recognition robust to changes in the walking direction. Moreover, we propose a gait feature, named Frame Difference Frieze Pattern (FDFP), which is robust to high frequency noise. The efficiency of the proposed method is demonstrated through experiments using a database that includes 41 subjects.

Conventional methods of gait analysis for person identification use features extracted from a sequence of camera images taken during one or more gait cycles. An implicit assumption is made that the walking direction does not change. However, cameras deployed in real-world environments (and often placed at corners) capture images of humans who walk on paths that, for a variety of reasons, such as turning corners or avoiding obstacles, are not straight but curved. This change of the direction of the velocity vector causes a decrease in performance for conventional methods. In this paper we address this aspect, and propose a method that offers improved identification results for people walking on curved trajectories. The large diversity of curved trajectories makes the collection of complete real world data infeasible. The proposed method utilizes a 4D gait database consisting of multiple 3D shape models of walking subjects and adaptive virtual image synthesis. Each frame, for the duration of a gait cycle, is used to estimate a walking direction for the subject, and consequently a virtual image corresponding to this estimated direction is synthesized from the 4D gait database. The identification uses affine moment invariants as gait features. Experiments using the 4D gait database of 21 subjects show that the proposed method has a higher recognition performance than conventional methods.

This paper presents an approximate belief propagation algorithm that replaces outgoing messages from a node with the averaged outgoing message and propagates messages from a low resolution graph to the original graph hierarchically. The proposed method reduces the computational time by half or two-thirds and reduces the required amount of memory by 60% compared with the standard belief propagation algorithm when applied to an image. The proposed method was implemented on CPU and GPU, and was evaluated against Middlebury stereo benchmark dataset in comparison with the standard belief propagation algorithm. It is shown that the proposed method outperforms the other in terms of both the computational time and the required amount of memory with minor loss of accuracy.

Repeated patterns are useful clues to learn previously unknown events in an unsupervised way. This paper presents a novel method that detects relatively long variable-length unknown repeated patterns in a motion sequence efficiently. The major contribution of the paper is two-fold: (1) Partly Locality Sensitive Hashing (PLSH) [1] is employed to find repeated patterns efficiently and (2) the problem of finding consecutive time frames that have a large number of repeated patterns is formulated as a combinatorial optimization problem which is solved via Dynamic Programming (DP) in polynomial time O(N1+1/α) thanks to PLSH where N is the total amount of data. The proposed method was evaluated by detecting repeated interactions between objects in everyday manipulation tasks and outperformed previous methods in terms of accuracy or computational time.

In this paper, we propose a method that detects repeated motion patterns in a long motion sequence efficiently. Repeated motion patterns are the structured information that can be obtained without knowledge of the context of motions. They can be used as a seed to find causal relationships between motions or to obtain contextual information of human activity, which is useful for intelligent systems that support human activity in everyday environment. The major contribution of the proposed method is two-fold: (1) motion density is proposed as a repeatability measure and (2) the problem of finding consecutive time frames with large motion density is formulated as a combinatorial optimization problem which is solved via Dynamic Programming (DP) in polynomial time O(N log N) where N is the total amount of data. The proposed method was evaluated by detecting repeated interactions between objects in everyday manipulation tasks and outperformed the previous method in terms of both detectability and computational time.

This paper presents a supervised foreground segmentation method that uses local and global feature similarity with edge constraint. This framework integrates and extends the notion of region growing and classification to deal with local and global fitness. It parameterizes constraint of growing using Chebyshev's inequality. The constraint is used to stop segmentation before matting. Matting relies on both local and global information. The proposed method outperforms many of the current methods in the sense of correctness and minimal user interaction, and it does so in a reasonable computation time.

In this paper, we propose a method for estimating meaningful actions from long-term observation of everyday manipulation tasks without prior knowledge as part of an action understanding framework for life support robotic systems. The target task is defined as a sequence of interactions between objects. An interaction that appears many times is assumed to be meaningful and repetitious relative motion patterns are detected from trajectories of multiple objects. The main contribution is that the problem is formulated as a combinatorial optimization problem with two parameters, target object labels and correspondences on similar motion patterns, and is solved using local and global Dynamic Programming (DP) in polynomial time O(N logN), where N is a total amount of data. The proposed method is evaluated against manipulation tasks using everyday objects such as a cup and a tea-pot.

This paper presents a novel whole body motion estimation method by fitting a deformable articulated model of the human body into the 3D reconstructed volume obtained from multiple video streams. The advantage of the proposed method is two fold: (1) combination of a robust estimator and ICP algorithm with Kd-tree search in pose and normal space make it possible to track complex and dynamic motion robustly against noise and interference between limb and torso, (2) the hierarchical estimation and backtrack re-estimation algorithm enable accurate estimation. The power to track challenging whole body motion in real environment is also presented.

This paper presents visual perception discovered in high-level manipulator planning for a robot to reproduce the procedure involved in human painting. First, we apply a technique of 2D object segmentation that considers region similarity as an objective function and edge as a constraint with artificial intelligent used as a criterion function. The system can segment images more effectively than most of existing methods, even if the foreground is very similar to the background. Second, we propose a novel color perception model that shows similarity to human perception. The method outperforms many existing color reduction algorithms. Third, we propose a novel global orientation map perception using a radial basis function. Finally, we use the derived model along with the brush's position- and force-sensing to produce a visual feedback drawing. Experiments show that our system can generate good paintings including portraits.

This paper presents visual perception discovered in high-level manipulator planning for a robot to reproduce the procedure involved in human painting. First, we propose a technique of 3D object segmentation that can work well even when the precision of the cameras is inadequate. Second, we apply a simple yet powerful fast color perception model that shows similarity to human perception. The method outperforms many existing interactive color perception algorithms. Third, we generate global orientation map perception using a radial basis function. Finally, we use the derived foreground, color segments, and orientation map to produce a visual feedback drawing. Our main contributions are 3D object segmentation and color perception schemes.

The learning from observation (LFO) paradigm has been widely applied in various types of robot systems. It helps reduce the work of the programmer. However, the applications of available systems are limited to manipulation of rigid objects. Manipulation of deformable objects is rarely considered, because it is difficult to design a method for representing states of deformable objects and operations against them. Furthermore, too many operations are possible on them. In this paper, we choose knot tying as a case study for manipulating deformable objects, because the knot theory is available and the types of operations possible in knot tying are limited. We propose a knot planning from observation (KPO) paradigm, a KPO theory, and a KPO system.

We propose an efficient and effective image generation system for an experimental mixed-reality traffic space. Our enhanced traffic/driving simulation system represents the view through a hybrid that combines virtual geometry with real images to realize high photo-reality with little human cost. Images for datasets are captured from the real world, and the view for the simulation system is created by synthesizing image datasets - with a conventional driving simulator.

The Programming by Demonstration (PbD) technique aims at teaching a robot to accomplish a task by learning from a human demonstration. In a manipulation context, recognizing the demonstrator's hand gestures, specifically when and how objects are grasped, plays a significant role. Here, a system is presented that uses both hand shape and contact-point information obtained from a data glove and tactile sensors to recognize continuous human-grasp sequences. The sensor fusion, grasp classification, and task segmentation are made by a hidden Markov model recognizer. Twelve different grasp types from a general, task-independent taxonomy are recognized. An accuracy of up to 95% could be achieved for a multiple-user system.

Scanning from the air is one of the most efficient methods for obtaining 3D data of large-scale objects. For this purpose, we have been developing a flying laser range finder that is suspended under a balloon. Even though the scanning speed of the finder is quite rapid, it is difficult to eliminate the influence of the swing of a balloon. As a result the scanned data have some distortion due to the intra-scanning movement. In order to compensate this intra-scanning movement, we propose a evolutional registration algorithm which not only aligns multiple range images to determine inter-scanning movement parameters, but also rectifies distortion of range image by determining intra-scanning movement parameters. In this paper, we describe our aerial scanning system especially focusing on the design of the flying laser range finder and deformation registration algorithm. To show the effectiveness of our method, we evaluate its performance using synthesized and real data.

This paper describes a method to realize flexible cooperation between human and robot which reflects the intention and state of human by using gaze information. This physiological information expresses the process of thinking directly, so it enables us to read the internal condition such as hesitation or search in decision making process. We propose a method to interpret the intention and condition from the latest history of gaze movement and determine an appropriate cooperative action of a robot based on it so that the task proceeds smoothly. Finally, we show experimental results by using a humanoid-type robot.

Learning from Observation (LFO) has been widely applied in various types of robot system. It helps reduce the work of the programmer. But the available systems have application limited to rigid objects. Deformable objects are not considered because: 1) it is difficult to describe their state, and 2) too many operations are possible on them. In this paper, we choose the knot tying as case study for operating on nonrigid bodies, because a "knot theory" is available and the type of operations is limited. We describe the Knot Planning from Observation (KPO) paradigm, KPO theory and KPO system.

To learn a new everyday task under the "Learning from Observation" framework, the system needs to detect which parts of the demonstration are essential to complete the task without task-dependent knowledge. In the previous research, we proposed a technique to estimate essential interactions in a task by integrating multiple demonstrations which represent virtually the same task. Although, the technique could automatically segment the essential interactions and determine the number of the interactions, the segmentation algorithm depends on some heuristics and only stationary interactions could be obtained. In this paper, a novel technique is proposed, which overcomes this limitation and can estimate almost any types of interactions. In this approach, a demonstrator needs to give a explicit signal once during each essential interaction as a hint on the occurrence of the essential interaction. From visual information and these signals, the system automatically analyzes the essential parts of the task and their periods, and also detects which environmental objects are interacted with the manipulated object. These information is hard to be obtained from a single demonstration, because of the ambiguity in interpreting the interaction especially in cluttered environment. The proposed method is evaluated in a simulation and also in a real world by using a humanoid robot.

Various methods to recognize assembly tasks using possible local displacement of objects have been proposed. To calculate this displacement, the screw theory is employed. It is equivalent to the first order Taylor expansion of the displacement. However, such methods can treat polyhedral objects only. Because the screw theory cannot treat curvature information of objects. In this paper, we propose a method to calculate possible local displacement of curve objects using improved screw theory, which is equivalent to the second order Taylor expansion of the displacement, and verify the validity of the proposed method.

A technique to recognize the shape of a grasping hand during manipulation tasks is proposed; which utilizes 3D articulated hand model and a reconstructed 3D volume from infrared cameras. Vision-based recognition of a grasping hand is a tough problem, because a hand may be partially occluded by a grasped object and the ratio of occlusion changes along the progress of the task. To recognize the shape in a single time frame, a robust recognition method of an articulated object is proposed. In this method, 3D volumetric representation of a hand is reconstructed from multiple silhouette images and 3D articulated object model is fitted to be reconstructed data to estimate the pose and the joint angles. To deal with large occlusion, a technique to simultaneously estimate time series reconstructed volumes with the above method is proposed, which can automatically suppress the effect form badly reconstructed volumes. The proposed techniques are verified in simulation as well as in a real world.

This paper describes a new approach on how to teach a robot everyday manipulation tasks under the "learning from observation" framework. Most of the approaches so far assume that a demonstration can be well understood from a single demonstration. However, a single demonstration contains ambiguity, in that interactions which are essential to complete a task cannot be discerned without prior task dependent knowledge, which should be obtained from observation. To address these issues, we propose a technique to integrate multiple observations of demonstrations. The demonstrations differ, but are virtually the same task. The shared interactions among all the demonstrations are considered to be essential and we form a task model from their symbolic representations. Then the relative trajectories corresponding to each essential interaction are generalized by calculating their mean and variance and are also stored in the task model, which is used to reproduce a skilled behavior. We examine this approach by using a human-form robot, which successfully imitates human demonstrations of everyday tasks.

This paper describes a new approach on how to teach everyday manipulation tasks to a robot under the "Learning from Observation" framework. In our previous work, to acquire low-level action primitives of a task automatically, we proposed a technique to estimate essential interactions to complete a task by integrating multiple observations of similar demonstrations. But after many demonstrations are performed, there may be interactions which are the same in nature. These identical interactions should be grouped so that each action primitive becomes unique. For this purpose, a Hidden Markov Model based clustering algorithm is presented which automatically determines the number of independent interactions. We also show that the obtained interactions can be used as discriminators of human behavior. Finally, simulation and experimental results in which a real humanoid robot learns and recognizes essential actions by observing demonstrations are presented.

The completion of robot programs requires long development time and much effort. To shorten the programming time and to minimize the effort, we have been developing a system which we refer to as the "assembly-plan-from-observation (APO) system." This system requires assembly task recognition from observing human performance. Observation data by a robot's vision system is usually error contaminated, and, thus, we cannot use those data directly. This paper proposes two methods to clean up those errors by using contact relations and their transitions. The first one corrects the observed configuration from contact relations observed. The second one identifies wrongly determined contact relations from an analysis of configuration space (C-space). We have implemented both methods on our test bed and have verified their effectiveness.

This paper describes our current research on learning task level representations by a robot through observation of human demonstrations. We focus on human hand actions and represent such hand actions in symbolic task models. We propose a framework of such models by efficiently integrating multiple observations based on attention points; we then evaluate the model by using a human-form robot. We propose a two-step observation mechanism. At the first step, the system roughly observes the entire sequence of the human demonstration, builds a rough task model and extracts attention points (APs). The attention points indicate the time and position in the observation sequence that requires further detailed analysis. At the second step, the system closely examines the sequence around the APs and the obtained attribute values for the task model, such as what to grasp, which hand to be used, or what is the precise trajectory of the manipulated object. We implemented this system on a human form robot and demonstrated its effectiveness.

Describes research on learning task level representations by a robot through observation of human demonstrations. We focus on human hand actions and develop a construction method of a human task model which integrates multiple observations to solve ambiguity based on attention points (AP). So far, this analysis constructs a symbolic task model efficiently in a coarse-to-fine way through two steps. However, to represent delicate motion appearing in a task, the system must incorporate the information about precise motion of the manipulated objects into an abstract task model. We propose a method to identify the manipulated object through repeated observations of both human and robot behavior. To this end, we present a method which combines 2D and 3D template matching techniques to localize an object in 3D space generated from a depth and an intensity image. We apply this technique to recognition of human and robot behavior by obtaining the precise trajectory of the manipulated objects. We also present the experimental results achieved through the use of a human-form robot equipped with a 9-eye stereo vision system.

The completion of robot programs requires long development time and much effort. To shorten this programming time and minimize the effort, we have been developing a system which we refer to as "assembly-plan-from-observation (APO) system;" this system provides the ability for a robot to observe a human performing an assembly tasks, understand the tasks, and subsequently generate a program to perform that same task. One of the necessary tasks in APO is to create a trajectory of robot hand movement from observing human performance. The previous system developed a direct observation method based on the trajectory of a human movement. Though simple and handy, the system was susceptible to noise. This paper proposes a method to make the observation robust against noise by using symbolic representations of a trajectory based on contact analysis. The system divides the trajectory into small segments based on the contact analysis, then allocates an operation element referred to as a sub-skill to those segments; the result is a robust trajectory-based APO system.

This paper presents a novel method of constructing a human task model by attention point (AP) analysis. The AP analysis consists of two steps: at the first step, it broadly observes human task, constructs rough human task model and finds APs which require detailed analysis; and at the second step, by applying time-consuming analysis on APs in the same human task, it can enhance the human task model. This human task model is highly abstracted and is able to change the degree of abstraction adapting to the environment so as to be applicable in a different environment. We describe this method and its implementation using data gloves and a stereo vision system. We also show an experimental result in which a real robot observed a human task and performed the same human task successfully in a different environment using this model.

The completion of robot programs requires a long development time and much effort. To shorten this programming time and to minimize the effort, we have been developing a system which we refer to as the "Assembly Plan from Observation (APO) system". This system provides the ability for a robot to observe a human performing an assembly task, to recognize the task and to then generate a program to perform that same task. One of the necessary tasks in APO is to create the trajectory of a robot hand movement after having observed a human's performance. The previous system developed a direct observation method based on the trajectory of a human's movement. Although it was simple and handy, the system was susceptible to noise. This paper proposes a method to make the observation robust against noise by analyzing topological contact relations. The system divides the trajectory into small segments based on this contact analysis, and then allocates an operation element, referred to as a "sub-skill", to those segments. The result is a robust, trajectory-based APO system.