A method for detecting the position and orientation of freely curved pipes is proposed for the purpose of bin picking. We assume that the geometrical model of the object of interest is given and the bin of objects can be observed using a widely available low-cost 3D sensor. As freely curved pipes do not have distinguishable feature points, typical feature-based matching methods cannot be applied directly in this case. Although the iterative closest point (ICP) algorithm [1] is a suitable strategy for registering a 3D model with free-form surfaces to its observed 3D data, two difficulties arise when applying the ICP algorithm in the case where the background is filled with objects with the same shape. One of these is the initial estimation of the position and orientation of an object from among similar shaped point clouds. The other difficulty is robust point matching between the 3D model and the observed data belonging to a single object. To solve these issues, we investigated two key ideas: 1) estimation of pipe orientation by checking the shape of the cross-section of the observed 3D data while changing cutting directions of the sections; and 2) point matching based on ridge lines with respect to the camera direction. Experiments using actual observed data show the effectiveness of the proposed method.

This paper proposes a method for bin-picking for objects without assuming the precise geometrical model of objects. We consider the case where the shape of objects are not uniform but are similarly approximated by cylinders. By using the point cloud of a single object, we extract the probabilistic properties with respect to the difference between an object and a cylinder and consider applying the probabilistic properties to the pick-and-place motion planner of an object stacked on a table. By using the probabilistic properties, we can also realize the contact state where a finger maintain contact with the target object while avoiding contact with other objects. We further consider approximating the region occupied by fingers by a rectangular parallelepiped. The pick-and-place motion is planned by using a set of regions in combination with the probabilistic properties. Finally, the effectiveness of the proposed method is confirmed by some numerical examples and experimental result.

Dexterous manipulation is an important function for working robots. Manipulator tasks such as grasping, assembly and disassembly can generally be divided into several motion primitives. We call such motion primitives “skills” and explain how most manipulator tasks can be composed of sequences of these skills. We will address the issues involved with various types of robots such as maintenance robots and service robots. We have considered hierarchizing the manipulation tasks of these robots since their tasks have become more complex than ever before. Additionally, as errors are seen likely to increase in complex tasks, it is important to implement effective error recovery technology. This paper presents our proposal for a new type of error recovery that uses the concepts of task stratification and error classification which can be expressed specifically using flow charts.

This paper proposes a method for planning the pick-and-place motion of an object by dual-arm manipulators. Our planner is composed of the offline and the online phases. The offline phase generates a set of regions on the object and the environment surfaces and calculates several parameters needed in the online phase. In the online phase, the planner selects a grasping pose of the robot and a putting posture of the object by searching for the regions calculated in the offline phase. By using the proposed method, we can also plan the trajectory of the robot, and the regrasping strategy of the dual-arm. Here, the putting posture of the object can be planned by considering stability of the object placed on the environment. The effectiveness of the proposed method is confirmed by simulation and experimental results by using the dual-arm robot NX-HIRO.

This paper proposes an object placement planner for a grasped object during pick-and-place tasks. The proposed planner automatically determines the pose of an object stably placed near a user assigned point on an environment surface. The proposed method first constructs a polygon model of the surrounding environment, and then clusters the polygon model of both the environment and the object where each cluster is approximated by a planar region. The placement of the object can be determined by selecting a pair of clusters between the object and the environment. We further impose several conditions to determine the pose of the object placed on the environment. We show that we can determine the position/orientation of the object placed on the environment for several cases such as hanging a mug cup on a bar. The effectiveness of the proposed research is confirmed through several numerical examples.

Snap assembly automation remains a challenging task. While progress is being made in localization of parts, force controllers, and control strategies, little work has been done to help the robot reason about its current state, such that if necessary, the robot can assume corrective actions to accomplish the task. Error prone situations caused by the unexpected motion of parts, localization errors, jamming or wedging, cannot be solved by force controllers alone. For this reason we propose a snap assemblies verification system for cantilever-snap fasteners. The verification works in concert with a control strategy that makes use of constraint designs embedded in the snap parts' physical design. The constrained assembly motion generates similar sensory-signal patterns across trials that facilitates force signal discrimination into higher level abstractions of intuitive behavior. This work's contribution is the design of a hierarchical taxonomy for cantilever-snap verification based on increasingly abstract layers that encode relative-change in the task's force signatures. A five-layered taxonomy is built on the concept that relative-change patterns can be classified through a small category set and aided by contextual information. The verification system yielded human apropos intuitive categorizations of task behavior for every state and effectively determined the assembly result. This simple yet effective approach will be expanded to perform probabilistic online system verification to aid in fault tolerance and the automation of cantilever-based snap assemblies.

The present paper describes a method for automatic 3D contour model creation for stereo-vision systems. The object model is a triangular surface mesh and a set of aspect models, which consists of model features and model points. Model features and model points are generated using 3D contours, which are estimated by the projected images of the triangular surface mesh from multiple discrete viewing directions. Using a non-photorealistic rendering approach, we extract not only the outer contours but also the inner contours of the projected images. Using both the inner and outer contours of the projected images, we create the object model which has 3D inner contour features and 3D contour generator features. Experimental results obtained using the 3D localization algorithm demonstrate the effectiveness of the proposed model.

This paper presents a grasping system for picking up an indicated object in a complex real-world environment using a parallel jaw gripper. The proposed grasping scheme comprises the following three main steps: (1) A user indicates a target object and provides the system with a task instruction on how to grasp it, (2) the system acquires geometric information about the target object and constructs a 3D environment model around the target by stereo vision using the information obtained from the task instruction, and (3) the system finds a grasp point based on grasp evaluation using the acquired information. As an example of the scheme, we examined the picking up of a cylindrical object by grasping at the brim. An important and advantageous feature of this scheme is that the user can easily instruct the robot on how to perform the object-picking task through simple clicking operations, and the robot can execute the task without exact models of the target object and the environment being available in advance.

This paper describes a method to localize 3D objects, which is the extension of the segment-based object recognition method to use on a STL CAD model. Models for localization are automatically generated using contour generators, which are estimated by occluding contours of projected images of the CAD model from multiple viewing directions and depth images computed with a graphics accelerator. In addition, the model is dynamically updated in a recognition process according to the viewing direction. Localization process is based onmultiple hypotheses verification using the model and 3D boundaries reconstructed from stereo images. Experimental results show the effectiveness of the proposed method.

This paper describes a method to localize objects which are partially buried in an unstructured environment. The objects are limited to surfaces of revolution in the present study. Candidate regions are extracted as convex surface regions from a range data that is obtained by area-based stereo matching. The surface type at each point is determined based on the signs of the maximum and minimum curvatures, which are calculated by fitting a quadric in the local window. For each candidate region, multiple hypotheses for the position and direction of the rotational axis are generated for each object model. Each hypothesis is verified and improved by an iterative method, and the most reliable hypothesis is adopted. The experimental results demonstrate the effectiveness of the proposed method for actual range data.

As a part of research on autonomous loading system by wheel loader, a method for appropriate arrangement of bucket trajectory for scooping motion will be proposed. In this paper, relation between resistance force and advancing direction of the bucket is analyzed theoretically. Advancing direction is dominant factor for resistance force on the bucket during scooping. Based on this analysis, simple rules for bucket trajectory arrangement are proposed. In scooping procedure, scooped volume is estimated using 3D model obtained with stereo-vision system. The shape of the pile is measured and converted into pile model prior to scooping. The developed method and system are installed on experimental scale model of wheel loader. The results show good performance for different condition of pile.

For cooperative work of robots and humans in the real world, a communicative function based on speech is indispensable for robots. To realize such a function in a noisy real environment, it is essential that robots be able to extract target speech spoken by humans from a mixture of sounds by their own resources. We have developed a method of detecting and extracting speech events based on the fusion of audio and video information. In this method, audio information (sound localization using a microphone array) and video information (human tracking using a camera) are fused by a Bayesian network to enable the detection of speech events. The information of detected speech events is then utilized in sound separation using adaptive beam forming. In this paper, some basic investigations for applying the above system to the humanoid robot HRP-2 are reported. Input devices, namely a microphone array and a camera, were mounted on the head of HRP-2, and acoustic characteristics for sound localization/separation performance were investigated. Also, the human tracking system was improved so that it can be used in a dynamic situation. Finally, overall performance of the system was tested via off-line experiments.

We have developed a humanoid robot HRP-2P with a biped locomotion controller, stereo vision software and aural human interface to realize cooperative works by a human and a humanoid robot. The robot can find a target object by the vision, and carry it cooperatively with a human by biped locomotion according to the voice commands by the human. A cooperative control is applied to the arms of the robot while it carries the object, and the walking direction of the robot is controlled by the interactive force and torque through the force/torque sensor on the wrists. The experimental results are presented in the paper.

We propose a new method to recognize 3D free-form objects from their apparent contours. It is the extension of our established method to recognize objects with fixed edges. Object models are compared with 3D boundaries which are extracted by segment-based stereo vision. Based on the local shapes of the boundaries, candidate transformations are generated. The candidates are verified and adjusted based on the whole shapes of the boundaries. The models are built from all-around range data of the objects. Experimental results show the effectiveness of the method.

Presents a method to search for stereo correspondence based on the connectivity of segments. The data structure is defined by boundary representation. The free curve can be treated by this data structure not as a straight line approximation, but as a curved line. Some candidates for correspondence pairs consist of two segments in left and right images under epipolar condition. The connectivity of two pairs is decided by their distance, intensity, and angle, based on each boundary in the left image. The similarity of paths, which are sequences of pairs, is evaluated. The main feature of our method is that it can evaluate in general situations because the similarity is calculated based on the connectivity of pairs. Multiple correspondence between pairs is removed using this similarity, and the 3D structure is reconstructed. Applications of our work include recognition for free form objects and tracking in 3D space.

We are developing an advanced 3D vision system VVV (Versatile Volumetric Vision) which can be used for many purposes in many fields; robotics, medical and welfare, CAD, VR, etc. The basic processes of 3D vision are range sensing of a 3D scene, describing 3D shape of objects in the scene, and recognizing the 3D position and orientation of objects by matching with object models. Also, moving objects can be tracked by repeating these processes. For the system to be versatile, it consistently executes these processes and makes no assumption on objects; objects may be any size, any shape (figures, polyhedra to free forms), covered with textured or glossy surfaces, at any position, in any direction, together with other objects, and partially occluded by them. The basic technical approach is structural analysis and partial pattern matching. And, the basic policy of development is to separate "functions" which are common to all tasks and "knowledges" which are specific to each task.

A new approach to integrate range data from multiple viewpoints is described. Complicated objects, for example flowers, are observed by a range finder from multiple viewpoints whose relative position is unknown. After simple segmentation, correspondences of regions between two consecutive data are examined. This matching process relies on regions which are not supposed to be influenced by occlusion. Transformation parameters between two data are calculated by referring to the matching result. An iteration process to minimize difference estimates of corresponding regions is devised so that a more accurate result is obtained. Data observed from multiple viewpoints are transferred to standard coordinates and integrated. Results on some objects, for example flowers, show that so far that this method is promising.>