In order to insert a fork into a hole of a pallet by a forklift located in front of a pallet, it is necessary to control the height position, reach position, and tilt angle of the fork to match the position and orientation of the hole of the pallet. In order to make AGF (Autonomous Guided Forklift) do this automatically, we propose an image measurement method to measure the pitch inclination of the pallet in the camera coordinate system from an image obtained by using a wide-angle camera. In addition, we propose an image measurement method to easily acquire the calibration information between the camera coordinate system and the fork coordinate system necessary to apply the measurements in the camera coordinate system to the fork control. In the experiment space, a wide-angle camera was fixed at the backrest of a reach type forklift. The wide-angle images taken by placing a pallet in front of the camera were processed. As a result of evaluating the error by comparing the image measurement value with the hand measurement value when changing the pitch inclination angle of the pallet, the relative height of the pallet and the fork, and whether the pallet is loaded or not, it was confirmed that the error was within the allowable range for safely inserting the fork.

We propose a method of virtually flattening the surface of a clothing item to a two-dimensional plane using the geodesic distance over the surface of the item. If a clothing item is flatly opened on a table, the recognition of the item is much easier than that of the same item having an arbitrary shape. However, it is difficult and troublesome to physically flatten a clothing item from its arbitrary shape automatically. We therefore propose to develop the surface of the clothing item held in the air into a two-dimensional shape using three-dimensional observation data of the surface. To this end, boundary points of the observed clothing region are sampled as start/end nodes for calculating geodesic lines, the lengths of which become two-dimensional distances between the points when the surface is flattened to a plane. To robustly calculate geodesic lines using a three-dimensional point cloud of the surface, we adopt a method that interpolates the depth and normal direction at any point on the surface using the element-free Galerkin method. The shortest path along the surface between two points on the surface is then calculated using these depths and normal directions in the framework of the “zero-length spring analogy”. The two-dimensional coordinates of the points on the plane are obtained by solving simultaneous equations determined by the geodesic distances. We also propose a method of using the flattened view for the classification of clothing type and detection of important parts of clothing. Preliminary experiments using long-sleeve shirts and trousers as clothing items demonstrate the promise of the proposed methods.

In this paper, we propose a method of visual recognition in cooperation with actions for automatic handling of clothing by a robot. Difficulty in visual recognition of clothing largely depends on the observed shape of the clothing. Therefore, strategy of actively making clothing into the shape easier to recognize should be effective. First, after clothing is observed by a trinocular stereo vision system, it is checked whether the observation gives enough information to recognize the clothing shape robustly or not. If not, proper “recognition-aid” actions, such as rotating and/or spreading the clothing, are automatically planned based on the visual analysis of the current shape. After executing the planned action, the clothing is observed again to recognize. The effect of the action of spreading clothes was demonstrated through experimental results using an actual humanoid.

In this paper, we propose a deformable-model-driven method to recognize the state of hanging clothes using three-dimensional (3D) observed data. For the task to pick up a specific part of the clothes, it is indispensable to obtain the 3D position and posture of the part. In order to robustly obtain such information from 3D observed data of the clothes, we take a deformable-model-driven approach[4], that recognizes the clothes state by comparing the observed data with candidate shapes which are predicted in advance. To carry out this approach despite large shape variation of the clothes, we propose a two-staged method. First, small number of representative 3D shapes are calculated through physical simulations of hanging the clothes. Then, after observing clothes, each representative shape is deformed so as to fit the observed 3D data better. The consistency between the adjusted shapes and the observed data is checked to select the correct state. Experimental results using actual observations have shown the good prospect of the proposed method.

In this paper, we propose a strategy for a dual-arm robot to pick up a specific part of clothing with one hand while holding the item of clothing with its other hand. Due to the large deformability of clothing, the handling requirements differ from those required for the handling of rigid objects. In the case of holding a specific part of clothing, large deformation leads to a large variety of positions and orientations of the target part, requiring flexibility of both visual recognition and motion control. On the other hand, since the clothing can flexibly curve over the hand, a relatively large range of suitable actions is allowed for grasping the clothing. By considering these characteristics, the following three-stage strategy is proposed. First, the state of the clothing is recognized from visual observation of the clothing using a deformable-model [1]. Then, the theoretically optimal position and orientation of the hand for handling a specific part of the clothing is calculated based on the recognition results. Finally, the position and orientation of the hand is modified by considering the executable motion range of the dual arms. Preliminary experimental results using actual observations of a humanoid robot were used to validate the effectiveness of the proposed strategy.

In this paper, we propose a deformable model-driven method to obtain the 3D information necessary for handling clothes by manipulators from observation with stereo cameras. The task considered in this paper is to hold up a target part of clothes (e.g. one shoulder of a pullover) by the second manipulator, when the clothes are held in the air at any point by the first manipulator. First, the method calculates possible 3D shapes of the hanging clothes by simulating the clothes deformation. The 3D shape whose appearance gives the best fit with the observed images is selected as estimation of the current state. Then, based on the estimated shape, the 3D position and normal direction of the part where the second manipulator should hold are calculated. The results of preliminary experiments using actual two manipulators have shown the good potential of the proposed method.

Aiming at manipulating clothes, a model-driven method of estimating the state of hanging clothes is proposed. We suppose a system consisting of two manipulators and a camera. The task considered in this paper is to hold a pullover at its two shoulders by two manipulators respectively, as a first step for folding it. The proposed method estimates the state of the clothes held by one manipulator in a model-driven way and indicates the position to be held next by the other manipulator. First, the possible appearances of the pullover when it is held at one point are roughly predicted. Using discriminative features of the predicted appearances, the possible states for the observed appearance are selected. Each appearance of the possible state is partially deformed so as to get close to the observed appearance. The state whose appearance successfully approaches closest to the observed appearance is selected as the final decision. The point to be held next is determined according to the state. The results of preliminary experiments using actual images have shown the good potential of the proposed method.

Work in simultaneous localisation and map-building ("SLAM") for mobile robots has focused on the simplified case in which a robot is considered to move in two dimensions on a ground plane. While this is often a good approximation, a large number of real-world applications require robots to move around terrain which has significant slopes and undulations, and it is desirable that these robots too should be able to estimate their locations by building maps of natural features. We describe a real-time EKF-based-SLAM system permitting unconstrained 3D localisation, and in particular develop models for the motion of a wheeled robot in the presence of unknown slope variations. In a fully automatic implementation, our robot observes visual point features using fixating stereo vision and builds a sparse map on-the-fly. Combining this visual measurement with information from odometry and a roll/pitch accelerometer sensor, the robot performs accurate, repeatable localisation while traversing an undulating course.

Industrial inspection robots, which are to move around in a nuclear power plant and carry out inspection tasks, must follow a given route while observing static and dynamic features of the environment. A stereo active vision system has been developed, equipped with foveated wide-field image acquisition and mounted on a mobile robot. With this or other types of active sensors, only the foveated center of the field of view can extract precise information, so the camera pose must be controlled to fixate the target area for execution of a particular visual task. The activities of inspection robots lead to multiple visual tasks, requiring both reactive and premeditated types of attention, and therefore the robot must apply its sensing resources sequentially among multiple possibilities. In this paper, we propose a new framework for visual attention control for a mobile robot, describe the implementation of the proposed framework, and show some examples in which the camera pose is controlled to achieve multiple visual tasks.

A method for quick determination of the position and pose of a 3D free-form object with respect to its 2D projective image(s) is proposed. It is a precondition of the method that a 3D model of the object and an initial estimation of the state are given. We (1998) have previously proposed a 3D-2D registration method for the real-time registration of a 3D model of a cerebral vessel tree to an X-ray image of the vessel. Here we extend this method to meet a more general purpose. First, the formulae for obtaining the 3D model transformation from 3D-2D point pairs are generalized by describing camera's coordinates independently from world coordinates. The second generalization is to use the occluding contour instead of the skeleton of the blood vessel as the feature for matching. We use a 3D graphics system like OpenGL into our 3D-2D registration method. The applicability improvements are shown using two applications: a position and pose estimation of a 3D vessel model using multiple views, and visual feedback on the position and pose of an active camera head.

In the routine inspection of industrial or other areas, teams of robots with various sensors could operate together to great effect, but require reliable, accurate and flexible localisation capabilities to be able to move around safely. We demonstrate accurate localisation for an inspection team consisting of a robot with stereo active vision and its companion with an active lighting system, and show that in this case a single sensor can be used for measuring the position of known or unknown scene features, measuring the relative location of the two robots, and actually carrying out an inspection task.

A method to determine the position and pose of an active camera-head by aligning a 3D model of its surrounding environment with an observed 2D image is proposed. The camera-head is mounted on a mobile robot and freely moves in a 3D space. We aim at visual feedback to correct the estimation error of its position and pose obtained from dead reckoning. Since the nuclear power plant where the robot moves about consists of many pipes without particular marks, most of features in the observed images are occluding edges of the pipes. For robustly finding 3D-2D point correspondences on the occluding edges, two-type predicted images which are calculated from the 3D environmental model by a view simulator are used as follows: 1) 3D model points which correspond to the observed occluding edges are quickly obtained from the predicted depth image; 2) The predicted intensity image is used to select only the 3D model points which are expected to appear clearly in the observed image. As a result, point correspondences between the observed image and the 3D model can be robustly found even in complicated scenes. Preliminary experiments using an actual plant mock-up have shown that the method is promising.

We present a novel design and implementation of a foveated wide angle lens. It is strongly motivated towards applications for active vision in complex and dynamic environments. The projection curve is designed so that it facilitates active vision algorithms for motion analysis, object identification, and precise fixation. The implementation achieves small size and light weight so that the lens can be swung at a high speed.

A hand/eye system for handling flexible materials is under development. Our concern is to increase the effectiveness of a cooperative sensing of touch and vision for handling flexible and limp materials. We consider that a cooperative sensing of touch and vision is more important when flexible or limp objects such as fabrics are handled. This paper presents a strategy of sensor-based manipulation for unfolding a fabric piece in a case of primitive fabric handling movements. Vision and tactile sensing are used for picking up the folded part.

A novel framework for multiple robot cooperation called "cooperation by observation" is presented. It introduces many interesting issues such as a viewpoint constraint and role interchange, as well as novel concepts like "attentional structure". The framework has the potential to realize a high level of task coordination by decentralized autonomous robots allowing minimum explicit communication. Its source of power lies in an advanced capability given to each robot for recognizing other agent's actions by (primarily visual) observation. This provides rich information about the current task situation around each robot which facilitates highly-structured task coordination. The basic visuo-motor routines are described. Concrete examples and experiments using real mobile robots are also presented.>

This paper presents some advanced examples of reactive vision-based cooperative behaviors: 1) chasing and posing against another robot among others; 2) unblocking the path of another robot by removing an obstacle; and 3) passing an object from one to another. These behaviors are demonstrated using real mobile robots equipped with CCD cameras, in a complex environment, and with no central controller or explicit communication among the robots. The action observation is based on real time processing of optical flow analysis and stereo tracking. An extended behavior-based architecture for "cooperation by observation" is presented. The core extension consists of a mobile space buffer and an image space buffer with manipulable markers which control the internal flow of information, thereby coordinating parallel behaviors and achieving purposive tasks in complex and dynamic environments.>

This paper presents a stereo active vision system which performs tracking tasks on smoothly moving objects in complex backgrounds. Dynamic control of the vergence angle adapts the horopter geometry to the target position and allows to pick it up easily on the basis of stereoscopic disparity features. We introduce a novel vergence control strategy based on the computation of "virtual horopters" to track a target movement generating rapid changes of disparity. The control strategy is implemented on a binocular head, whose right and left pan angles are controlled independently. Experimental results of gaze holding on a smoothly moving target translating and rotating in a complex surrounding demonstrate the efficiency of the tracking system.>

Proposes a method which makes a personal recognition dictionary from pseudo-characters that are automatically generated by using a computer model of human handwriting. The method consists of three stages. First a handwriting model mimicking an individual person is derived from his real handwritten characters. Next, pseudo-handwritten characters are generated by using the model, and finally a personal recognition dictionary is built from those generated characters. The proposed method requires much fewer of the user's real handwritten characters than conventional methods. This results in reducing the users' burden. The experimental results demonstrate that pseudo handwritten characters have a high enough quality to make useful personal dictionaries for handwritten character recognition.>

The authors propose a simple method for locating a 3D object from a single 2D image. Aiming at rough and speedy location of a target object, the method simply checks shape similarity between observed regions and possible appearances of the target. To reduce the number of the appearances necessary for representing an object, the authors develop a 2D shape description, called concentric circular description, which is insensitive to viewpoint changes while keeping a sensitivity to 3D shape difference. The method is applied to some real images of a cluttered desk scene.>