In this paper, we propose a wide-range and high-accuracy fisheye stereo camera using the fisheye vertical stereo method. In stereo measurement with two fisheye cameras placed horizontally, increased mismatching occurs due to template matching along curved epipolar lines. Additionally, because the baseline is horizontal, there is a decrease in the distance measurement accuracy in the left and right areas of the image. Therefore, by placing fisheye cameras vertically and straightening the epipolar lines, we reduce mismatching during template matching in stereo measurement, achieving high-accuracy stereo measurement. Moreover, by making the baseline vertical, we improve the distance measurement accuracy in the left and right areas. Experiments demonstrate that the distance measurement accuracy of our proposed method is higher than that of conventional methods.

This paper presents a novel method to control an underactuated hand by using only a monocular camera, not using any internal sensors. In food factories, robots are required to handle a wide variety of foods without damaging them. To accomplish this, the use of underactuated hands is effective because they can adapt to various food shapes. However, if internal sensors such as tactile sensors and force sensors are used in the underactuated hands, it may cause a problem with hygiene and require complicated calibration. Moreover, if external sensors such as cameras are used, it is necessary to grasp foods without damaging them by using external information such as images. In our method, to tackle these problems, a camera is used as an external sensor. First, contact between the hand and the object is detected by using the contours of both, obtained from a camera image. Then, to avoid damaging the object, the following information is extracted from camera images and observed: the centroid of both the hand and object, the deformation of the object, and the occlusion rate of the hand. Furthermore, to prevent the object from dropping while the robotic arm is in motion, the distance between the centroid of the hand and the object is calculated. The experiments were conducted using twelve different food items.

In this paper, we improve the robustness of a multi-camera gesture recognition system in multi-person situations by identifying and tracking the operator. This system is meant as an intelligent room to operate and interact with surrounding devices based on pointing gestures. In the method, we identify the operator by a hand-raising gesture, followed by tracking using the coordinates of the center of the operator’s head and extracting only the operator’s whole body. From the experimental results, we confirmed that highly accurate tracking could be performed in a multi-person situation of 2 to 5 persons, and that the success rate of extracting images of the operator’s whole body was more than 70%. We also clarified issues in the operator identification process and the extraction process.

In-hand manipulation (IHM) is an important ability for robotic hands. This ability refers to changing the position and orientation of a grasped object without dropping it from the hand workspace. One major challenge of IHM is to achieve a large range of manipulation (especially rotation), regardless of the shape, size, and the orientation during manipulation of the grasped object. There are two main challenges - the manipulation range (due to the range of motion of the hand) and keeping the object grasped under all shapes and orientations. Specifically, even when the contact points between the hand and the object switch and the positions of these points change due to its shape and changing orientation, constant grasp of the object is required. This paper presents an IHM method for a robotic hand with belts, based on the prediction of the contact-point changes via image information. The focus is on a robotic hand that has a two-fingered parallel gripper with conveyor belts which can continuously manipulate an object through a large range. A stereo camera is attached to the hand. First, the contour of the grasped object is acquired from the camera. From the contour, the switching of the contact points between the surfaces of the belts and the object is predicted. Then, the positions of the contact points in the next frame are estimated by rotating the contour. The velocities of the belts are calculated based on the prediction of the switching. The fingers are controlled to follow the estimated positions of the contact points, via a feed-forward control. The effectiveness of the proposed method is verified through in-hand manipulation experiments for 22 objects of various shapes and sizes.

The purpose of this paper is to construct and evaluate a system to operate home appliances by pointing. In Human Machine Interface (HMI) design, a natural operating method is important. Pointing is a universal gesture for selecting an object. Arm-pointing to an appliance and selecting it to perform a simple operation is a very intuitive and easy-to-use method of operation. Many studies prepare data with locations of appliances and their sizes. In this paper, we a camera-based system where the user can simply point at an appliance to select and operate it is proposed. The user’s pointing direction and appliance locations are estimated automatically from image frames. This eliminates the need for any preparation beforehand and the appliances can be moved during operation. The proposed method was implemented and experimentally evaluated. It was found that the average recognition rates were about 87% and 57% when a humidifier and a TV were operated.

This study presents a novel apprenticeship learning method to enable a learner to utilize demonstrations observed in an incompatible feature space. It is assumed that an expert and a learner follow non-identical Markov decision processes (MDPs), and a mapping function is estimated to obtain feature expectation of the demonstrations in an agent space. A conditional density estimation technique is used to represent the feature expectation in closed-form. The proposed method is useful because it is expected to alleviate intractable processes to explicitly specify correspondence of heterogeneous MDPs for apprenticeship learning. Additionally, the method does not require any sampling method to approximate integrals over an agent feature space. A simulation is used to demonstrate the validity of the proposed method in three domains in which it is not possible to directly compare the features of the expert and learner.

This paper deals with the background maintenance problem and proposes a novel pixel-wise solution. The proposed background maintenance algorithm is histogram-based. The algorithm has the following main features: fast background initialization, high accuracy in describing the real background and fast reaction to sudden changes. The basic idea of our algorithm is that the pixels are updated only if a statistic measure on the intensity variations of each pixels is greater to an adaptive threshold, thus reducing the I/O channel occupation. Experimental results on dynamic scenes taken from a fixed camera show that the proposed algorithm produces background images with an improved quality with respect to classical pixel-wise algorithms.

Apprenticeship learning based on inconsistent demonstrations is presented in this paper. We address a problem where given demonstrations are not directly applicable to reward function estimation due to the non-stationarity of an environment or the difference between the dynamics of a robot and a demonstrator. A basic idea of the proposed method is to use a subset of the trajectories sampled from the baseline policy as training data for inverse reinforcement learning. All consistent sample trajectories and inconsistent demonstrations are abstracted by an affine transformation invariant feature. Using the feature, the importance of each sample trajectory is estimated. Rating the sample trajectories based on importance, the training data for inverse reinforcement learning are identified. The validity of our approach is verified through simulation in two scenarios: inconsistency caused by variation of an environment and performance of a robot.

This paper proposes a human detection method that combines range image segmentation and human detection based on image local features. The method uses a stereo vision system called Subtraction Stereo, which extracts a range image of foreground regions. An extracted range image is segmented for each object by Mean Shift Clustering. Human detection based on local features is applied to each segment of foreground regions to detect humans. In this process, regions to scan a detection window for extracting local features are restricted. In addition, the size of the detection window is obtained using the distance information of a range image and camera parameters. Therefore, processing time and false detection can be reduced. Joint HOG features are used as the image local features. When applying the Joint HOG based human detection, occlusion of multiple humans is considered in construction of a classifier and in integration of detection windows, which improves the detection performance for the occluded humans. The proposed method is evaluated by experiments comparing with the method using Joint HOG features only. 11fps fast human detection is achieved.

A fast human detection system using a stereo camera is constructed. “Subtraction stereo”, that can measure distance information of foreground regions, is used to restrict regions for human detection and to adapt the detection window size. Two methods are introduced for human detection. One is a method based on template matching using gradient images, and the other is a method using approximated Shape Context (aSC) descriptors focusing on human upper bodies. High human detection performance better than the standard HOG-based method with low calculation cost is achieved by the combination of the two methods. The effectiveness of the proposed system is verified experimentally.

In this paper, a compact range image sensor used for short-range measurements is constructed with a multi-slit laser projector. Three-dimensional (3D) information obtained with a sensor is important for a robot that grasps an object. Sensors attached to robots for measurement are often hindered by occlusion by the hand immediately before an object is grasped. The constructed sensor is compact enough to be attached to a robot's hand, and occlusion can thus be avoided Compactness of the sensor is achieved by using a small CMOS camera and a laser projector and setting the baseline length between them as short as possible. The short baseline length also enables measurement in a short distance. Some experiments verify that the constructed sensor obtains accurate 3D information of objects in a short distance.

In this paper, we present an integrated framework, named NaviComf, which constructs pedestrian navigation systems for comfort in varying environments by using multi-modal sensing technologies. With NaviComf we aim to systematically provide solutions to three key problems: (1) how to build the environmental data warehouse (EDW) which works as an infrastructure providing comprehensive and predictive environmental information, (2) how to integrate heterogeneous environmental information from multi-modal sensors into an aggregate value which facilitates further processing, and (3) how to determine the optimal path plans in environments which are varying continuously. In NaviComf the multidimensional data model and data prediction method are applied to build the EDW. Then a novel multi-factor cost (MFC) model is proposed as the fundamental concept to integrate the multi-modal sensor data. Based on the former two solutions, the optimal path planning (PP) problem is solved in a time-dependent network by applying a dynamic programming method. In the evaluations of NaviComf, sensor data for temperature, humidity, and pedestrian traffic flow have been gathered in real environments and a prototype system has been implemented with the data. Evaluations are conducted by using the prototype system and the results show that NaviComf can efficiently navigate pedestrians through more comfortable paths as compared to the traditional navigation method.

In this paper, we propose a fast and stable human detection based on "subtraction stereo" which can measure distance information of foreground regions. Scanning an input image by detection windows is controlled in their window sizes and number using the distance information obtained from subtraction stereo. This control can skip a large number of detection windows and leads to reduce the computational time and false detection for fast and stable human detection. Additionally, we propose two-step boosting as a new training way of classifier with whole and upper human body models. Experimental results show that the proposal is faster and less false detection than the method described in the reference [1].

In this paper we present a method for the detection of moving objects for unknown and generic environments under cast shadow and periodic movements of non relevant objects (like waving leaves), using a combination of non-parametric thresholding algorithms and local cast shadow analysis with stereo camera information. Good detection rates were achieved in several environments under different lighting conditions, and objects could be detected independently of scene illumination, shadow, and periodic changes.

We propose a method for multi-object segmentation in a projection plane. Our algorithm requires a stereo camera system called Subtraction Stereo, which extracts foreground information with a fixed stereo camera. The main contribution of this paper is how the image sequences that include partial occlusion of the foreground objects can be accurately segmented using mean shift clustering in real-time processing. The proposed method is suitable for inside a medium-sized environment, such as a room. Finally, we try to segment the sequences that include occlusion and show the accuracy of the proposed method.

Detection of obstacles on a plane is important for a mobile robot that moves in a living space, especially for a humanoid that falls down even with a small obstacle. In this paper, a range image sensor for detecting small obstacles on a plane is constructed using a multi-slit laser projector. The sensor consists of a commercially available laser projector and a CCD camera. It measures a relative disparity map (RDMap) whose measurement errors are not affected by the distance. From the obtained RDMap, a plane is estimated using RANSAC and regions out of the plane are detected as obstacles. Experiments show that planes can be obtained with small errors in RDMaps with the constructed sensor, and that a humanoid with the constructed sensor can detect small obstacles such as a moving ping-pong ball and a LAN cable on a plane by the proposed methods while walking.

In this paper, a high-speed range image sensor using a multi-spot laser projector is constructed. Several high-speed range image sensors have been developed recently. Their sampling rate is around the video rate (30Hz or so) and a faster sensor is required. The proposed sensor has achieved 200Hz measurement. It consists of a commercially available laser projector and a high-speed CCD camera. The number of pixels is 361 and the measurement range is 800–2000mm. Although the acquired range image is sparse, the proposed sensor is thought to be adequate for several applications such as robot vision because of its high-speed imaging and compactness. Some characteristics such as measurement errors are discussed, and the effectiveness of the proposed sensor is verified by experiments.

In this paper, methods for detecting obstacles on a plane using a relative disparity map (RDMap) are proposed and discussed. The RDMap, which was formerly introduced by the author Umeda, is relative to a plane that is observed at first as the reference. It has an interesting feature that a plane in real 3D space also becomes a plane in the map, and has homogeneous characteristics compared to an ordinary range image. The proposed methods work even when the pose of the sensor changes significantly, which is the case in humanoid walking. First, a method to detect planar regions and obstacles by fitting a plane to the RDMap and a method to obtain the pose parameters from the RDMap are introduced. Fundamental experiments are then conducted to verify that a plane in real 3D space becomes a plane in the RDMap and that obstacles can be detected using the residual sum of squares for the fitted plane, and measurement errors in pose parameters are then evaluated. Finally, an experimental system with a humanoid and a small range image sensor is constructed, and it is demonstrated that the humanoid can detect obstacles on a plane by the proposed methods while walking.

In this paper, the sampling real-time QMDP value method is proposed and applied to a goalkeeper task of a soccer robot. Soccer is a challenging task for an autonomous robot that has poor computing resources and sensors, and a good subject in the study of decision making in dynamic environments. A robot frequently decides its behavior without enough observation of the environment. In the proposed method, the risk of a score is solved beforehand toward every set of position of the goalkeeper and motion of the ball. The shortage of observation is calculated and represented by particle filters. The proposed method uses the risk function and the particle filters so as to choose appropriate behavior of the goalkeeper. The experiment with an actual robot suggests that the method can decide actions reflexively toward the motion of the ball.

Time optimal control method for quadruped walking robots are developed and installed into a practical robot system. Each leg is modeled as a two link manipulator whose time optimal control theory has already been established by Bobrow. However, in legged robot systems, each leg supports their body weight, and the reaction forces from its ground must be inside of their friction corn. Moreover, the ZMP (zero moment point) of the robot is constrained for stable walk. Therefore, time optimal control inputs must be designed considering these constraints. SONY ERS-7 is used as a quadruped walking robot and a fundamental experiment is done. From the experimental results, the effectiveness of the developed control algorithm is verified

We have proposed the real-time QMDP method for decision making of a robot under uncertain state recognition. This method evaluates every action and chooses the best one with a particle filter for estimation and a state-value function of dynamic programming. Different from our past work, this paper applies it to a complicated decision making task that yields local maxima and discontinuity on the state-value function. We then verify whether the method can choose proper actions or not in such a condition. As an example, total behavior of a goalkeeper for robot soccer is planned by using value iteration. This task contains three strategies, which are related to three kinds of local maxima respectively. Simulations, experiments and actual games have suggested that the method can decide actions effectively according as uncertain result of state estimation.

This paper discusses a range image sensor using a multi-spot laser projector. In many situations, it is inevitable for robots to behave in three-dimensional (3D) environment for various tasks and range images are important sensor information. The sensor in this paper is thought to be suitable to acquire range images for many robotic applications. Characteristics of the sensor are discussed as its principle, its measurement range and precision. Rotation of the CCD camera is presented to improve the sensor's efficiency. A prototype of the sensor is constructed using a commercial laser projector, and its efficiency is evaluated by experiments. Furthermore, a novel method for detecting planar regions is proposed as an application of this sensor. Relative disparity map is introduced and proved to be effective for detecting planar regions.

This paper proposes an intelligent room that is free of operator's position based on gesture recognition technologies. Intention and position of an operator are recognized by detecting hand waving, and pan-tilt cameras are zoomed and focused on the operator. The hand region is extracted using color information, and direction or number of fingers and motion of the hand region are detected. Home appliances such as a television set are controlled by using the direction or number of fingers and hand motions.

Uncertainty of self-localization is one of the most serious problems related to navigation of autonomous mobile robots. There are some studies that deal with this problem. However, most of them assume that the extent of the uncertainty is known or has been measured previously, though it changes easily with some trivial changes of the environment. To avoid this assumption, we take the following approach: 1) a robot uses a self- localization method that is quite robust against sensing noise and the environment change, 2) the plan for robot behavior is based on the assumption that the robot can recognize the environment perfectly, 3) a novel decision-making algorithm computes proper behavior from the planning result and self-localization results in real time. These algorithms were implemented on a soccer robot of the RoboCup four-legged robot league, and their efficiency was verified with experiments.

Focuses on a method of estimating contact force dynamically between environment and an object, which is grasped by the soft planar fingers with tactile sensors. The external contact force acting on the grasped object causes the tactile sensors to generate the displacement distributions and the force distributions. We present a method of dynamic tactual image acquisitions for both distributions at the pseudo video rate. Experimental results show the measurement of the tactual image flows due to the deflection of an object and the magnitude and orientation of force vectors operating on the finger according to the external contact force. The contact position of the external force acting on the grasped object is estimated within 15% from the tactual image distributions.

This paper proposes a new methodology of stereo vision: the subpixel stereo method. The method controls the disparity less than one pixel with very small baseline length. Although the accuracy of measured distance is not high, this method is effective for robot vision because of the avoidance of the false correspondence problem, and simple high speed operation. Applications of the method to low resolution image and to the active method are also discussed.

The improvement of micro vision sensor is required for milli-machines and micromachines. This paper proposes a compound-eye-type micro vision sensor with simple structure for milli-machines. The sensor consists of only a photo sensor array and a cover. A prototype of the sensor is constructed. As an application of the sensor, detection of shape and motion is discussed. Experiments with the prototype show that the proposed sensor can obtain rough shape and motion, and is applicable for practical use.

Describes the sensing behavior of a high compliant tactile sensor on an object's surface and a formulation of the tactual sensing using methods of differential geometry. The tactile sensor has been manufactured with a function of high compliance, which is achieved with silicone rubber placed on a sensing mechanism organized for the detection of contact patterns. The mechanism is constructed as an optical wave guide which is capable of transmitting the contact pattern to a CCD camera. From the contact pattern, the tactile sensor detects the displacement distribution and the force distribution corresponding to the object's surface shape. By using a method derived from differential geometry, the geometrical contact mechanisms between the sensor and the surface of an object are discussed. The first fundamental form and the second fundamental form are defined by the displacement distribution generated by the tactual sensing. The local shape of the object's surface in, the neighborhood of a contact point is determined from the Gaussian curvature and the mean curvature given by the first fundamental form and the second fundamental form defined by the displacement distribution. Experimental results for estimating an object's local surface shape are presented for a tactile sensor manufactured for trial experiments.

A method of determining optimal grasping forces for a parallel two-fingered hand with compliant 2D tactile sensors is described. The tactile sensors can get 2D patterns of pressure distribution and have many potential applications such as shape recognition of a grasped object, assembly tasks and so on. In order to achieve assembly tasks using robot hands, information about the position of contact point and the contact force between the grasped object and its environment is essential. When contact force is applied to a grasped object from the environment, the pressure patterns of tactile sensors change according to not only amplitude of the applied force but also the contact position. Thus, a way to derive the contact position and force by tactile sensor data has been studied until now, and it is proven that the maximum measurable contact force from the environment depends on not only sensor characteristics but also grasping force. In the paper a way to determine the optimal grasping force to measure the largest contact force is proposed and some fundamental experiments for testing the derived grasping forces are done.

This paper proposes methods of fusing tactual and visual data for a hand-eye cooperation system, and realizes the methods using a developed tactile sensor. Since tactual and visual data convey information with different characteristics, smart fusion of these two sensor data is expected to provide useful additional information. In this paper, visually identified edges and planar regions or edges from tactual data are utilized and three fusion methods are introduced (1) measurement of a 3D edge when a tactile sensor contacts a plane, (2) measurement of a 3D edge when a tactile sensor contacts a plane at the edge and (3) measurements of a cylinder when a tactile sensor contacts the cylinder at an edge.

Gesture is one of the important channels of man-machine interface. In the paper, methods to recognize hand gestures using range images are proposed. Five hand gestures: "come on", "go away", "turn right", "turn left" and "stop" are dealt with. A sequence of velocity vectors and normal vectors of a moving hand are extracted from range images, and gestures are recognized using the features. A recognition system is constructed which utilizes the recognition methods. Experiments to recognize persons' gestures using the recognition system are performed to show the effectiveness of the proposed methods and the recognition system.

This paper proposes methods for recognizing three dimensional (3D) shape with range images and intensity images which are measured by multiple robots. Distributed sensing is a key technology for multiple robot systems. As sensory information for the robot system, range images and intensity images are both useful and complementary, and thus fusion of the two images is thought to be effective. In this paper each robot is assumed to have a range image sensor and/or an intensity image sensor. Planar regions, 3D edges and cylindrical regions are extracted by the distributed sensing system as robust features for 3D shape recognition. Methods of the feature extraction which are based on sensor fusion technology, and a prototype of a model matching method with the features are proposed. Experiments are performed to show the effectiveness of the proposed methods of feature extraction and model matching.

This paper presents methods for fusing a range image and an intensity image. The fusion is thought to be an important technology for 3D recognition of an intelligent robot system. In this paper, planar and cylindrical regions are selected as the features from a range image and an edge from an intensity image, and the following methods of fusing the features are proposed and formulated: (1) extraction of planar and cylindrical regions from a range image as the pre-process of the fusion, (2) measurement of 3D edges by fusing a planar region and an edge, (3) update of parameters of a cylindrical region by fusing an edge. Some experiments show the effectiveness of these methods for 3D shape recognition.

Gesture recognition using images is recently studied as a technology of human friendly man-machine interface. In this paper recognition of 3D gesture by using range images is discussed. Two methods for recognizing head motion from a sequence of range images are proposed One method uses differential range images. Vertically/horizontally weighted averages of the difference values are utilized and 'Yes' motion is robustly identified. The other method detects and tracks nose position. 'Yes'/'No' motions are identified by the vertical/horizontal amplitude of vibration. Experiments to recognize a person's gesture using a range sensor system are performed to show the effectiveness of the methods.

Fusion of a range image and an intensity image is a key technology for robot vision. This paper presents (1) strategy for the fusion, and (2) some fundamental algorithms of the fusion. The characteristics of the two images are discussed and the strategy of the fusion: "detect surfaces from a range image, and edges and vertices from an intensity image; and then fuse these features in 2D or 3D space" is insisted. Algorithms of (a) classification of edges, (b) merging of 3D surfaces by an edge, and (c) measurement of 3D edges, are proposed as the fusion algorithms based on the strategy. Experiments with a sparse range image and an intensity image indicate the effectiveness of the algorithms.

Detecting three dimensional motion is one of the most important issues on robot vision. In this paper, an essential equation, the motion-range equation, is introduced for detecting 3D motion from 'range images'. The equation is represented in linear form against 3D motion parameters. Using the equation, the motion parameters are directly obtained without detecting the positions of an object. Statistical analysis is made and computational method is established by means of the equation. Analysis of the equation is performed. Experimental results on actual range images prove that the motion detection method described is simple and efficient.>