Image-based 6-DoF pose estimation of arbitrarily shaped 3D objects based on their shape is a rarely studied problem. Most existing image-based methods for pose estimation either exploit textural information in form of local features or, if shape-based, rely on the extraction of straight line segments or other primitives. Straight-forward extensions of 2D approaches are potentially more general, but in practice assume a limited range of possible view angles. The general problem is that a 3D object can potentially produce completely different 2D projections depending on its relative pose to the observing camera. One way to reduce the solution space is to exploit temporal information, i.e. perform tracking. Again, existing model-based tracking approaches rely on relatively simple object geometries. In this paper, we propose a particle filter based tracking approach that can deal with arbitrary shapes and arbitrary or even no texture, i.e. it offers a general solution to the rigid object tracking problem. As our approach can deal with occlusions, it is in particular of interest in the context of goal-directed imitation learning involving the observation of object manipulations. Results of simulation experiments as well as real-world experiments with different object types prove the practical applicability of our approach.

A new approach to the 3D human motion tracking problem is proposed, which combines several particle filters with a physical simulation of a flexible body model. The flexible body model allows the partitioning of the state space of the human model into much smaller subsets, while finding a solution considering all the partial results of the particle filters. The flexible model also creates the necessary interaction between the different particle filters and allows effective semi-hierarchical tracking of the human body. The physical simulation does not require inverse kinematics calculations and is hence fast and easy to implement. Furthermore the system also builds an appearance model on-the-fly which allows it to work without a foreground segmentation. The system is able to start tracking automatically with a convenient initialization procedure. The implementation runs with 10 Hz on a regular PC using a stereo camera and is hence suitable for Human-Robot Interaction applications.

The problem of accurate 6-DoF pose estimation of 3D objects based on their shape has so far been solved only for specific object geometries. Edge-based recognition and tracking methods rely on the extraction of straight line segments or other primitives. Straight-forward extensions of 2D approaches are potentially more general, but assume a limited range of possible view angles. The general problem is that a 3D object can potentially produce completely different 2D projections depending on the view angle. One way to tackle this problem is to use canonical views. However, accurate shape-based 6-DoF pose estimation requires more information than matching of canonical views can provide. In this paper, we present a novel approach to 6-DoF pose estimation of single-colored objects based on their shape. Our approach combines stereo triangulation with matching against a high-resolution view set of the object, each view having associated orientation information. The errors that arise from separating the position and orientation computation in first place are corrected by a subsequent correction procedure based on online 3D model projection. The proposed approach can estimate the pose of a single object within 20 ms using conventional hardware.

In the recent past, the recognition and localization of objects based on local point features has become a widely accepted and utilized method. Among the most popular features are currently the SIFT features, the more recent SURF features, and region-based features such as the MSER. For time-critical application of object recognition and localization systems operating on such features, the SIFT features are too slow (500-600 ms for images of size 640×480 on a 3 GHz CPU). The faster SURF achieve a computation time of 150-240 ms, which is still too slow for active tracking of objects or visual servoing applications. In this paper, we present a combination of the Harris corner detector and the SIFT descriptor, which computes features with a high repeatability and very good matching properties within approx. 20 ms. While just computing the SIFT descriptors for computed Harris interest points would lead to an approach that is not scale-invariant, we will show how scale-invariance can be achieved without a time-consuming scale space analysis. Furthermore, we will present results of successful application of the proposed features within our system for recognition and localization of textured objects. An extensive experimental evaluation proves the practical applicability of our approach.

In this paper, we present a framework for perception, visualization, reproduction and recognition of human motion. On the perception side, various human motion capture systems exist, all of them having in common to calculate a sequence of configuration vectors for the human model in the core of the system. These human models may be 2D or 3D kinematic models, or on a lower level, 2D or 3D positions of markers. However, for appropriate visualization in terms of a 3D animation, and for reproduction on an actual robot, the acquired motion must be mapped to the target 3D kinematic model. On the understanding side, various action and activity recognition systems exist, which assume input of different kinds. However, given human motion capture data in terms of a high-dimensional 3D kinematic model, it is possible to transform the configurations into the appropriate representation which is specific to the recognition module. We will propose a complete architecture, allowing the replacement of any perception, visualization, reproduction module, or target platform. In the core of our architecture, we define a reference 3D kinematic model, which we intend to become a common standard in the robotics community, to allow sharing different software modules and having common benchmarks.

In this paper, we present an image-based markerless human motion capture system, intended for humanoid robot systems. The restrictions set by this ambitious goal are numerous. The input of the system is a sequence of stereo image pairs only, captured by cameras positioned at approximately eye distance. No artificial markers can be used to simplify the estimation problem. Furthermore, the complexity of all algorithms incorporated must be suitable for real-time application, which is maybe the biggest problem when considering the high dimensionality of the search space. Finally, the system must not depend on a static camera setup and has to find the initial configuration automatically. We present a system, which tackles these problems by combining multiple cues within a particle filter framework, allowing the system to recover from wrong estimations in a natural way. We make extensive use of the benefit of having a calibrated stereo setup. To reduce search space implicitly, we use the 3D positions of the hands and the head, computed by a separate hand and head tracker using a linear motion model for each entity to be tracked. With stereo input image sequences at a resolution of 320 times 240 pixels, the processing rate of our system is 15 Hz on a 3 GHz CPU. Experimental results documenting the performance of our system are available in form of several videos.

Robust vision-based grasping is still a hard problem for humanoid robot systems. When being restricted to using the camera system built-in into the robot's head for object localization, the scenarios get often very simplified in order to allow the robot to grasp autonomously. Within the computer vision community, many object recognition and localization systems exist, but in general, they are not tailored to the application on a humanoid robot. In particular, accurate 6D object localization in the camera coordinate system with respect to a 3D rigid model is crucial for a general framework for grasping. While many approaches try to avoid the use of stereo calibration, we will present a system that makes explicit use of the stereo camera system in order to achieve maximum depth accuracy. Our system can deal with textured objects as well as objects that can be segmented globally and are defined by their shape. Thus, it covers the cases of objects with complex texture and complex shape. Our work is directly linked to a grasping framework being implemented on the humanoid robot ARM AR and serves as its perception module for various grasping and manipulation experiments in a kitchen scenario.

A general solution for image-based object recognition and localization is still a goal far away. Therefore, the only way to tackle the problem is to apply the suitable approach for each specific problem. The most common techniques can be classified into global appearance-based, model-based, or histogram-based approaches, and approaches based on local features. In this paper, we concentrate on recognition and full 6D localization of solid colored objects of any geometry for real-time application on a humanoid robot system. State-of-the-art model-based methods can only deal with object geometries which can be broken down into 3D lines and planes, and thus can be efficiently projected into the image plane, which is not the case for most objects in a realistic scenario. In contrast, appearance-based methods have the power to be applicable for any object geometry, but are rarely combined with full 6D localization of objects, which is required for any realistic application in the context of grasping with a humanoid robot. We present a system which combines the benefits of global appearance-based and model-based approaches, resulting in a system which can acquire object representations automatically given its 3D model, and can recognize and localize solid-colored objects in 6D in an arbitrary scene in real-time

In this paper we present a framework for grasp planning with a humanoid robot arm and a five-fingered hand. The aim is to provide the humanoid robot with the ability of grasping objects that appear in a kitchen environment. Our approach is based on the use of an object model database that contains the description of all the objects that can appear in the robot workspace. This database is completed with two modules that make use of this object representation: an exhaustive offline grasp analysis system and a real-time stereo vision system. The offline grasp analysis system determines the best grasp for the objects by employing a simulation system, together with CAD models of the objects and the five-fingered hand. The results of this analysis are added to the object database using a description suited to the requirements of the grasp execution modules. A stereo camera system is used for a real-time object localization using a combination of appearance-based and model-based methods. The different components are integrated in a controller architecture to achieve manipulation task goals for the humanoid robot

In this paper we present a new approach to 3D shape acquisition. This implementation enables a robot arm to move the scan unit over the object without the scan unit being tracked. The chosen structured light approach uses an initially unknown white noise pattern that can easily be projected with any fixed pattern projection system. Object points are acquired on the basis of finding correspondences in the pattern in the camera image of the current scene. This is achieved by using a fast SSD correlation algorithm. In the current test setup we are using a standard video beamer and a standard digital camera, but we have also set up a scan head for fixation on a robots’ wrist. The special requirements of this miniaturized system will be explained as well as our implementation. Our approach reduces hardware complexity to a minimum using only one calibrated camera and one calibrated fixed pattern projector. In this paper we present the whole system also including the calibration procedure. The focus will be on the hardware setup, but we also give an introduction to the software methods used.

In this paper we want to introduce an approach to 3D scanning of dynamic scenes. This implementation enables the user not only to manually move a scan head over the object to scan but also to capture moving objects. Registration from scan to scan is done in real-time. The user interacts with the system and watches the scene assembling. He can immediately respond on shadings that occur due to undercuts in the scene. The chosen structured light scan method uses a primarily unknown speckle image that can be easily etched on a chrome-on-glass slide and projected using a strobe light. In the current large scale implementation we are using a standard video beamer and a standard digital camera. Miniaturization and adaptation for special purposes (i. e. medical applications) are scheduled for next year. Focus in this paper shall be laid on calibration issues regarding camera and projector.