In this paper, a strategy that combines the tracking of an object and the selected grasp on it is presented. This strategy is intended to provide enough information to a control system for executing the grasp. The tracking of the object uses the ICondensation and AARM prediction algorithms to predict the current object position and shape in the image space. Using a set of quality criteria, stable grasps are searched for on the object contour and the best one is selected. The selected grasp is tracked together with the object contour in the sequence of images.

Research has focused on developing control strategies that allow to work with imprecisely calibrated or even uncalibrated robot systems. A popular approach is to close the control loop with visual feedback (visual servoing), even though this requires continuous, high rate image processing. In contrast, we developed a strategy that can compensate calibration errors by integrating sparse visual feedback asynchronously. This strategy was motivated by neuroscientific studies which show that human grasping definitely requires only sparse asynchronous visual feedback, even with disturbed visual input. We analyze the performance of our control strategy on a robotic hand-eye system consisting of a 6 d.o.f. manipulator and a pan-tilt head with a stereo camera system. To analyze thoroughly which calibration errors can be compensated by sparse feedback, we furthermore built a complete simulation model of the real hand-eye system, including the vision modules. Taken together, experimental and simulation results show that calibration errors can be compensated with a very moderate amount of (asynchronous) visual feedback. Thus, the proposed control strategy can indeed be seen as a serious alternative to visual servoing for real robotics applications. This shows that robotics can profit from taking a closer look at the results of neuroscience.

In the field of robot motion control, visual servoing has been proposed as the suitable strategy to cope with imprecise models and calibration errors. Remaining problems such as the necessity of a high rate of visual feedback are deemed to be solvable by the development of real-time vision modules. However, human grasping, which still outshines its robotic counterparts especially with respect to robustness and flexibility, definitely requires only sparse, asynchronous visual feedback. We therefore examined current neuroscientific models for the control of human reach-to-grasp movements with the emphasis lying on the visual control strategy used. From this, we developed a control model that unifies the two robotic strategies look-then-move and visual servoing, thereby compensating the problems that each strategy shows when used alone.

In the field of hand-eye coordination, most state-of-the-art systems still require the user to select the grasping points manually. We present a system which autonomously determines 3D grasping points on unknown objects from a pair of greyscale images. The object to be grasped is segmented automatically when put into the scene. Grasping points are searched on the object silhouette; their stability is evaluated by a heuristic algorithm, primarily based on the skeleton of the region. The 3D grasping pose is estimated by triangulation using a simplified geometrical model of the camera system; the corresponding points in the second image are determined via dynamic programming. The whole system has been implemented and validated on the experimental hand-eye system MINERVA.