Planetary exploration systems, operating under severe environmental and operating conditions, have thus far successfully employed carefully calibrated stereo cameras and manipulators to achieve desired precision in instrument placement activities. However, the environmental and functional restrictions imposed by the remote operation of semi-autonomous robots in a harsh planetary atmosphere for long periods of time limit the ultimate operational accuracy of this approach. This paper builds on an algorithm, referred to as Hybrid Image-Plane/Stereo (HIPS), developed to optimize the positioning accuracy of a manipulator. The HIPS method generates camera models through direct visual sensing of the manipulator end-effector. It estimates and subsequently uses these models to position the manipulator at a target location specified in the image-planes of a stereo camera pair using stereo correlation and triangulation. While positioning control of manipulation systems is important, orientation control of these systems is also crucial. Many planetary manipulation tasks being considered for the Mars Science Laboratory rover, due to launch in 2009, and subsequent missions, will require precise orientation control of manipulators. This paper studies the effect of using HIPS models to control the position and orientation of manipulator end-effector. As seen in previous position control experiments, the static version of the HIPS technique reduces position error by almost an order of magnitude. This paper additionally shows that orientation error is reduced by almost a factor of two.

The requirements for planetary manipulation systems are becoming more stringent with time. Missions are moving beyond simple driving and instrument placement to include sample handling, target selection from meters away, and precision re-sampling of a location. In one of these tasks, remote target selection, the target is selected in a high-viewpoint camera pair. The rover begins tracking the target in these cameras as the rover approaches the target. When the rover gets close enough that the target is in view of the low-viewpoint cameras, target handoff occurs. Target handoff is the process of identifying the same target in each set of cameras so that the low-viewpoint cameras can commence tracking. Tracking is then done in the lower cameras until the rover is within optimal range for manipulation operations. Target handoff also has applications in areas such as surveillance, security, and multi-viewpoint tracking. Planetary manipulation systems typically employ two sets of calibrated stereo cameras to achieve precision on the order of one centimeter or ten pixels. As imprecision in handoff translates to location error of the final sample, this level of precision is not sufficient for upcoming missions. By utilizing the method of hybrid image-plane/stereo (HIPS), initially developed for visually guided manipulation, handoffs have been achieved with errors of about one and a half pixels.

Lemur IIa, the second-generation Limbed Excursion Mechanical Utility Robot, is a six-limbed walking robot developed at NASA' s Jet Propulsion Laboratory. Lemur is distinct in part because each limb was designed be used for both walking and manipulation. It is intended to be a prototype of a dexterous, operationally flexible robot for on-orbit assembly, inspection, and maintenance. This video illustrates some of the current capabilities of this platform. Demonstrations included are omnidirectional walking without rotating the robot, body pose control from a fixed stance, visually guided docking with a recharging station, and manipulation and inspection with multiple tools (on multiple limbs). These capabilities represent a sample of the current tasks Lemur is performing. Please see www-robotics.jpl.nasa.gov for more information and demonstrations about Lemur.

Lemur IIa, the second-generation Limbed Excursion Mechanical Utility Robot, is a six-limbed walking robot developed at NASA' s Jet Propulsion Laboratory. Lemur is distinct in part because each limb was designed be used for both walking and manipulation. It is intended to be a prototype of a dexterous, operationally flexible robot for on-orbit assembly, inspection, and maintenance. This video illustrates Lemur completing an autonomous assembly task. Lemur, as part of a robotic team, is helping to assemble a structure. In this demonstration, Lemur autonomously walks through a truss, locates a task board, and tightens a bolt on that board. These demonstrations represent a sample of the current tasks Lemur is performing. Please see www-robotics.jpl.nasa.gov for more information and demonstrations about Lemur.

Ego-motion (self-motion) of the camera is a considerable problem in outdoor rover applications. Stereo tracking of a single moving target is a difficult problem that becomes even more challenging when rough terrain causes significant and high-acceleration motion of the camera in the world. The paper discusses the use of inertial measurements to estimate camera ego-motion and the use of these estimates to augment stereo tracking. Some initial results from an outdoor rover are presented, illustrating the efficacy of the method. The method causes fast but predictable image location transients, but reduces the amplitude of image location transients due to the rough terrain.

In this paper, we present methods for tracking complex, articulated objects. We assume that an appearance model and the kinematic structure of the object to be tracked are given, leading to what is termed a model-based object tracker. At each time step, this tracker observes a new monocular grayscale image of the scene and combines information gathered from this image with knowledge of the previous configuration of the object to estimate the configuration of the object at the time the image was acquired. Each degree of freedom in the model has an uncertainty associated with it, indicating the confidence in the current estimate for that degree of freedom. These uncertainty estimates are updated after each observation. An extended Kalman filter with appropriate observation and system models is used to implement this updating process. The methods that we describe are potentially beneficial to areas such as automated visual tracking in general, visual servo control, and human computer interaction.

The pervasiveness of computers in everyday life coupled with recent rapid advances in computer technology have created both the need and the means for sophisticated human-computer interaction (HCI) technology. Despite all the progress in computer technology and robotic manipulation, the interfaces for controlling manipulators have changed very little in the last decade. Therefore human-computer interfaces for controlling robotic manipulators are of great interest. A flexible and useful robotic manipulator is one capable of movement in three translational degrees of freedom, and three rotational degrees of freedom. In addition to research labs, six degree of freedom robots can be found in construction areas or other environments unfavorable for human beings. This paper proposes an intuitive and convenient visually guided interface for controlling a robot with six degrees of freedom. Two orthogonal cameras are used to track the position and the orientation of the hand of the user. This allows the user to control the robotic arm in a natural way.

Performance estimation for feature tracking is a critical issue, if feature tracking results are to be used intelligently. In this paper, we derive quantitative measures for the spatial accuracy of a particular feature tracker. This method uses the results from the sum-of-squared-differences correlation measure commonly used for feature tracking to estimate the accuracy (in the image plane) of the feature tracking result. In this way, feature tracking results can be analyzed and exploited to a greater extent without placing undue confidence in inaccurate results or throwing out accurate results. We argue that this interpretation of results is more flexible and useful than simply using a confidence measure on tracking results to accept or reject features. For example, and extended Kalman filtering framework can assimilate these tracking results directly to monitor the uncertainty in the estimation process for the state of an articulated object.

We describe a model-based object tracking system that updates the configuration parameters of an object model based upon information gathered from a sequence of monocular images. Realistic object and imaging models are used to determine the expected visibility of object features, and to determine the expected appearance of all visible features. We formulate the tracking problem as one of parameter estimation from partially observed data, and apply the extended Kalman filtering (EKF) algorithm. The models are also used to determine what point feature movement reveals about the configuration parameters of the object. This information is used by the EKF to update estimates for parameters, and for the uncertainty in the current estimates, based on observations of point features in monocular images.