While both navigation and manipulation are chal-lenging topics in isolation, many tasks require the ability to both navigate and manipulate in concert. To this end, we propose a mobile manipulation system that leverages novel navigation and shape completion methods to manipulate an object with a mobile robot. Our system utilizes uncertainty in the initial estimation of a manipulation target to calculate a predicted next-best-view. Without the need of localization, the robot then uses the predicted panoramic view at the next-best-view location to navigate to the desired location, capture a second view of the object, create a new model that predicts the shape of object more accurately than a single image alone, and uses this model for grasp planning. We show that the system is highly effective for mobile manipulation tasks through simulation experiments using real world data, as well as ablations on each component of our system.

We propose a vision-based reinforcement learning (RL) approach for closed-loop trajectory generation in an arm reaching problem. Arm trajectory generation is a fundamental robotics problem which entails finding collision-free paths to move the robot’s body (e.g. arm) in order to satisfy a goal (e.g. place end-effector at a point). While classical methods typically require the model of the environment to solve a planning, search or optimization problem, learning-based approaches hold the promise of directly mapping from observations to robot actions. However, learning a collision-avoidance policy using RL remains a challenge for various reasons, including, but not limited to, partial observability, poor exploration, low sample efficiency, and learning instabilities. To address these challenges, we present a residual-RL method that leverages a greedy goal-reaching RL policy as the base to improve exploration, and the base policy is augmented with residual state-action values and residual actions learned from images to avoid obstacles. Further more, we introduce novel learning objectives and techniques to improve 3D understanding from multiple image views and sample efficiency of our algorithm. Compared to RL baselines, our method achieves superior performance in terms of success rate.

Grasping in dynamic environments presents a unique set of challenges. A stable and reachable grasp can become unreachable and unstable as the target object moves, motion planning needs to be adaptive and in real time, the delay in computation makes prediction necessary. In this paper, we present a dynamic grasping framework that is reachability-aware and motion-aware. Specifically, we model the reachability space of the robot using a signed distance field which enables us to quickly screen unreachable grasps. Also, we train a neural network to predict the grasp quality conditioned on the current motion of the target. Using these as ranking functions, we quickly filter a large grasp database to a few grasps in real time. In addition, we present a seeding approach for arm motion generation that utilizes solution from previous time step. This quickly generates a new arm trajectory that is close to the previous plan and prevents fluctuation. We implement a recurrent neural network (RNN) for modelling and predicting the object motion. Our extensive experiments demonstrate the importance of each of these components and we validate our pipeline on a real robot.

In reinforcement learning (RL), sparse rewards are a natural way to specify the task to be learned. However, most RL algorithms struggle to learn in this setting since the learning signal is mostly zeros. In contrast, humans are good at assessing and predicting the future consequences of actions and can serve as good reward/policy shapers to accelerate the robot learning process. Previous works have shown that the human brain generates an error-related signal, measurable using electroencephelography (EEG), when the human perceives the task being done erroneously. In this work, we propose a method that uses evaluative feedback obtained from human brain signals measured via scalp EEG to accelerate RL for robotic agents in sparse reward settings. As the robot learns the task, the EEG of a human observer watching the robot attempts is recorded and decoded into noisy error feedback signal. From this feedback, we use supervised learning to obtain a policy that subsequently augments the behavior policy and guides exploration in the early stages of RL. This bootstraps the RL learning process to enable learning from sparse reward. Using a simple robotic navigation task as a test bed, we show that our method achieves a stable obstacle-avoidance policy with high success rate, outperforming learning from sparse rewards only that struggles to achieve obstacle avoidance behavior or fails to advance to the goal.

Recent advancements in Learning from Human Feedback present an effective way to train robot agents via inputs from non-expert humans, without a need for a specially designed reward function. However, this approach needs a human to be present and attentive during robot learning to provide evaluative feedback. In addition, the amount of feedback needed grows with the level of task difficulty and the quality of human feedback might decrease over time because of fatigue. To overcome these limitations and enable learning more robot tasks with higher complexities, there is a need to maximize the quality of expensive feedback received and reduce the amount of human cognitive involvement required. In this work, we present an approach that uses active learning to smartly choose queries for the human supervisor based on the uncertainty of the robot and effectively reduces the amount of feedback needed to learn a given task. We also use a novel multiple buffer system to improve robustness to feedback noise and guard against catastrophic forgetting as the robot learning evolves. This makes it possible to learn tasks with more complexity using lesser amounts of human feedback compared to previous methods. We demonstrate the utility of our proposed method on a robot arm reaching task where the robot learns to reach a location in 3D without colliding with obstacles. Our approach is able to learn this task faster, with less human feedback and cognitive involvement, compared to previous methods that do not use active learning.

We present a robot navigation system that uses an imitation learning framework to successfully navigate in complex environments. Our framework takes a pre-built 3D scan of a real environment and trains an agent from pre-generated expert trajectories to navigate to any position given a panoramic view of the goal and the current visual input without relying on map, compass, odometry, or relative position of the target at runtime. Our end-to-end trained agent uses RGB and depth (RGBD) information and can handle large environments (up to 1031m2) across multiple rooms (up to 40) and generalizes to unseen targets. We show that when compared to several baselines our method (1) requires fewer training examples and less training time, (2) reaches the goal location with higher accuracy, and (3) produces better solutions with shorter paths for long-range navigation tasks.

Recent advances in deep reinforcement learning (RL) have demonstrated its potential to learn complex robotic manipulation tasks. However, RL still requires the robot to collect a large amount of real-world experience. To address this problem, recent works have proposed learning from expert demonstrations (LfD), particularly via inverse reinforcement learning (IRL), given its ability to achieve robust performance with only a small number of expert demonstrations. Nevertheless, deploying IRL on real robots is still challenging due to the large number of robot experiences it requires. This paper aims to address this scalability challenge with a robust, sample-efficient, and general meta-IRL algorithm, SQUIRL, that performs a new but related long-horizon task robustly given only a single video demonstration. First, this algorithm bootstraps the learning of a task encoder and a task-conditioned policy using behavioral cloning (BC). It then collects real-robot experiences and bypasses reward learning by directly recovering a Q-function from the combined robot and expert trajectories. Next, this algorithm uses the learned Q-function to re-evaluate all cumulative experiences collected by the robot to improve the policy quickly. In the end, the policy performs more robustly (90%+ success) than BC on new tasks while requiring no experiences at test time. Finally, our real-robot and simulated experiments demonstrate our algorithm's generality across different state spaces, action spaces, and vision-based manipulation tasks, e.g., pick-pour-place and pick-carry-drop.

This work provides an architecture that incorporates depth and tactile information to create rich and accurate 3D models useful for robotic manipulation tasks. This is accomplished through the use of a 3D convolutional neural network (CNN). Offline, the network is provided with both depth and tactile information and trained to predict the object's geometry, thus filling in regions of occlusion. At runtime, the network is provided a partial view of an object. Tactile information is acquired to augment the captured depth information. The network can then reason about the object's geometry by utilizing both the collected tactile and depth information. We demonstrate that even small amounts of additional tactile information can be incredibly helpful in reasoning about object geometry. This is particularly true when information from depth alone fails to produce an accurate geometric prediction. Our method is benchmarked against and outperforms other visual-tactile approaches to general geometric reasoning. We also provide experimental results comparing grasping success with our method.

Recent advances in on-policy reinforcement learning (RL) methods enabled learning agents in virtual environments to master complex tasks with high-dimensional and continuous observation and action spaces. However, leveraging this family of algorithms in multi-fingered robotic grasping remains a challenge due to large sim-to-real fidelity gaps and the high sample complexity of on-policy RL algorithms. This work aims to bridge these gaps by first reinforcement-learning a multi-fingered robotic grasping policy in simulation that operates in the pixel space of the input: a single depth image. Using a mapping from pixel space to Cartesian space according to the depth map, this method transfers to the real world with high fidelity and introduces a novel attention mechanism that substantially improves grasp success rate in cluttered environments. Finally, the direct-generative nature of this method allows learning of multi-fingered grasps that have flexible end-effector positions, orientations and rotations, as well as all degrees of freedom of the hand.

This work provides a framework for a workspace aware online grasp planner. This framework greatly improves the performance of standard online grasp planning algorithms by incorporating a notion of reachability into the online grasp planning process. Offline, a database of hundreds of thousands of unique end-effector poses were queried for feasibility. At runtime, our grasp planner uses this database to bias the hand towards reachable end-effector configurations. The bias keeps the grasp planner in accessible regions of the planning scene so that the resulting grasps are tailored to the situation at hand. This results in a higher percentage of reachable grasps, a higher percentage of successful grasp executions, and a reduced planning time. We also present experimental results using simulated and real environments.

Robotic manipulation of deformable objects is a difficult problem especially because of the complexity of the many different ways an object can deform. Searching such a high-dimensional state space makes it difficult to recognize, track, and manipulate deformable objects. In this paper, we introduce a predictive, model-driven approach to address this challenge, using a precomputed, simulated database of deformable object models. Mesh models of common deformable garments are simulated with the garments picked up in multiple different poses under gravity, and stored in a database for fast and efficient retrieval. To validate this approach, we developed a comprehensive pipeline for manipulating clothing as in a typical laundry task. First, the database is used for category and the pose estimation is used for a garment in an arbitrary position. A fully featured 3-D model of the garment is constructed in real time, and volumetric features are then used to obtain the most similar model in the database to predict the object category and pose. Second, the database can significantly benefit the manipulation of deformable objects via nonrigid registration, providing accurate correspondences between the reconstructed object model and the database models. Third, the accurate model simulation can also be used to optimize the trajectories for the manipulation of deformable objects, such as the folding of garments. Extensive experimental results are shown for the above tasks using a variety of different clothings. Note to Practitioners-This paper provides an open source, extensible, 3-D database for dissemination to the robotics and graphics communities. Model-driven methods are proliferating, and they need to be applied, tested, and validated in real environments. A key idea we have exploited is to have an innovative and novel use of simulation. This database will serve as infrastructure for developing advanced robotic machine learning algorithms. We want to address this machine learning idea ourselves, but we expect the dissemination of the database to other researchers with different agendas and task applications, which will bring wide progress in this area. Our proposed methods, as mentioned earlier, can be easily applied to interrelated areas. One example is that the 3-D shape-based matching algorithm can be used for other objects, such as bottles, papers, and food. After integrating with other robotic systems, the use of the robot can be easily extended to other tasks, such as making food, cleaning room, and fetching objects, to assist our daily life.

This work provides an architecture to enable robotic grasp planning via shape completion. Shape completion is accomplished through the use of a 3D convolutional neural network (CNN). The network is trained on our own new open source dataset of over 440,000 3D exemplars captured from varying viewpoints. At runtime, a 2.5D pointcloud captured from a single point of view is fed into the CNN, which fills in the occluded regions of the scene, allowing grasps to be planned and executed on the completed object. Runtime shape completion is very rapid because most of the computational costs of shape completion are borne during offline training. We explore how the quality of completions vary based on several factors. These include whether or not the object being completed existed in the training data and how many object models were used to train the network. We also look at the ability of the network to generalize to novel objects allowing the system to complete previously unseen objects at runtime. Finally, experimentation is done both in simulation and on actual robotic hardware to explore the relationship between completion quality and the utility of the completed mesh model for grasping.

Exemplar-based models have achieved great success on localizing the parts of semi-rigid objects. However, their efficacy on highly articulated objects such as humans is yet to be explored. Inspired by hierarchical object representation and recent application of Deep Convolutional Neural Networks (DCNNs) on human pose estimation, we propose a novel formulation that incorporates both hierarchical exemplar-based models and DCNNs in the spatial terms. Specifically, we obtain more expressive spatial models by assuming independence between exemplars at different levels in the hierarchy; we also obtain stronger spatial constraints by inferring the spatial relations between parts at the same level. As our method strikes a good balance between expressiveness and strength of spatial models, it is both effective and generalizable, achieving state-of-the-art results on different benchmarks: Leeds Sports Dataset and CUB-200-2011.

Robotic manipulation of deformable objects remains a challenging task. One such task is to iron a piece of cloth autonomously. Given a roughly flattened cloth, the goal is to have an ironing plan that can iteratively apply a regular iron to remove all the major wrinkles by a robot. We present a novel solution to analyze the cloth surface by fusing two surface scan techniques: a curvature scan and a discontinuity scan. The curvature scan can estimate the height deviation of the cloth surface, while the discontinuity scan can effectively detect sharp surface features, such as wrinkles. We use this information to detect the regions that need to be pulled and extended before ironing, and the other regions where we want to detect wrinkles and apply ironing to remove the wrinkles. We demonstrate that our hybrid scan technique is able to capture and classify wrinkles over the surface robustly. Given detected wrinkles, we enable a robot to iron them using shape features. Experimental results show that using our wrinkle analysis algorithm, our robot is able to iron the cloth surface and effectively remove the wrinkles.

We present RoboBench, a novel platform for sharing robot full-system simulations for benchmarking. The creation of this platform and benchmark suite is motivated by a need for reproducible research. A challenge in creating a full-system benchmarks are incompatibilities in software created by different groups and the difficulty of reproducing software environments. We solve this problem by using software containers, an emerging virtualization technology. RoboBench enables sharing robot software in a runnable state, capturing the software behavior of robots carrying out missions. These simulations make clear the performance impact and resource usage of programs and algorithms relative to other software involved in the mission. These containers are integrated with the CITK platform for reproducible research, which automates generation and publishing of the containers. We present an overview of the system, a description of our prototype set of benchmark missions, along with a validation study comparing the computational load profile of a mission performed on a real and simulated robot. Additionally, we present preliminary results of an overall analysis of the benchmarks in the RoboBench suite, showing where computational work is expended in robotics common robotics tasks. RoboBench is extensible, and is the first step toward a robust, quantitative approach to engineering computationally-efficient robots.

Deformable objects such as garments are highly unstructured, making them difficult to recognize and manipulate. In this paper, we propose a novel method to teach a two-arm robot to efficiently track the states of a garment from an unknown state to a known state by iterative regrasping. The problem is formulated as a constrained weighted evaluation metric for evaluating the two desired grasping points during regrasping, which can also be used for a convergence criterion The result is then adopted as an estimation to initialize a regrasping, which is then considered as a new state for evaluation. The process stops when the predicted thin shell conclusively agrees with reconstruction. We show experimental results for regrasping a number of different garments including sweater, knitwear, pants, and leggings, etc.

Robotic manipulation of deformable objects remains a challenging task. One such task is folding a garment autonomously. Given start and end folding positions, what is an optimal trajectory to move the robotic arm to fold a garment? Certain trajectories will cause the garment to move, creating wrinkles, and gaps, other trajectories will fail altogether. We present a novel solution to find an optimal trajectory that avoids such problematic scenarios. The trajectory is optimized by minimizing a quadratic objective function in an off-line simulator, which includes material properties of the garment and frictional force on the table. The function measures the dissimilarity between a user folded shape and the folded garment in simulation, which is then used as an error measurement to create an optimal trajectory. We demonstrate that our two-arm robot can follow the optimized trajectories, achieving accurate and efficient manipulations of deformable objects.

This paper presents a deep learning architecture for detecting the palm and fingertip positions of stable grasps directly from partial object views. The architecture is trained using RGBD image patches of fingertip and palm positions from grasps computed on complete object models using a grasping simulator. At runtime, the architecture is able to estimate grasp quality metrics without the need to explicitly calculate the given metric. This ability is useful as the exact calculation of these quality functions is impossible from an incomplete view of a novel object without any tactile feedback. This architecture for grasp quality prediction provides a framework for generalizing grasp experience from known to novel objects.

We present a novel method for classifying and estimating the categories and poses of deformable objects, such as clothing, from a set of depth images. The framework presented here represents the recognition part of the entire pipeline of dexterous manipulation of deformable objects, which contains grasping, recognition, regrasping, placing flat, and folding. We first create an off-line simulation of the deformable objects and capture depth images from different view points as training data. Then by extracting features and applying sparse coding and dictionary learning, we build up a codebook for a set of different poses of a particular deformable object category. The whole framework contains two layers which yield a robust system that first classifies deformable objects on category level and then estimates the current pose from a group of predefined poses of a single deformable object. The system is tested on a variety of similar deformable objects and achieves a high output accuracy. By knowing the current pose of the garment, we can continue with further tasks such as regrasping and folding.

Pose estimation of deformable objects is a fundamental and challenging problem in robotics. We present a novel solution to this problem by first reconstructing a 3D model of the object from a low-cost depth sensor such as Kinect, and then searching a database of simulated models in different poses to predict the pose. Given noisy depth images from 360-degree views of the target object acquired from the Kinect sensor, we reconstruct a smooth 3D model of the object using depth image segmentation and volumetric fusion. Then with an efficient feature extraction and matching scheme, we search the database, which contains a large number of deformable objects in different poses, to obtain the most similar model, whose pose is then adopted as the prediction. Extensive experiments demonstrate better accuracy and orders of magnitude speed-up compared to our previous work. An additional benefit of our method is that it produces a high-quality mesh model and camera pose, which is necessary for other tasks such as regrasping and object manipulation.

Surgeons perform minimally invasive surgery using an image delivered by a laparoscope and a camera system that provides a high definition 2D image, but this leaves the surgeon without 3D depth perception. The lack of depth perception can slow the surgeon, increase the risk of misidentifying structures, and/or inadvertently cause unwanted injury to tissues surrounding the surgical site. To address the lack of depth perception, we propose a Surgical Structured Light (SSL) system that includes a 3D sensor capable of measuring and modeling the surgical site during a procedure. The 3D information provided by this system can enable the surgeon to: 1) improve the navigation of tools based on precise localization of instruments in relation to structures in the surgical site, 2) allow 3D visualizations side-by-side with a standard 2D color image, and 3) precisely measure sizes of structures (e.g., tumors) and distances between structures with simple mouse clicks. We demonstrate the accuracy of our SSL system using ex-vivo data on both a cylinder calibration object as well as various plastic organs.

We present a joint demonstration between the Robotics, Autonomous Systems, and Controls Laboratory (RASCAL) at UC Davis and the Columbia University Robotics Group, wherein a human-in-the-loop robotic grasping platform in the Columbia lab (New York, NY) is controlled to select and grasp an object by a C3-C4 spinal cord injury (SCI) subject in the UC Davis lab (Davis, CA) using a new single-signal, multi-degree-of-freedom surface electromyography (sEMG) human-robot interface. The grasping system breaks the grasping task into a multi-stage pipeline that can be navigated with only a few inputs. It integrates pre-planned grasps with on-line grasp planning capability and an object recognition and target selection system capable of handling multi-object scenes with moderate occlusion. Previous work performed in the RASCAL lab demonstrated that by continuously modulating the power in two individual bands in the frequency spectrum of a single sEMG signal, users were able to control a cursor in 2D for cursor to target tasks. Using this paradigm, four targets were presented in order for the subject to command the multi-stage grasping pipeline. We demonstrate that using this system, operators are able to grasp objects in a remote location using a robotic grasping platform.

Due to pose uncertainty, merely executing a planned-to-be stable grasp usually results in an unstable grasp in the physical world. In our previous work [1], we proposed a tactile experience based grasping pipeline which utilizes tactile feedback to adjust hand posture during the grasping task of known objects and improves the performance of robotic grasping under pose uncertainty. In this paper, we extend our work to grasp novel objects by utilizing local geometric similarity. To do this, we select a series of shape primitives to parameterize potential local geometries which novel objects may share in common. We then build a tactile experience database that stores information of stable grasps on these local geometries. Using this tactile experience database, our method is able to guide a grasp adjustment process to grasp novel objects around similar local geometries. Experiments indicate that our approach improves the grasping performance on novel objects with similar local geometries under pose uncertainty.

There has been considerable interest in producing grasping platforms using non-invasive, low bandwidth brain computer interfaces(BCIs). Most of this work focuses on low level control of simple hands. Using complex hands improves the versatility of a grasping platform at the cost of increasing its complexity. In order to control more complex hands with these low bandwidth signals, we need to use higher level abstractions. Here, we present a user interface which allows the user to combine the speed and convenience of offline preplanned grasps with the versatility of an online planner. This system incorporates a database of pre-planned grasps with the ability to refine these grasps using an online planner designed for arbitrarily complex hands. Only four commands are necessary to control the entire grasping pipeline, allowing us to use a low cost, noninvasive commercial BCI device to produce robust grasps that reflect user intent. We demonstrate the efficacy of this system with results from five subjects and present results using this system to grasp unknown objects.

We deal with the problem of blind grasping where we use tactile feedback to predict the stability of a robotic grasp given no visual or geometric information about the object being grasped. We first simulated tactile feedback using a soft finger contact model in GraspIt! [1] and computed tactile contacts of thousands of grasps with a robotic hand using the Columbia Grasp Database [2]. We used the K-means clustering method to learn a contact dictionary from the tactile contacts, which is a codebook that models the contact space. The feature vector for a grasp is a histogram computed based on the distribution of its contacts over the contact space defined by the dictionary. An SVM is then trained to predict the stability of a robotic grasp given this feature vector. Experiments indicate that this model which requires low-dimension feature input is useful in predicting the stability of a grasp.

Underactuated compliant robotic hands exploit passive mechanics and joint coupling to reduce the number of actuators required to achieve grasp robustness in unstructured environments. Reduced actuation requirements generally serve to decrease design cost and improve grasp planning efficiency, but overzealous simplification of an actuation topology, coupled with insufficient tuning of mechanical compliance and hand kinematics, can adversely affect grasp quality and adaptability. This paper presents a computational framework for reducing the mechanical complexity of robotic hand actuation topologies without significantly decreasing grasp robustness. Open-source grasp planning software and well-established grasp quality metrics are used to simulate a fully-actuated, 24 DOF anthropomorphic robotic hand grasping a set of daily living objects. DOFs are systematically demoted or removed from the hand actuation topology according to their contribution to grasp quality. The resulting actuation topology contained 22% fewer DOFs, 51% less aggregate joint motion, and required 82% less grasp planning time than the fully-actuated design, but decreased average grasp quality by only 11%.

Grasp quality metrics which analyze the contact wrench space are commonly used to synthesize and analyze preplanned grasps. Preplanned grasping approaches rely on the robustness of stored solutions. Analyzing the robustness of such solutions for large databases of preplanned grasps is a limiting factor for the applicability of data driven approaches to grasping. In this work, we will focus on the stability of the widely used grasp wrench space epsilon quality metric over a large range of poses in simulation. We examine a large number of grasps from the Columbia Grasp Database for the Barrett hand. We find that in most cases the grasp with the most robust force closure with respect to pose error for a particular object is not the grasp with the highest epsilon quality. We demonstrate that grasps can be reranked by an estimate of the stability of their epsilon quality. We find that the grasps ranked best by this method are successful more often in physical experiments than grasps ranked best by the epsilon quality.

We design an example based planning framework to generate semantic grasps, stable grasps that are functionally suitable for specific object manipulation tasks. We propose to use partial object geometry, tactile contacts, and hand kinematic data as proxies to encode semantic constraints, which are task-related constraints. We introduce a semantic affordance map, which relates local geometry to a set of predefined semantic grasps that are appropriate to different tasks. Using this map, the pose of a robotic hand can be estimated so that the hand is adjusted to achieve the ideal approach direction required by a particular task. A grasp planner is then used to generate a set of final grasps which have appropriate stability, tactile contacts, and hand kinematics along this approach direction. We show experiments planning semantic grasps on everyday objects and executing these grasps with a physical robot.

We propose a machine learning approach to the perception of a stable robotic grasp based on tactile feedback and hand kinematic data, which we call blind grasping. We first discuss a method for simulating tactile feedback using a soft finger contact model in Grasplt!, which is a robotic grasping simulator [10]. Using this simulation technique, we compute tactile contacts of thousands of grasps with a robotic hand using the Columbia Grasp Database [6]. The tactile contacts along with the hand kinematic data are then input to a Support Vector Machine (SVM) which is trained to estimate the stability of a given grasp based on this tactile feedback and also the robotic hand kinematics. Experimental results indicate that the tactile feedback along with the hand kinematic data carry meaningful information for the prediction of the stability of a blind robotic grasp.

Continuum robots offer significant advantages for surgical intervention due to their down-scalability, dexterity, and structural flexibility. While structural compliance offers a passive way to guard against trauma, it necessitates robust methods for online estimation of the robot configuration in order to enable precise position and manipulation control. In this paper, we address the pose estimation problem by applying a novel mapping of the robot configuration to a feature descriptor space using stereo vision. We generate a mapping of known features through a supervised learning algorithm that relates the feature descriptor to known ground truth. Features are represented in a reduced sub-space, which we call eigen-features. The descriptor provides some robustness to occlusions, which are inherent to surgical environments, and the methodology that we describe can be applied to multi-segment continuum robots for closed-loop control. Experimental validation on a single-segment continuum robot demonstrates the robustness and efficacy of the algorithm for configuration estimation. Results show that the errors are in the range of 1°.

This paper describes a novel underactuated robotic hand design. The hand is highly underactuated as it contains three fingers with three joints each controlled by a single motor. One of the fingers (“thumb”) can also be rotated about the base of the hand, yielding a total of two controllable degrees-of-freedom. A key component of the design is the addition of position and tactile sensors which provide precise angle feedback and binary force feedback. Our mechanical design can be analyzed theoretically to predict contact forces as well as hand position given a particular object shape

Passively adaptive and underactuated robotic hands have shown the potential to achieve reliable grasping in unstructured environments without expensive mechanisms or sensors. Instead of complex run-time algorithms, such hands use design-time analysis to improve performance for a wide range of tasks. Along these directions, we present an optimization framework for underactuated compliant hands. Our approach uses a pre-defined set of grasps in a quasistatic equilibrium formulation to compute the actuation mechanism design parameters that provide optimal performance. We apply our method to a class of tendon-actuated hands; for the simplified design of a two-fingered gripper, we show how a global optimum for the design optimization problem can be computed. We have implemented the results of this analysis in the construction of a gripper prototype, capable of a wide range of grasping tasks over a variety of objects.

This paper presents the task specifications for designing a novel Insertable Robotic Effectors Platform (IREP) with integrated stereo vision and surgical intervention tools for Single Port Access Surgery (SPAS). This design provides a compact deployable mechanical architecture that may be inserted through a single Ø15 mm access port. Dexterous surgical intervention and stereo vision are achieved via the use of two snake-like continuum robots and two controllable CCD cameras. Simulations and dexterity evaluation of our proposed design are compared to several design alternatives with different kinematic arrangements. Results of these simulations show that dexterity is improved by using an independent revolute joint at the tip of a continuum robot instead of achieving distal rotation by transmission of rotation about the backbone of the continuum robot. Further, it is shown that designs with two robotic continuum robots as surgical arms have diminished dexterity if the bases of these arms are close to each other. This result justifies our design and points to ways of improving the performance of existing designs that use continuum robots as surgical arms.

We deal with the problem of teaching a robot to manipulate everyday objects through human demonstration. We first design a task descriptor which encapsulates important elements of a task. The design originates from observations that manipulations involved in many everyday object tasks can be considered as a series of sequential rotations and translations, which we call manipulation primitives. We then propose a method that enables a robot to decompose a demonstrated task into sequential manipulation primitives and construct a task descriptor. We also show how to transfer a task descriptor learned from one object to similar objects. In the end, we argue that this framework is highly generic. Particularly, it can be used to construct a robot task database that serves as a manipulation knowledge base for a robot to succeed in manipulating everyday objects.

In this paper we present an online algorithm for robustly tracking surgical tools in dynamic environments that can assist a surgeon during in-vivo robotic surgery procedures. The next generation of in-vivo robotic surgical devices includes integrated imaging and effector platforms that need to be controlled through real-time visual feedback. Our tracking algorithm learns the appearance of the tool online to account for appearance and perspective changes. In addition, the tracker uses multiple features working together to model the object and discover new areas of the tool as it moves quickly, exits and re-enters the scene, or becomes occluded and requires recovery. The algorithm can persist through changes in lighting and pose by using a memory database, which is built online, using a series of features working together to exploit different aspects of the object being tracked. We present results using real in-vivo imaging data from a human partial nephrectomy.

Collecting grasp data for learning and benchmarking purposes is very expensive. It would be helpful to have a standard database of graspable objects, along with a set of stable grasps for each object, but no such database exists. In this work we show how to automate the construction of a database consisting of several hands, thousands of objects, and hundreds of thousands of grasps. Using this database, we demonstrate a novel grasp planning algorithm that exploits geometric similarity between a 3D model and the objects in the database to synthesize form closure grasps. Our contributions are this algorithm, and the database itself, which we are releasing to the community as a tool for both grasp planning and benchmarking.

Highly underactuated and passively adaptive robotic hands have shown great promise for robust performance in unstructured settings. In order to fully realize this potential, efficient tools are needed to analyze the execution of a grasp when using this class of devices. Along this line, this paper introduces a quasistatic analysis method for underactuated hands. First, we predict whether initial contacts between the fingers and the object are stable throughout the execution of a grasp, or the fingers will slip as the hand closes. Second, we compute the unbalanced forces applied to the object during the grasping process. Finally, once the grasp is complete, we analyze its stability as actuator forces are increased. These computations are performed in 3D, allow arbitrary kinematic structure of the fingers or geometry of the target object and take into account frictional constraints. We discuss applications of this method focusing on both on-line computation to execute a specific grasping task and off-line optimization to increase the range of grasps that can be performed using a given hand model.

To grasp a novel object, we can index it into a database of known 3D models and use precomputed grasp data for those models to suggest a new grasp. We refer to this idea as data-driven grasping, and we have previously introduced the Columbia Grasp Database for this purpose. In this paper we demonstrate a data-driven grasp planner that requires only partial 3D data of an object in order to grasp it. To achieve this, we introduce a new shape descriptor for partial 3D range data, along with an alignment method that can rigidly register partial 3D models to models that are globally similar but not identical. Our method uses SIFT features of depth images, and encapsulates ¿nearby¿ views of an object in a compact shape descriptor.

This paper presents a novel design and preliminary kinematic analysis of an insertable robotic effector platform (IREP) for single port access (SPA) surgery. The IREP robot can be deployed into body cavity through a Ø15 mm skin incision to perform SPA procedures. It consists of two snake-like continuum robots as slave surgical assistants for tissue manipulation, two parallelogram mechanisms for the continuum robots' placement, and one controllable stereo vision module with integrated light source for depth perception and tool tracking. Design considerations and alternatives, calculations and preliminary simulations of this 17-DoF surgical robotic system are presented in this paper. The overall control system hierarchy for tele-manipulation using the IREP robot is also presented.

This paper describes work we have done in developing an insertable surgical imaging device with multiple degrees-of-freedom for minimally invasive surgery. The device is fully insertable into the abdomen using standard 12 mm trocars. It consists of a modular camera and lens system which has pan and tilt capability provided by 2 small DC servo motors. It also has its own integrated lighting system that is part of the camera assembly. Once the camera is inserted into the abdomen, the insertion port is available for additional tooling, motivating the idea of single port surgery. A third zoom axis has been designed for the camera as well, allowing close-up and far-away imaging of surgical sites with a single camera unit. In animal tests with the device we have performed surgical procedures including cholecystectomy, appendectomy, running (measuring) the bowel, suturing, and nephrectomy. The tests show that the new device is: (1) Easier and more intuitive to use than a standard laparoscope. (2) Joystick operation requires no specialized operator training. (3) Field of view and access to relevant regions of the body were superior to a standard laparoscope using a single port. (4) Time to perform procedures was better or equivalent to a standard laparoscope. We believe these insertable platforms will be an integral part of future surgical systems. The platforms can be used with tooling as well as imaging systems, allowing many surgical procedures to be done using such a platform.

In this paper we outline a grasp planning system designed to augment the cortical control of a prosthetic arm and hand. A key aspect of this system it the ability to combine online user input and autonomous planning to enable the execution of stable grasping tasks. While user input can ultimately be of any modality, the system is being designed to adapt to partial or noisy information obtained from grasp-related activity in the primate motor cortex. First, principal component analysis is applied to the observed kinematics of physiologic grasping to reduce the dimensionality of hand posture space and simplify the planning task for on-line use. The planner then accepts control input in this reduced-dimensionality space, and uses it as a seed for a hand posture optimization algorithm based on simulated annealing. We present two applications of this algorithm, using data collected from both primate and human subjects during grasping, to demonstrate its ability to synthesize stable grasps using partial control input in real or near-real time.

Text search on databases of 3D models has traditionally worked poorly, as text annotations on 3D models are often unreliable or incomplete. We attempt to improve the recall of text search by automatically assigning appropriate tags to models. Our algorithm finds relevant tags by appealing to a large corpus of partially labeled example models, which does not have to be preclassified or otherwise prepared. For this purpose we use a copy of Google 3D Warehouse, a database of user contributed models which is publicly available on the Internet. Given a model to tag, we find geometrically similar models in the corpus, based on distances in a reduced dimensional space derived from Zernike descriptors. The labels of these neighbors are used as tag candidates for the model with probabilities proportional to the degree of geometric similarity. We show experimentally that text based search for 3D models using our computed tags can approach the quality of geometry based search.

Training a 3D model classifier on a small dataset is very challenging. However, large datasets of partially classified models are now commonly available online. We use an external training set of models with associated text tags to automatically assign tags to both training and query models. The similarity between these tags, used in conjunction with a standard shape descriptor, yields a multiclassifier that outperforms the standalone shape descriptor.

Planning realizable and stable grasps on 3D objects is crucial for many robotics applications, but grasp planners often ignore the relative sizes of the robotic hand and the object being grasped or do not account for physical joint and positioning limitations. We present a grasp planner that can consider the full range of parameters of a real hand and an arbitrary object, including physical and material properties as well as environmental obstacles and forces, and produce an output grasp that can be immediately executed. We do this by decomposing a 3D model into a superquadric 'decomposition tree' which we use to prune the intractably large space of possible grasps into a subspace that is likely to contain many good grasps. This subspace can be sampled and evaluated in GraspIt!, our 3D grasping simulator, to find a set of highly stable grasps, all of which are physically realizable. We show grasp results on various models using a Barrett hand.

In this paper we address the joint problems of automated data acquisition and view planning for large-scale indoor and outdoor sites. Our method proceeds in two distinct stages. In the initial stage, the system is given a 2-D map with which it plans a minimal set of sufficient covering views. We then use a 3-D laser scanner to take scans at each of these views. When this planning system is combined with our mobile robot, it automatically computes and executes a tour of these viewing locations and acquires the views with the robot's onboard laser scanner. These initial scans serve as an approximate 3-D model of the site. The planning software then enters a second stage in which it updates this model by using a voxel-based occupancy procedure to plan the next best view. This next best view is acquired, and further next best views are sequentially computed and acquired until a complete 3-D model is obtained. Results are shown for Fort Jay on Governors Island in the City of New York and for the church of Saint Menoux in the Bourbonnais region of France.

In this paper, we build upon recent advances in neuroscience research which have shown that control of the human hand during grasping is dominated by movement in a configuration space of highly reduced dimensionality. We extend this concept to robotic hands and show how a similar dimensionality reduction can be defined for a number of different hand models. This framework can be used to derive planning algorithms that produce stable grasps even for highly complex hand designs. Furthermore, it offers a unified approach for controlling different hands, even if the kinematic structures of the models are significantly different. We illustrate these concepts by building a comprehensive grasp planner that can be used on a large variety of robotic hands under various constraints.

Endoscopic imaging is still dominated by the paradigm of pushing long sticks into small openings. This approach has a number of limitations for minimal access surgery, such as narrow angle imaging, limited workspace, counter-intuitive motions and additional incisions for the endoscpic instruments. Our intent is to go beyond this paradigm, and remotize sensors and effectors directly into the body cavity. To this end, we have developed a prototype of a novel insertable pan/tilt endoscopic camera with an integrated light source. The package has a size of 110 mm in length and 10 mm in diameter and can be inserted into the abdomen through a standard trocar and then anchored onto the abdominal wall, leaving the incision port open for access. The camera package contains three parts: an imaging module, an illumination module, and a pan/tilt motion platform. The imaging module includes a lens and CCD imaging sensor. The illumination module attaches to the imaging module and has an array of LED light sources. The pan/tilt platform provides the imaging module with pan of 120 degrees and tilt motion of 90 degrees using small servo motors. A fixing mechanism is designed to hold the device in the cavity. A standard joy stick can be used to control the motion of the camera in a natural way. The design allows for multiple camera packages to be inserted through a single incision as well.

We present a systematic method for constructing 3-D models of large outdoor sites. The method is designed for a mobile robot platform and incorporates automated acquisition of scanned data as well as automated view planning and model construction. In our modeling process, we first use a preliminary view or set of preplanned views to yield an initial, approximate, 3-D model of the target structure. Then, we update this model by using a voxel-based procedure to plan and acquire the next best view. This updating is repeated sequentially until an accurate and complete 3-D model is finally obtained. The method was successfully tested on a portion of the Columbia University campus

The human hand is unrivaled in its ability to grasp and manipulate objects, but we still do not understand all of its complexities. One benefit it has over traditional robot hands is the fact that our fingers conform to a grasped object's shape, giving rise to larger contact areas and the ability to apply larger frictional forces. In this paper, we demonstrate how we have extended our simulation and analysis system with finite element modeling to allow us to evaluate these complex contact types. We also propose a new contact model that better accounts for the deformations and show how grasp quality is affected. This work is part of a larger project to understand the benefits the human hand has in grasping.

Finding appropriate stable grasps for a hand (either robotic or human) on an arbitrary object has proved to be a challenging and difficult problem. The space of grasping parameters coupled with the degrees-of-freedom and geometry of the object to be grasped creates a high-dimensional, non-smooth manifold. Traditional search methods applied to this manifold are typically not powerful enough to find appropriate stable grasping solutions, let alone optimal grasps. We address this issue in this paper, which attempts to find optimal grasps of objects using a grasping simulator. Our unique approach to the problem involves a combination of numerical methods to recover parts of the grasp quality surface with any robotic hand, and contemporary machine learning methods to interpolate that surface, in order to find the optimal grasp.

We present a system for protein crystal micromanipulation with focus on automated crystal mounting for the purposes of X-ray data collection. The system features a set of newly designed micropositioner end-effectors we call microshovels which address some limitations of the traditional cryogenic loops. We have used micro-electrical mechanical system (MEMS) techniques to design and manufacture various shapes and quantities of microshovels. Visual feedback from a camera mounted on the microscope is used to control the micropositioner as it lowers a microshovel into the liquid containing the crystals and approaches a selected crystal for pickup. We present experimental results that illustrate the applicability of our approach.

This paper addresses the problems of building a functional mobile robot for urban site navigation and modeling with focus on keeping track of the robot location. We have developed a localization system that employs two methods. The first method uses odometry, a compass and tilt sensor, and a global positioning sensor. An extended Kalman filter integrates the sensor data and keeps track of the uncertainty associated with it. The second method is based on camera pose estimation. It is used when the uncertainty from the first method becomes very large. The pose estimation is done by matching linear features in the image with a simple and compact environmental model. We have demonstrated the functionality of the robot and the localization methods with real-world experiments.

Preserving cultural heritage and historic sites is an important problem. These sites are subject to erosion, vandalism, and as long-lived artifacts, they have gone through many phases of construction, damage and repair. It is important to keep an accurate record of these sites using 3-D model building technology as they currently are, so preservationists can track changes, foresee structural problems, and allow a wider audience to "virtually" see and tour these sites. Due to the complexity of these sites, building 3-D models is time consuming and difficult, usually involving much manual effort. This paper discusses new methods that can reduce the time to build a model using automatic methods. Examples of these methods are shown in reconstructing a model of the Cathedral of Ste. Pierre in Beauvais, France.

We have demonstrated that a properly-equipped mobile robot can easily construct a detailed map of the wireless coverage of an urban environment. The Autonomous Vehicle for Exploration and Navigation of Urban Environments (AVENUE) mobile robot was successfully used to generate such maps in both manual and autonomous modes of operation. The resulting database contained a wealth of information for many different positions in the region, with a list of all access points viewable from each location together with a quality measure (the signal-to-noise ratio) of every detected signal. At a later time, the AVENUE system effectively used the data in this map to determine the approximate position of the robot as it traveled through the urban area.

Automatic grasp planning for robotic hands is a difficult problem because of the huge number of possible hand configurations. However, humans simplify the problem by choosing an appropriate prehensile posture appropriate for the object and task to be performed. By modeling an object as a set of shape primitives, such as spheres, cylinders, cones and boxes, we can use a set of rules to generate a set of grasp starting positions and pregrasp shapes that can then be tested on the object model. Each grasp is tested and evaluated within our grasping simulator "GraspIt!", and the best grasps are presented to the user. The simulator can also plan grasps in a complex environment involving obstacles and the reachability constraints of a robot arm.

We present a system for topologically localizing a mobile robot using color histogram matching of omnidirectional images. The system is intended for use as a navigational tool for the autonomous vehicle for exploration and navigation of urban environments (AVENUE) mobile robot. Our method makes use of omnidirectional images which are acquired from the robot's on-board camera. The method is fast and rotation invariant. Our tests have indicated that normalized color histograms are best for an outdoor environment while normalization is not required for indoor work. The system quickly narrows down the robot's location to one or two regions within the much larger test environment. Using this regional localization information, other vision systems that we have developed can further localize the robot.

This paper addresses the problem of mobile robot localization in urban environments. Typically, GPS is the preferred sensor for outdoor operation. However, using GPS-only localization methods leads to significant performance degradation in urban areas where nearby tall structures obstruct the clear view of the satellites. In our work, we use vision-based techniques to supplement GPS and odometry and provide accurate localization. The vision system identifies prominent linear features in the scene and matches them with a reduced model of nearby buildings, yielding improved pose estimation of the robot.

In this paper, we present a framework for cell manipulation tasks with visual servoing micromanipulation strategies. A vision based micropositioner is designed in order to address the requirement of high precision needed to perform manipulation of objects under 100 /spl mu/m in size. The system calibration (microscope-camera-micropositioner) and the model of the observed scene are not known. Experimental results for micropositioning tasks with respect to protein cells are presented and demonstrate the validity of the proposed approach.

We are building a system that can automatically acquire 3D range scans and 2D images to build geometrically correct, texture mapped 3D models of urban environments. This paper deals with the problem of automatically registering the 3D range scans with images acquired at other times and with unknown camera calibration and location. The method involves the utilization of parallelism and orthogonality constraints that naturally exist in urban environments. We present results for building a texture mapped 3-D model of an urban building.

Describes a synergistic integration of a grasping simulator and a real-time visual tracking system, that work in concert to (1) find an object's pose, (2) plan grasps and movement trajectories, and (3) visually monitor task execution. Starting with a CAD model of an object to be grasped, the system can find the object's pose through vision which then synchronizes the state of the robot workcell with an online, model-based grasp planning and visualization system we have developed called GraspIt. GraspIt can then plan a stable grasp for the object, and direct the robotic hand system to perform the grasp. It can also generate trajectories for the movement of the grasped object, which are used by the visual control system to monitor the task and compare the actual grasp and trajectory with the planned ones. We present experimental results using typical grasping tasks.

There are many design factors and choices when mounting a vision system for robot control. Such factors may include the kinematic and dynamic characteristics in the robot's degrees of freedom (DOF), which determine what velocities and fields-of-view a camera can achieve. Another factor is that additional motion components (such as pan-tilt units) are often mounted on a robot and introduce synchronization problems. When a task does not require visually servoing every robot DOF, the designer must choose which ones to servo. Questions then arise as to what roles, if any, do the remaining DOF play in the task. Without an analytical framework, the designer resorts to intuition and try-and-see implementations. This paper presents a frequency-based framework that identifies the parameters that factor into tracking. This framework gives design insight which was then used to synthesize a control law that exploits the kinematic and dynamic attributes of each DOF. The resulting multi-input multi-output control law, which we call partitioning, defines an underlying joint coupling to servo camera motions. The net effect is that by employing both visual and kinematic feedback loops, a robot can quickly position and orient a camera in a large assembly workcell. Real-time experiments tracking people and robot hands are presented using a 5-DOF hybrid (3-DOF Cartesian gantry plus 2-DOF pan-tilt unit) robot.

This paper deals with the automated creation of geometric and photometric correct 3-D models of the world. Those models can be used for virtual reality, tele-presence, digital cinematography and urban planning applications. The combination of range (dense depth estimates) and image sensing (color information) provides data-sets which allow us to create geometrically correct, photorealistic models of high quality. The 3-D models are first built from range data using a volumetric set intersection method previously developed by us. Photometry can be napped onto these models by registering features from both the 3-D and 2-D data sets. Range data segmentation algorithms have been developed to identify planar regions, determine linear features from planar intersections that can serve as features for registration with 2-D imagery lines, and reduce the overall complexity of the models. Results are shown for building models of large buildings on our campus using real data acquired from multiple sensors.

A distributed, modular, heterogeneous architecture is presented that illustrates an approach to solving and integrating common tasks in mobile robotics, such as path planning, localization, sensor fusion, environmental modeling, and motion control. Experimental results are shown for an autonomous navigation task to confirm the applicability of our approach.

Humans have degrees-of-freedom (DOF) of varying bandwidths and one casually observes that we coordinate these DOF while visually tracking. This suggests that joint interplay aids tracking performance. In a control scheme we call partitioning, both image and kinematic data are used to visually-servo a 5-DOF robot by defining a joint-coupling among the rotational and translational DOF. Analysis of simulations and experiments reveal that a robot's fast bandwidth joints physically serve as lead compensators when coupled to slower joints thus reducing tracking lag.

The automated extraction of photo-realistic 3D models of the world that can be used in applications such as virtual reality, tele-presence, digital cinematography and urban planning, is the focus of this paper. The combination of range (dense depth estimates) and image sensing (color information) provides data-sets which allow us to create photo-realistic models of high quality. The challenges are the simplification of the 3D data set, the extraction of meaningful features in both the range and 2D images and the fusion of those data-sets using the extracted features. We address all these challenges and provide results on data we gathered in outdoor scenes by a range and image sensor based on a mobile robot. Our ultimate goal is an autonomous 3D model creation system which minimizes the amount of human interaction.

The swept volume problem is practical, difficult and interesting enough to have received a great deal of attention over the years, and the literature contains much discussion of methods for computing swept volumes in many situations. The method presented here permits an arbitrary polyhedral object (given in a typical boundary representation) to be swept through an arbitrary trajectory. A polyhedral approximation to the volume swept by this moving object is computed and output in a typical boundary representation. A number of examples are presented demonstrating the practicality of this method. Copyright © 2000 John Wiley & Sons, Ltd.

We describe an automated scene modeling system that consists of two components operating in an interleaved fashion: an incremental modeler that builds solid models from range imagery; and a sensor planner that analyzes the resulting model and computes the next sensor position. This planning component is target-driven and computes sensor positions using model information about the imaged surfaces and the unexplored space in a scene. The method is shape-independent and uses a continuous-space representation that preserves the accuracy of sensed data. It is able to completely acquire a scene by repeatedly planning sensor positions, utilizing a partial model to determine volumes of visibility for contiguous areas of unexplored scene. These visibility volumes are combined with sensor placement constraints to compute sets of occlusion-free sensor positions that are guaranteed to improve the quality of the model. We show results for the acquisition of a scene that includes multiple, distinct objects with high occlusion.

Previous grasp quality research is mainly theoretical, and has assumed that contact types and positions are given, in order to preserve the generality of the proposed quality measures. The example results provided by these works either ignore hand geometry and kinematics entirely or involve only the simplest of grippers. We present a unique grasp analysis system that, when given a 3D object, hand, and pose for the hand, can accurately determine the types of contacts that will occur between the links of the hand and the object, and compute two measures of quality for the grasp. Using models of two articulated robotic hands, we analyze several grasps of a polyhedral model of a telephone handset, and we use a novel technique to visualize the 6D space used in these computations. In addition, we demonstrate the possibility of using this system for synthesizing high quality grasps by performing a search over a subset of possible hand configurations.

We present a novel approach we call partitioning where the robot's degrees-of-freedom (DOF) are categorized into two classes based on joint kinematics and dynamics to design a coupled multi-input control system. We use image data to visually servo the first class of joints that have quick response time. Position-based data is used to kinematically servo the second class of joints that have large kinematic range. The net effect is an active-vision system that synergistically tracks a diverse range of targets (without using CAD-based models) over a wide bandwidth of motion dynamics.

In this article we present a method for automatically constructing a solid (in the CAD sense) model of an unknown object from range images. The model acquisition system provides facilities for range image acquisition, solid model construction and model merging: a solid representation is derived from a mesh surface that models the range data from each view, which is then merged with the model built from previous sensing operations. This modeling system has the benefit of constructing a solid model at each phase of the acquisition process, and is able to model parts that are difficult or impossible using other methods, such as extremely thin parts or those with deep through-holes. We introduce a technique that utilizes the incomplete model resulting from the merging process to plan the next sensing operation by finding a sensor viewpoint that will improve the fidelity of the model. A number of examples are given for the modeling of various objects that include planar and curved surfaces, features such as through-holes, and large self-occlusions.

This paper describes an interactive sensor planning system, that can be used to select viewpoints subject to camera visibility, field of view and task constraints. Application areas for this method include surveillance planning, safety monitoring, architectural site design planning, and automated site modeling. Given a description, of the sensor's characteristics, the objects in the 3-D scene, and the targets to be viewed, our algorithms compute the set of admissible view points that satisfy the constraints. The system first builds topologically correct solid models of the scene from a variety of data sources. Viewing targets are then selected, and visibility volumes and field of view cones are computed and intersected to create viewing volumes where cameras can be placed. The user can interactively manipulate the scene and select multiple target features to be viewed by a camera. The user can also select candidate viewpoints within this volume to synthesize views and verify the correctness of the planning system. We present experimental results for the planning system on an actual complex city model.

Image-based servoing systems are often used to track moving targets and their underlying control architecture is a regulation of the image. This regulation is a function of rigid camera-to-target geometric constraints. Satisfying such constraints requires that the robot motors have sufficient velocity bandwidths, and often these bandwidths are limited. This paper lays down the foundation for a partitioned controller. Such a controller would coordinate a camera's DOF into a synergistic move to overcome bandwidth limitations. Tracking experiments are shown on a custom designed 5-DOF gantry robot which highlight the limitations of regulator-based control, as well as show how partitioning can be used to achieve more robust control.

We introduce two methods for the registration of range images when a prior estimate of the transformation between views is not available and the overlap between images is relatively small. The methods are an extension to the work of Gueziec and Ayache (1994) and Turk and Levoy (1994) and consists of 2 stages. First, we find the initial estimated transformation by extracting and matching 3D space curves from different scans of the same object. If no salient features are available on the object we use fiducial marks to find the initial transformation. This allows us to always find a satisfactory and even highly accurate transformation independent of the geometry of the object. Second, we apply a modified iterative closest points algorithm (ICP) to improve the accuracy of registration. We define a weighted distance function based on surface curvature which can reduce the number of iterations and requires a less accurate initial estimate of the transformation.

We present an incremental system that builds accurate CAD models of objects from multiple range images. Using a hybrid of surface mesh and volumetric representations, the system creates a "water-tight" 3D model at each step of the modeling process, allowing reasonable models to be built from a small number of views. We also present a method that can be used to plan the next view and reduce the number of scans needed to recover the object. Results are presented for the creation of 3D models of a computer game controller, a hip joint prosthesis, and a mechanical strut.

Most robotic hands are either sensorless or lack the ability to accurately and robustly report position and force information relating to contact. This paper describes a robotic hand system that uses a limited set of native-joint position and force sensing along with custom-designed tactile sensors and real-time vision modules to accurately compute finger contacts and applied forces for grasping tasks. Three experiments are described: integration of real-time visual trackers in conjunction with internal strain gauge sensing to correctly localize and compute finger forces, determination of contact points on the inner and outer links of a finger through tactile sensing and visual sensing, and determination of vertical displacement by tactile sensing for a grasping task.

This paper describes a robotic system that builds a 3D CAD model of an object incrementally from multiple range images. It motivates the generation of a solid model at each stage of the modeling process, allowing the use of well-defined geometric algorithms to perform the merging and integration task. The data from each imaging operation is represented by a mesh, which is then extruded in the viewing direction to form a solid model. These solids are merged as they are acquired into a composite model of the object. We describe an algorithm that builds a solid model from a mesh surface and present experimental results of reconstructing a complex object. In addition, we discuss an approach to completely automating the model acquisition process by integration with previous sensor-planning results.

Automatically planning a camera viewpoint for tasks such as inspection in an active robot work-cell is a difficult problem. This paper discusses new methods for computing viewpoints which meet the feature detectability constraints of focus, field-of-view, visibility, and resolution. A theoretical outline of the method is presented, followed by experimental results and a discussion of future work.

This paper discusses an approach to automating CAD model acquisition by creating binary space partitioning (BSP) trees from laser range data. Distinct views of the object, each represented by a BSP tree, are then merged using set operations. BSP trees have proven their utility in 3D modeling, graphics and image processing, and their tree structure allows efficient algorithms to be developed that are compact and numerically robust. These properties are of primary importance when considering an intermediate representation between raw sensor data and existing CAD models.

A survey of research in the area of vision sensor planning is presented. The problem can be summarized as follows: given information about the environment as well as information about the task that the vision system is to accomplish, develop strategies to automatically determine sensor parameter values that achieve this task with a certain degree of satisfaction. With such strategies, sensor parameters values can be selected and can be purposefully changed in order to effectively perform the task at hand. The focus here is on vision sensor planning for the task of robustly detecting object features. For this task, camera and illumination parameters such as position, orientation, and optical settings are determined so that object features are, for example, visible, in focus, within the sensor field of view, magnified as required, and imaged with sufficient contrast. References to, and a brief description of, representative sensing strategies for the tasks of object recognition and scene reconstruction are also presented. For these tasks, sensor configurations are sought that will prove most useful when trying to identify an object or reconstruct a scene.>

The MVP (machine vision planner) model-based sensor planning system for robotic vision is presented. MVP automatically synthesizes desirable camera views of a scene based on geometric models of the environment, optical models of the vision sensors, and models of the task to be achieved. The generic task of feature detectability has been chosen since it is applicable to many robot-controlled vision systems. For such a task, features of interest in the environment are required to simultaneously be visible, inside the field of view, in focus, and magnified as required. In this paper, we present a technique that poses the vision sensor planning problem in an optimization setting and determines viewpoints that satisfy all previous requirements simultaneously and with a margin. In addition, we present experimental results of this technique when applied to a robotic vision system that consists of a camera mounted on a robot manipulator in a hand-eye configuration.>

We describe a method for the visual control of a robotic system which does not require the formulation of an explicit calibration between image coordinates and the world coordinates. By extracting control information directly from the image, we free our technique from the errors normally associated with a fixed calibration. We attach a camera system to a robot such that the camera system and the robot's gripper rotate simultaneously. As the camera system rotates about the gripper's rotational axis, the circular path traced out by a point-like feature projects to an elliptical path in image space. We gather the projected feature points over part of a rotation and fit the gathered data to an ellipse. The distance from the rotational axis to the feature point in world space is proportional to the size of the generated ellipse. As the rotational axis gets closer to the feature, the feature's projected path will form smaller and smaller ellipses. When the rotational axis is directly above the object, the trajectory degenerates from an ellipse to a single point. We demonstrate the efficacy of the algorithm on the peg-in-hole problem.>

This paper discusses the implementation of complex manipulation tasks with a dextrous hand. The approach used is to build a set of primitive manipulation functions and combine them to form complex tasks. Only fingertip, or precision, manipulations are considered. Each function performs a simple two-dimensional translation or rotation that can be generalized to work with objects of different sizes and using different grasping forces. Complex tasks are sequential combinations of the primitive functions. They are formed by analyzing the workspaces of the individual tasks and controlled by finite state machines. We present a number of examples, including a complex manipulation removing the top of a child-proof medicine bottle-that incorporates different hybrid position/force specifications of the primitive functions of which it is composed. The work has been implemented with a robot hand system using a Utah-MIT hand.>

This paper proposes a path-planning method for mobile robots in the presence of uncertainty. We analyze environment and control uncertainty and propose methods for incorporating each of them into the planning algorithm. We model the environment using the pyramid structure that encodes the information on occupancy probabilities for each pixel as well as the partial information on conditional probabilities among different pixels. This structure allows for efficient and accurate computation of collision probabilities in the presence of environment uncertainty. The control uncertainty is mainly characterized by its expansion in space and time and is accordingly modeled by a stochastic differential equation that mathematically captures this phenomenon. Models that we develop are inevitably approximate but experiments confirm that they can be used as a reasonable model for motion planning. We have conducted a series of experiments on the mobile platform and some of these results are presented.>

Proposes a method for visual control of a robotic system which does not require the formulation of an explicit calibration between image space and the world coordinate system. Calibration is known to be a difficult and error prone process. By extracting control information directly from the image, the authors free their technique from the errors normally associated with a fixed calibration. The authors demonstrate this by performing a peg-in-hole alignment using an uncalibrated camera to control the positioning of the peg. The algorithm utilizes feedback from a simple geometric effect, rotational invariance, to control the positioning servo loop. The method uses an approximation to the image Jacobian to provide smooth, near-continuous control.>

This paper considers adding autonomy to robot hands used in teleoperation systems. Currently, the finger positions of robot hands in teleoperation systems are controlled via a robot master using a Dataglove or exoskeleton. There are several difficulties with this approach: accurate calibration is hard to achieve; robot hands have different capabilities from human hands; and complex force reflection is difficult. In this paper we propose a model of hand teleoperation in which the input device commands the motions of a grasped object rather than the joint displacements of the fingers. To achieve this goal, the hand requires greater autonomy and the capability to perform high-level functions with minimal external input. Therefore, a set of general, primitive manipulation functions that can be performed automatically is defined. These elementary functions control simple rotations and translations of the grasped objects. They are incorporated into a teleoperation system by using a simple input device as a control signal. Preliminary implementations with a Utah/MIT are discussed.>

An attempt to achieve a high level of interaction between a real-time vision system capable of tracking moving objects in 3-D and a robot arm with gripper that can be used to pick up a moving object is described. The interplay of hand-eye coordination in dynamic grasping tasks such as grasping of parts on a moving conveyor system, assembly of articulated parts, or for grasping from a mobile robotic system is explored. The goal is to build an integrated sensing and actuation system that can operate in dynamic as opposed to static environments. The system built addresses three distinct problems in using robotic hand-eye coordination for grasping moving objects: fast computation of 3-D motion parameters from vision, predictive control of a moving robotic arm to track a moving object, and interception and grasping. The system operates at approximately human arm movement rates. Experimental results in which a moving model train is tracked, stably grasped, and picked up by the system are presented. The algorithms developed to relate sensing to actuation are quite general and applicable to a variety of complex robotic tasks.>

The authors explore the requirements for grasping a moving object. This task requires proper coordination between at least three separate subsystems: real-time vision sensing, trajectory-planning/arm-control, and grasp planning. As with humans, the system first visually tracks the object's 3D position. Because the object is in motion, this must be done in real-time to coordinate the motion of the robotic arm as it tracks the object. The vision system is used to feed an arm control algorithm that plans a trajectory. The arm control algorithm is implemented into two steps: filtering and prediction and kinematic transformation computation. Once the trajectory of the object is tracked, the hand must intercept the object to actually grasp it. Experimental results are presented in which which a moving model train was tracked, stably grasped, and picked up by the system.>

A real-time tracking algorithm in conjunction with a predictive filter to allow real-time visual servoing of a robotic arm that is tracking a moving object is described. The system consists of two calibrated (but unregistered) cameras that provide images to a real-time, pipeline-parallel optic-flow algorithm that can robustly compute optic-flow and calculate the 3-D position of a moving object at approximately 5-Hz rates. These 3-D positions of the moving object serve as input to a predictive kinematic control algorithm that uses an alpha - beta - gamma filter to update the position of a robotic arm tracking the moving object. Experimental results are presented for the tracking of a moving model train in a variety of different trajectories.>

Research in human haptics has revealed a number of exploratory procedures (EPs) that are used in determining attributes on an object, particularly shape. This research has been used as a paradigm for building an intelligent robotic system that can perform shape recognition from touch sensing. In particular, a number of mappings between EPs and shape modeling primitives have been found. The choice of shape primitive for each EP is discussed, and results from experiments with a Utah-MIT dextrous hand system are presented. A vision algorithm to complement active touch sensing for the task of autonomous shape recovery is also presented.>

The use of touch sensing as part of a large system being built for 3D shape recovery and object recognition using touch and vision methods is described. The authors focus on three exploratory procedures they have devised to acquire and interpret sparse 3D touch data: grasping by containment, planar surface exploration, and surface contour exploration. Experimental results for each of these procedures are presented.>

A system for programming and controlling a multisensor robotic hand (Utah-MIT Hand) is described. Using this system, a number of autonomous tasks that are easily programmed and include combinations of hand-arm actuation with force, position, and tactile sensing have been implemented. The system is controlled at the software level by a programming language DIAL that provides an easy method for expressing the parallel operation of robotic devices. It also provides a convenient way to implement task-level scripts that can then be bound to particular sensors, actuators, and methods for accomplishing a generic grasping or manipulation task. Experiments using the system to pick up and pour from a pitcher, unscrew a lightbulb, and explore planar surfaces are presented.>

The authors describe an integrated system for dextrous manipulation using a Utah-MIT hand that makes it possible to look at the higher levels of control in a number of grasping and manipulation tasks. The system consists of a number of low-level system primitives for integrated hand and robotic arm movement, tactile sensors mounted on the fingertips, sensing primitives to utilize joint position, tendon force and tactile array feedback, and a high-level programming environment that allows task level scripts to be created for grasping and manipulation tasks are described that have been implemented with this system.>

The use of a dextrous, multifingered hand for high-level object recognition tasks is considered. The paradigm is model-based recognition in which the objects are modeled and recovered as superquadratics, which are shown to have a number of important attributes that make them well suited for such a task. Experiments have been performed to recover the shape of objects using sparse contacts point data from the hand with promising results. The authors also propose an approach to using tactile data in conjunction with the dextrous hand to build a library of grasping and exploration primitives that can be used in recognizing and grasping more complex multipart objects.>