In this paper, we introduce a sensor designed for robotic fingers which can provide information on the displacements induced by external forces. Our sensor uses LEDs to sense the displacement between two plates connected by a transparent elastomer; when a force is applied to the finger, the elastomer displaces and the LED signals change. We show that using LEDs as both light emitters and receivers in this context provides high sensitivity, allowing such an emitter and receiver pairs to detect very small displacements. We characterize the standalone performance of the sensor by testing the ability of a supervised learning model to predict complete force and torque data from its raw signals, and obtain a mean error between 0.05 and 0.07 N across the three directions of force applied to the finger. Our method allows for compact packaging (fitting at the base of a finger) with no amplification electronics, low cost manufacturing, easy integration into a complete hand, and high overload shear forces and bending torques, suggesting future applicability to complete manipulation tasks.

The acoustic response of an object can reveal a lot about its global state, for example its material properties or the extrinsic contacts it is making with the world. In this work, we build an active acoustic sensing gripper equipped with two piezoelectric fingers: one for generating signals, the other for receiving them. By sending an acoustic vibration from one finger to the other through an object, we gain insight into an object’s acoustic properties and contact state. We use this system to classify objects, estimate grasping position, estimate poses of internal structures, and classify the types of extrinsic contacts an object is making with the environment. Using our contact type classification model, we tackle a standard long-horizon manipulation problem: peg insertion. We use a simple simulated transition model based on the performance of our sensor to train an imitation learning policy that is robust to imperfect predictions from the classifier. We finally demonstrate the policy on a UR5 robot with active acoustic sensing as the only feedback. Videos can be found at https://roamlab.github.io/vibecheck.

In this work, we use MEMS microphones as vibration sensors to simultaneously classify texture and estimate contact position and velocity. Vibration sensors are an important facet of both human and robotic tactile sensing, providing fast detection of contact and onset of slip. Microphones are an attractive option for implementing vibration sensing as they offer a fast response and can be sampled quickly, are affordable, and occupy a very small footprint. Our prototype sensor uses only a sparse array (8-9 mm spacing) of distributed MEMS microphones (<$1, 3.76×2.95×1.10 mm) embedded under an elastomer. We use transformer-based architectures for data analysis, taking advantage of the microphones’ high sampling rate to run our models on time-series data as opposed to individual snapshots. This approach allows us to obtain 77.3% average accuracy on 4-class texture classification (84.2% when excluding the slowest drag velocity), 1.8 mm mean error on contact localization, and 5.6 mm/s mean error on contact velocity. We show that the learned texture and localization models are robust to varying velocity and generalize to unseen velocities. We also report that our sensor provides fast contact detection, an important advantage of fast transducers. This investigation illustrates the capabilities one can achieve with a MEMS microphone array alone, leaving valuable sensor real estate available for integration with complementary tactile sensing modalities.

We introduce MORPH, a method for co-optimization of hardware design parameters and control policies in simulation using reinforcement learning. Like most co-optimization methods, MORPH relies on a model of the hardware being optimized, usually simulated based on the laws of physics. However, such a model is often difficult to integrate into an effective optimization routine. To address this, we introduce a proxy hardware model, which is always differentiable and enables efficient co-optimization alongside a long-horizon control policy using RL. MORPH is designed to ensure that the optimized hardware proxy remains as close as possible to its realistic counterpart, while still enabling task completion. We demonstrate our approach on simulated 2D reaching and 3D multi-fingered manipulation tasks.

Recently, reinforcement learning has led to dexterous manipulation skills of increasing complexity. Nonetheless, learning these skills in simulation still exhibits poor sample-efficiency which stems from the fact these skills are learned from scratch without the benefit of any domain expertise. In this work, we aim to improve the sample efficiency of learning dexterous in-hand manipulation skills using controllers available via domain knowledge. To this end, we design simple sub-skill controllers and demonstrate improved sample efficiency using a framework that guides exploration toward relevant state space by following actions from these controllers. We are the first to demonstrate learning hard-to-explore finger-gaiting in-hand manipulation skills without the use of an exploratory reset distribution.

In a Human-in-the-Loop paradigm, a robotic agent is able to act mostly autonomously in solving a task, but can request help from an external expert when needed. However, knowing when to request such assistance is critical: too few requests can lead to the robot making mistakes, but too many requests can overload the expert. In this paper, we present a Reinforcement Learning based approach to this problem, where a semi-autonomous agent asks for external assistance when it has low confidence in the eventual success of the task. The confidence level is computed by estimating the variance of the return from the current state. We show that this estimate can be iteratively improved during training using a Bellman-like recursion. On discrete navigation problems with both fully-and partially-observable state information, we show that our method makes effective use of a limited budget of expert calls at run-time, despite having no access to the expert at training time.

Underactuated manipulators reduce the number of bulky motors, thereby enabling compact and mechanically robust designs. However, fewer actuators than joints means that the manipulator can only access a specific manifold within the joint space, which is particular to a given hardware configuration and can be low-dimensional and/or discontinuous. Determining an appropriate set of hardware parameters for this class of mechanisms, therefore, is difficult - even for traditional task-based co-optimization methods. In this paper, our goal is to implement a task-based design and policy co-optimization method for underactuated, tendon-driven manipulators. We first formulate a general model for an underactuated, tendon-driven transmission. We then use this model to co-optimize a three-link, two-actuator kinematic chain using reinforcement learning. We demonstrate that our optimized tendon transmission and control policy can be transferred reliably to physical hardware with real-world reaching experiments.

We propose MetaEMG, a meta-learning approach for fast adaptation in intent inferral on a robotic hand orthosis for stroke. One key challenge in machine learning for assistive and rehabilitative robotics with disabled-bodied subjects is the difficulty of collecting labeled training data. Muscle tone and spasticity often vary significantly among stroke subjects, and hand function can even change across different use sessions of the device for the same subject. We investigate the use of meta-learning to mitigate the burden of data collection needed to adapt high-capacity neural networks to a new session or subject. Our experiments on real clinical data collected from five stroke subjects show that MetaEMG can improve the intent inferral accuracy with a small session- or subject-specific dataset and very few fine-tuning epochs. To the best of our knowledge, we are the first to formulate intent inferral on stroke subjects as a meta-learning problem and demonstrate fast adaptation to a new session or subject for controlling a robotic hand orthosis with EMG signals.

Tactile recognition of 3D objects remains a challenging task. Compared to 2D shapes, the complex geometry of 3D surfaces requires richer tactile signals, more dexterous actions, and more advanced encoding techniques. In this work, we propose TANDEM3D, a method that applies a co-training framework for exploration and decision making to 3D object recognition with tactile signals. Starting with our previous work, which introduced a co-training paradigm for 2D recognition problems, we introduce a number of advances that enable us to scale up to 3D. TANDEM3D is based on a novel encoder that builds 3D object representation from contact positions and normals using PointNet++. Furthermore, by enabling 6DOF movement, TANDEM3D explores and collects discriminative touch information with high efficiency. Our method is trained entirely in simulation and validated with real-world experiments. Compared to state-of-the-art baselines, TANDEM3D achieves higher accuracy and a lower number of actions in recognizing 3D objects and is also shown to be more robust to different types and amounts of sensor noise.

Controlling robotic manipulators with high-dimensional action spaces for dexterous tasks is a challenging problem. Inspired by human manipulation, researchers have studied generating and using postural synergies for robot hands to accomplish manipulation tasks, leveraging the lower dimensional nature of synergistic action spaces. However, many of these works require pre-collected data from an existing controller in order to derive such a subspace by means of dimensionality reduction. In this paper, we present a framework that simultaneously discovers both a synergy space and a multi-task policy that operates on this low-dimensional action space to accomplish diverse manipulation tasks. We demonstrate that our end-to-end method is able to perform multiple tasks using few synergies, and outperforms sequential methods that apply dimensionality reduction to independently collected data. We also show that deriving synergies using multiple tasks can lead to a subspace that enables robots to efficiently learn new manipulation tasks and interactions with new objects.

Finger-gaiting manipulation is an important skill to achieve large-angle in-hand re-orientation of objects. However, achieving these gaits with arbitrary orientations of the hand is challenging due to the unstable nature of the task. In this work, we use model-free reinforcement learning (RL) to learn finger-gaiting only via precision grasps and demonstrate finger-gaiting for rotation about an axis using only on-board proprioceptive and tactile feedback. To tackle the inherent instability of precision grasping, we propose the use of initial state distributions that enable effective exploration of the state space. Our method can learn finger gaiting with better sample complexity than the state-of-the-art. The policies we obtain are robust to noise and perturbations, and transfer to novel objects. Videos can be found at https://roamlab.github.io/learnfg/

In order to provide therapy in a functional context, controls for wearable robotic orthoses need to be robust and intuitive. We have previously introduced an intuitive, user-driven, EMG-based method to operate a robotic hand orthosis, but the process of training a control that is robust to concept drift (changes in the input signal) places a substantial burden on the user. In this paper, we explore semi-supervised learning as a paradigm for controlling a powered hand orthosis for stroke subjects. To the best of our knowledge, this is the first use of semi-supervised learning for an orthotic application. Specifically, we propose a disagreement-based semi-supervision algorithm for handling intrasession concept drift based on multimodal ipsilateral sensing. We evaluate the performance of our algorithm on data collected from five stroke subjects. Our results show that the proposed algorithm helps the device adapt to intrasession drift using unlabeled data and reduces the training burden placed on the user. We also validate the feasibility of our proposed algorithm with a functional task; in these experiments, two subjects successfully completed multiple instances of a pick-and-handover task.

Teleoperation is a valuable tool for robotic manipulators in highly unstructured environments. However, finding an intuitive mapping between a human hand and a nonanthropomorphic robot hand can be difficult, due to the hands’ dissimilar kinematics. In this article, we seek to create a mapping between the human hand and a fully actuated, nonanthropomorphic robot hand that is intuitive enough to enable effective real-time teleoperation, even for novice users. To accomplish this, we propose a low-dimensional teleoperation subspace that can be used as an intermediary for mapping between hand pose spaces. We present two different methods to define the teleoperation subspace: an empirical definition, which requires a person to define hand motions in an intuitive, hand-specific way, and an algorithmic definition, which is kinematically independent and uses objects to define the subspace. We use each of these definitions to create a teleoperation mapping for different hands. One of the main contributions of this article is the validation of both the empirical and algorithmic mappings with teleoperation experiments controlled by ten novices and performed on two kinematically distinct hands. The experiments show that the proposed subspace is relevant to teleoperation, intuitive enough to enable control by novices, and can generalize to nonanthropomorphic hands with different kinematics.Note to Practitioners—As robots move into our warehouses, workplaces, and homes, it is important to develop robotic controls that are intuitive and easy for novices to use. In particular, teleoperation can be valuable to guide robots when they encounter situations that autonomous programs are not prepared to deal with. In this article, we focus specifically on robotic grasping using nonanthropomorphic hands. Our method is intended for novice users to intuitively teleoperate such robots. We show that the teleoperation subspace we use can effectively enable pick-and-place tasks and in-hand manipulation tasks and that it is intuitive for novice operators. Our subspace outperforms state-of-the-art methods for pick-and-place tasks and performs as well as state-of-the-art methods for in-hand manipulation.

In this paper, we focus on a rarely used paradigm in the design of underactuated robot hands: the use of springs as agonists and tendons as antagonists. We formalize this approach in a design matrix also considering its interplay with the underactuation method used (one tendon for multiple joints vs. multiple tendons on one motor shaft). We then show how different cells in this design matrix can be combined in order to facilitate the implementation of desired postural synergies with a single motor. Furthermore, we show that when agonist and antagonist tendons are combined on the same motor shaft, the resulting spring force cancellation can be leveraged to produce multiple desirable behaviors, which we demonstrate in a physical prototype.

In this paper, we propose a method for generating undulatory gaits for snake robots. Instead of starting from a pre-defined movement pattern such as a serpenoid curve, we use a Model Predictive Control (MPC) approach to automatically generate effective locomotion gaits via trajectory optimization. An important advantage of this approach is that the resulting gaits are automatically adapted to the environment that is being modeled as part of the snake dynamics. To illustrate this, we use a novel model for anisotropic dry friction, along with existing models for viscous friction and fluid dynamic effects such as drag and added mass. For each of these models, gaits generated without any change in the method or its parameters are as efficient as Pareto-optimal serpenoid gaits tuned individually for each environment. Furthermore, the proposed method can also produce more complex or irregular gaits, e.g. for obstacle avoidance or executing sharp turns.

Grasp synergies represent a useful idea to reduce grasping complexity without compromising versatility. Synergies describe coordination patterns between joints, either in terms of position (joint angles) or effort (joint torques). In both of these cases, a grasp synergy can be represented as a low-dimensional manifold lying in the high-dimensional joint posture or torque space. In this article, we use the term mechanically realizable manifolds to refer to the subset of such manifolds (in either posture or torque space) that can be achieved via mechanical coupling of the joints in underactuated hands. We present a method to optimize the design parameters of an underactuated hand in order to shape the mechanically realizable manifolds to fit a predefined set of desired grasps. Our method guarantees that the resulting synergies can be physically implemented in an underactuated hand, and will enable the resulting hand to both reach the desired grasp postures and achieve quasi-static equilibrium while loading the grasps. We demonstrate this method on three concrete design examples motivated by a real use case, and evaluate and compare their performance in practice.

Passive reaction effects in grasp stability analysis occur when the contact forces and joint torques applied by a grasp change in response to external disturbances applied to the grasped object. For example, nonbackdrivable actuators (e.g., highly geared servos) will passively resist external disturbances without an actively applied command; for numerous robot hands using such motors, these effects can be highly beneficial as they increase grasp resistance without requiring active control. In this article, we introduce a grasp stability analysis method that can model these effects, and, for a given grasp, distinguish between disturbances that will be passively resisted and those that will not. We find that, in order to achieve this, the grasp model must include accurate energetic constraints. One way to achieve this is to consider the maximum dissipation principle (MDP), a part of the Coulomb friction model that is rarely used in grasp stability analysis. However, the MDP constraints are nonconvex, and difficult to solve efficiently. We thus introduce a convex relaxation method, along with an algorithm that successively refines this relaxation locally in order to obtain solutions to arbitrary accuracy efficiently. Our resulting algorithm can determine if a grasp is passively stable, solve for equilibrium contact forces and compute optimal actuator commands for stability. Its implementation is publicly available as part of the open-source GraspIt! simulator.

We present a method for EMG-driven teleoperation of non-anthropomorphic robot hands. EMG sensors are appealing as a wearable, inexpensive, and unobtrusive way to gather information about the teleoperator's hand pose. However, mapping from EMG signals to the pose space of a non-anthropomorphic hand presents multiple challenges. We present a method that first projects from forearm EMG into a subspace relevant to teleoperation. To increase robustness, we use a model which combines continuous and discrete predictors along different dimensions of this subspace. We then project from the teleoperation subspace into the pose space of the robot hand. Our method is effective and intuitive, as it enables novice users to teleoperate pick and place tasks faster and more robustly than state-of-the-art EMG teleoperation methods when applied to a non-anthropomorphic, multi-DOF robot hand.

Hand synergies, or joint coordination patterns, have become an effective tool for achieving versatile robotic grasping with simple hands or planning algorithms. Here we propose a method to determine the hand synergies such that they can be physically implemented in an underactuated fashion. Given a kinematic hand model and a set of desired grasps, our algorithm optimizes a Mechanically Realizable Manifold designed to be achievable by a physical underactuation mechanism, enabling the resulting hand to achieve the desired grasps with few actuators. Furthermore, in contrast to existing methods for determining synergies which are only concerned with hand posture, our method explicitly optimizes the stability of the target grasps. We implement this method in the design of a three-finger single-actuator hand as an example, and evaluate its effectiveness numerically and experimentally.

Human-in-the-loop manipulation is useful in when autonomous grasping is not able to deal sufficiently well with corner cases or cannot operate fast enough. Using the teleoperator's hand as an input device can provide an intuitive control method but requires mapping between pose spaces which may not be similar. We propose a low-dimensional and continuous teleoperation subspace which can be used as an intermediary for mapping between different hand pose spaces. We present an algorithm to project between pose space and teleoperation subspace. We use a non-anthropomorphic robot to experimentally prove that it is possible for teleoperation subspaces to effectively and intuitively enable teleoperation. In experiments, novice users completed pick and place tasks significantly faster using teleoperation subspace mapping than they did using state of the art teleoperation methods.

Tendon-driven hand orthoses have advantages over exoskeletons with respect to wearability and safety because of their low-profile design and ability to fit a range of patients without requiring custom joint alignment. However, no existing study on a wearable tendon-driven hand orthosis for stroke patients presents evidence that such devices can overcome spasticity given repeated use and fatigue, or discusses transmission efficiency. In this study, we propose two designs that provide effective force transmission by increasing moment arms around finger joints. We evaluate the designs with geometric models and experiment using a 3D-printed artificial finger to find force and joint angle characteristics of the suggested structures. We also perform clinical tests with stroke patients to demonstrate the feasibility of the designs. The testing supports the hypothesis that the proposed designs efficiently elicit extension of the digits in patients with spasticity as compared to existing baselines.

Grasping unknown objects has been an active research topic for decades. Approaches range from using various sensors (e.g. vision, tactile) to gain information about the object, to building passively compliant hands that react appropriately to contacts. In this paper, we focus on grasping unknown objects using proprioception (the combination of joint position and torque sensing). Our hypothesis is that proprioception alone can be the basis for versatile performance, including multiple types of grasps for objects with multiple shapes and sizes, and transitions between grasps. Using a series-elastic-actuated gripper, we propose a method for performing stable fingertip grasps for unknown objects with unknown contacts, formulated as multi-input-multi-output (MIMO) control. We also show that the proprioceptive gripper can perform enveloping grasps, as well as the transition from fingertip grasps to enveloping grasps.

In this paper, we focus on the following problem in multifingered robotic grasping: assuming that an external wrench is being applied to a grasped object, will the contact forces between the hand and the object, as well as the hand joints, respond in such a way to preserve quasi-static equilibrium? In particular, we assume that there is no change in the joint torques being actively exerted by the motors; any change in contact forces and joint torques is due exclusively to passive effects arising in response to the external disturbance. Such passive effects include, for example, joints that are driven by highly geared motors (a common occurrence in practice) and thus do not back drive in response to external torques. To account for nonlinear phenomena encountered in such cases, and which existing methods do not consider, we formulate the problem as a mixed-integer program used in the inner loop of an iterative solver. We present evidence showing that this formulation captures important effects for assessing the stability of a grasp employing some of the most commonly used actuation mechanisms.

Automated grasping has a long history of research that is increasing due to interest from industry. One grand challenge for robotics is Universal Picking: the ability to robustly grasp a broad variety of objects in diverse environments for applications from warehouses to assembly lines to homes. Although many researchers now openly share code and data, it is challenging to compare and/or reproduce experimental results to identify which aspects of which approaches work best due to variations in assumptions and experimental protocols, e.g., sensors, lighting, robot arms, grippers, and objects.

Traditional methods to achieve high localization accuracy with tactile sensors usually use a matrix of miniaturized individual sensors distributed on the area of interest. This approach usually comes at a price of increased complexity in fabrication and circuitry, and can be hard to adapt for non planar geometries. We propose to use low cost optic components mounted on the edges of the sensing area to measure how light traveling through an elastomer is affected by touch. Multiple light emitters and receivers provide us with a rich signal set that contains the necessary information to pinpoint both the location and depth of an indentation with high accuracy. We demonstrate sub-millimeter accuracy on location and depth on a 20mm by 20mm active sensing area. Our sensor provides high depth sensitivity as a result of two different modalities in how light is guided through our elastomer. This method results in a low cost, easy to manufacture sensor. We believe this approach can be adapted to cover non-planar surfaces, simplifying future integration in robot skin applications.

Fully wearable hand rehabilitation and assistive devices could extend training and improve quality of life for patients affected by hand impairments. However, such devices must deliver meaningful manipulation capabilities in a small and lightweight package. In this context, this paper investigates the capability of single-actuator devices to assist whole-hand movement patterns through a network of exotendons. Our prototypes combine a single linear actuator (mounted on a forearm splint) with a network of exotendons (routed on the surface of a soft glove). We investigated two possible tendon network configurations: one that produces full finger extension (overcoming flexor spasticity), and one that combines proximal flexion with distal extension at each finger. In experiments with stroke survivors, we measured the force levels needed to overcome various levels of spasticity and open the hand for grasping using the first of these configurations, and qualitatively demonstrated the ability to execute fingertip grasps using the second. Our results support the feasibility of developing future wearable devices able to assist a range of manipulation tasks.

Achieving high spatial resolution in contact sensing for robotic manipulation often comes at the price of increased complexity in fabrication and integration. One traditional approach is to fabricate a large number of taxels, each delivering an individual, isolated response to a stimulus. In contrast, we propose a method where the sensor simply consists of a continuous volume of piezoresistive elastomer with a number of electrodes embedded inside. We measure piezoresistive effects between all pairs of electrodes in the set, and count on this rich signal set containing the information needed to pinpoint contact location with high accuracy using regression algorithms. In our validation experiments, we demonstrate submillimeter median accuracy in locating contact on a 10mm by 16mm sensor using only four electrodes (creating six unique pairs). In addition to extracting more information from fewer wires, this approach lends itself to simple fabrication methods and makes no assumptions about the underlying geometry, simplifying future integration on robot fingers.

Computing grasps for an object is challenging when the object geometry is not known precisely. In this paper, we explore the use of Gaussian process implicit surfaces (GPISs) to represent shape uncertainty from RGBD point cloud observations of objects. We study the use of GPIS representations to select grasps on previously unknown objects, measuring grasp quality by the probability of force closure. Our main contribution is GP-GPIS-OPT, an algorithm for computing grasps for parallel-jaw grippers on 2D GPIS object representations. Specifically, our method optimizes an approximation to the probability of force closure subject to antipodal constraints on the parallel jaws using Sequential Convex Programming (SCP). We also introduce GPIS-Blur, a method for visualizing 2D GPIS models based on blending shape samples from a GPIS. We test the algorithm on a set of 8 planar objects with transparency, translucency, and specularity. Our experiments suggest that GP-GPIS-OPT computes grasps with higher probability of force closure than a planner that does not consider shape uncertainty on our test objects and may converge to a grasp plan up to 5.7×faster than using Monte-Carlo integration, a common method for grasp planning under shape uncertainty. Furthermore, initial experiments on the Willow Garage PR2 robot suggest that grasps selected with GP-GPIS-OPT are up to 90% more successful than those planned assuming a deterministic shape. Our dataset, code, and videos of our experiments are available at http://rll.berkeley.edu/icra2015grasping/.

The articles in this special section focus on the use of cloud computing in the robotics industry. The Internet and the availability of vast computational resources, ever-growing data and storage capacity have the potential to define a new paradigm for robotics and automation. An intelligent system connected to the Internet can expand its onboard local data, computation and sensors with huge data repositories from similar and very different domains, massive parallel computation from server farms and sensor/actuator streams from other robots and automata. It is the potential and also the research challenges of the field that become the focus on this special section. The goal is to group together and to show the state-of-the-art of this newly emerged field, identify the relevant advances and topics, point out the current lines of research and potential applications, and discuss the main research challenges and future work directions.

We design, optimize and demonstrate the behavior of a tendon-driven robotic gripper performing fingertip and enveloping grasps. The gripper consists of two fingers, each with two links, and is actuated using a single active tendon. During unobstructed closing, the distal links remain parallel, creating exact fingertip grasps. Conversely, if the proximal links are stopped by contact with an object, the distal links start flexing, creating a stable enveloping grasp. We optimize the route of the active tendon and the parameters of the springs providing passive extension forces in order to achieve this behavior. We show how an additional passive tendon can be used as a constraint preventing the gripper from entering undesirable parts of the joint workspace. Finally, we introduce a method for optimizing the dimensions of the links in order to achieve enveloping grasps of a large range of objects, and apply it to a set of common household objects.

We introduce ROS Commander (ROSCo), an open source system that enables expert users to construct, share, and deploy robot behaviors for home robots. A user builds a behavior in the form of a Hierarchical Finite State Machine (HFSM) out of generic, parameterized building blocks, with a real robot in the develop and test loop. Once constructed, users save behaviors in an open format for direct use with robots, or for use as parts of new behaviors. When the system is deployed, a user can show the robot where to apply behaviors relative to fiducial markers (AR Tags), which allows the robot to quickly become operational in a new environment. We show evidence that the underlying state machine representation and current building blocks are capable of spanning a variety of desirable behaviors for home robots, such as opening a refrigerator door with two arms, flipping a light switch, unlocking a door, and handing an object to someone. Our experiments show that sensor-driven behaviors constructed with ROSCo can be executed in realistic home environments with success rates between 80% and 100%. We conclude by describing a test in the home of a person with quadriplegia, in which the person was able to automate parts of his home using previously-built behaviors.

Human-in-the loop robotic systems have the potential to handle complex tasks in unstructured environments, by combining the cognitive skills of a human operator with autonomous tools and behaviors. Along these lines, we present a system for remote human-in-the-loop grasp execution. An operator uses a computer interface to visualize a physical robot and its surroundings, and a point-and-click mouse interface to command the robot. We implemented and analyzed four different strategies for performing grasping tasks, ranging from direct, real-time operator control of the end-effector pose, to autonomous motion and grasp planning that is simply adjusted or confirmed by the operator. Our controlled experiment (N=48) results indicate that people were able to successfully grasp more objects and caused fewer unwanted collisions when using the strategies with more autonomous assistance. We used an untethered robot over wireless communications, making our strategies applicable for remote, human-in-the-loop robotic applications.

The Robots for Humanity project aims to enable people with severe motor impairments to interact with their own bodies and their environment through the use of an assistive mobile manipulator, thereby improving their quality of life. Assistive mobile manipulators (AMMs) are mobile robots that physically manipulate the world in order to provide assistance to people with disabilities. They present an exciting frontier for assistive technology, as they can operate away from the user, have a large dexterous workspace (due to their mobility), and not directly encumber their users. The cornerstone of this project is an ongoing, interactive design process with a quadriplegic user, Henry Evans, and his wife and primary caregiver, Jane Evans. Henry has been enabled, through the use of a PR2 robot, to scratch his own face, shave, fetch a towel from his kitchen, and hand out Halloween candy to trick-ortreating children at a local mall.

We present a mobile manipulation platform operated by a motor-impaired person using input from a head-tracker, single-button mouse. The platform is used to perform varied and unscripted manipulation tasks in a real home, combining navigation, perception and manipulation. The operator can make use of a wide range of interaction methods and tools, from direct tele-operation of the gripper or mobile base to autonomous sub-modules performing collision-free base navigation or arm motion planning. We describe the complete set of tools that enable the execution of complex tasks, and share the lessons learned from testing them in a real user's home. In the context of grasping, we show how the use of autonomous sub-modules improves performance in complex, cluttered environments, and compare the results to those obtained by novice, able-bodied users operating the same system.

Grasp planning based on perceived sensor data of an object can be performed in different ways, depending on the chosen semantic interpretation of the sensed data. For example, if the object can be recognized and a complete 3D model is available, a different planning tool can be selected compared to the situation in which only the raw sensed data, such as a single point cloud, is available. Instead of choosing between these options, we present a framework that combines them, aiming to find consensus on how the object should be grasped by using the information from each object representation according to their confidence levels. We show that this method is robust to common errors in perception, such as incorrect object recognition, while also taking into account potential grasp execution errors due to imperfect robot calibration. We illustrate this method on the PR2 robot by grasping objects common in human environments.

Passively adaptive and underactuated robotic hands have shown the potential to achieve reliable grasping in unstructured environments without expensive mechanisms or sensors. Instead of complex run-time algorithms, such hands use design-time analysis to improve performance for a wide range of tasks. Along these directions, we present an optimization framework for underactuated compliant hands. Our approach uses a pre-defined set of grasps in a quasistatic equilibrium formulation to compute the actuation mechanism design parameters that provide optimal performance. We apply our method to a class of tendon-actuated hands; for the simplified design of a two-fingered gripper, we show how a global optimum for the design optimization problem can be computed. We have implemented the results of this analysis in the construction of a gripper prototype, capable of a wide range of grasping tasks over a variety of objects.

Robotic grasping in unstructured environments requires the ability to select grasps for unknown objects and execute them while dealing with uncertainty due to sensor noise or calibration errors. In this work, we propose a simple but robust approach to grasp selection for unknown objects, and a reactive adjustment approach to deal with uncertainty in object location and shape. The grasp selection method uses 3D sensor data directly to determine a ranked set of grasps for objects in a scene, using heuristics based on both the overall shape of the object and its local features. The reactive grasping approach uses tactile feedback from fingertip sensors to execute a compliant robust grasp. We present experimental results to validate our approach by grasping a wide range of unknown objects. Our results show that reactive grasping can correct for a fair amount of uncertainty in the measured position or shape of the objects, and that our grasp selection approach is successful in grasping objects with a variety of shapes.

Collecting grasp data for learning and benchmarking purposes is very expensive. It would be helpful to have a standard database of graspable objects, along with a set of stable grasps for each object, but no such database exists. In this work we show how to automate the construction of a database consisting of several hands, thousands of objects, and hundreds of thousands of grasps. Using this database, we demonstrate a novel grasp planning algorithm that exploits geometric similarity between a 3D model and the objects in the database to synthesize form closure grasps. Our contributions are this algorithm, and the database itself, which we are releasing to the community as a tool for both grasp planning and benchmarking.

Highly underactuated and passively adaptive robotic hands have shown great promise for robust performance in unstructured settings. In order to fully realize this potential, efficient tools are needed to analyze the execution of a grasp when using this class of devices. Along this line, this paper introduces a quasistatic analysis method for underactuated hands. First, we predict whether initial contacts between the fingers and the object are stable throughout the execution of a grasp, or the fingers will slip as the hand closes. Second, we compute the unbalanced forces applied to the object during the grasping process. Finally, once the grasp is complete, we analyze its stability as actuator forces are increased. These computations are performed in 3D, allow arbitrary kinematic structure of the fingers or geometry of the target object and take into account frictional constraints. We discuss applications of this method focusing on both on-line computation to execute a specific grasping task and off-line optimization to increase the range of grasps that can be performed using a given hand model.

To grasp a novel object, we can index it into a database of known 3D models and use precomputed grasp data for those models to suggest a new grasp. We refer to this idea as data-driven grasping, and we have previously introduced the Columbia Grasp Database for this purpose. In this paper we demonstrate a data-driven grasp planner that requires only partial 3D data of an object in order to grasp it. To achieve this, we introduce a new shape descriptor for partial 3D range data, along with an alignment method that can rigidly register partial 3D models to models that are globally similar but not identical. Our method uses SIFT features of depth images, and encapsulates ¿nearby¿ views of an object in a compact shape descriptor.

In this paper we outline a grasp planning system designed to augment the cortical control of a prosthetic arm and hand. A key aspect of this system it the ability to combine online user input and autonomous planning to enable the execution of stable grasping tasks. While user input can ultimately be of any modality, the system is being designed to adapt to partial or noisy information obtained from grasp-related activity in the primate motor cortex. First, principal component analysis is applied to the observed kinematics of physiologic grasping to reduce the dimensionality of hand posture space and simplify the planning task for on-line use. The planner then accepts control input in this reduced-dimensionality space, and uses it as a seed for a hand posture optimization algorithm based on simulated annealing. We present two applications of this algorithm, using data collected from both primate and human subjects during grasping, to demonstrate its ability to synthesize stable grasps using partial control input in real or near-real time.

In this paper, we build upon recent advances in neuroscience research which have shown that control of the human hand during grasping is dominated by movement in a configuration space of highly reduced dimensionality. We extend this concept to robotic hands and show how a similar dimensionality reduction can be defined for a number of different hand models. This framework can be used to derive planning algorithms that produce stable grasps even for highly complex hand designs. Furthermore, it offers a unified approach for controlling different hands, even if the kinematic structures of the models are significantly different. We illustrate these concepts by building a comprehensive grasp planner that can be used on a large variety of robotic hands under various constraints.

The human hand is unrivaled in its ability to grasp and manipulate objects, but we still do not understand all of its complexities. One benefit it has over traditional robot hands is the fact that our fingers conform to a grasped object's shape, giving rise to larger contact areas and the ability to apply larger frictional forces. In this paper, we demonstrate how we have extended our simulation and analysis system with finite element modeling to allow us to evaluate these complex contact types. We also propose a new contact model that better accounts for the deformations and show how grasp quality is affected. This work is part of a larger project to understand the benefits the human hand has in grasping.