Grasping is the process of picking up an object by applying forces and torques at a set of contacts. Recent advances in deep learning methods have allowed rapid progress in robotic object grasping. In this systematic review, we surveyed the publications over the last decade, with a particular interest in grasping an object using all six degrees of freedom of the end-effector pose. Our review found four common methodologies for robotic grasping: sampling-based approaches, direct regression, reinforcement learning, and exemplar approaches In addition, we found two “supporting methods” around grasping that use deep learning to support the grasping process, shape approximation, and affordances. We have distilled the publications found in this systematic review (85 papers) into ten key takeaways we consider crucial for future robotic grasping and manipulation research.

To match the ever increasing standards of fresh products, and the need to reduce waste, we devise an alternative to the destructive and highly variable fruit ripeness estimation by a penetrometer. We propose a fully automatic method to assess the ripeness of mango which is non-destructive, allows the user to test multiple surface areas with a single touch and is capable of dissociating between ripe and non-ripe fruits. A custom-made gripper equipped with a capacitive tactile sensor array is used to palpate the fruit. The ripeness is estimated as mango stiffness extracted through a simplified spring model. We test the framework on a set of 25 mangoes of the Keitt variety, and compare the results to penetrometer measurements. We show it is possible to correctly classify 88% of the mango without removing the skin of the fruit. The method can be a valuable substitute for non-destructive fruit ripeness testing. To the authors knowledge, this is the first robotics ripeness estimation system based on capacitive tactile sensing technology.

We aim to reliably predict whether a grasp on a known object is successful before it is executed in the real world. There is an entire suite of grasp metrics that has already been developed which rely on precisely known contact points between object and hand. However, it remains unclear whether and how they may be combined into a general purpose grasp stability predictor. In this paper, we analyze these questions by leveraging a large scale database of simulated grasps on a wide variety of objects. For each grasp, we compute the value of seven metrics. Each grasp is annotated by human subjects with ground truth stability labels. Given this data set, we train several classification methods to find out whether there is some underlying, non-trivial structure in the data that is difficult to model manually but can be learned. Quantitative and qualitative results show the complexity of the prediction problem. We found that a good prediction performance critically depends on using a combination of metrics as input features. Furthermore, non-parametric and non-linear classifiers best capture the structure in the data.

Many analytical metrics have been proposed to evaluate the quality of a grasp based on different criteria and principles. To use most of them in practical real applications, some operational parameters need to be determined: maximum and minimum values, normalization ratios, quality thresholds, robustness in front of position errors and, more importantly, relations between alternative metrics. This paper proposes a methodology to study and characterize the operational parameters that allow the use of several metrics in practical applications, and comparing them. The proposed approach uses exhaustive simulation testing to obtains statically significant results regarding the measurements of several quality metrics. This allows an informed setting of the practical operational values for each metric. Results are provided for a Barrett hand grasping a varied set of objects.

We review the work on data-driven grasp synthesis and the methodologies for sampling and ranking candidate grasps. We divide the approaches into three groups based on whether they synthesize grasps for known, familiar, or unknown objects. This structure allows us to identify common object representations and perceptual processes that facilitate the employed data-driven grasp synthesis technique. In the case of known objects, we concentrate on the approaches that are based on object recognition and pose estimation. In the case of familiar objects, the techniques use some form of a similarity matching to a set of previously encountered objects. Finally, for the approaches dealing with unknown objects, the core part is the extraction of specific features that are indicative of good grasps. Our survey provides an overview of the different methodologies and discusses open problems in the area of robot grasping. We also draw a parallel to the classical approaches that rely on analytic formulations.

Prosthetic hands have evolved and improved over the years, helping people gaining manipulation capabilities. Having a simulation tool able to obtain quantitative evaluation of the grasp capabilities of such hands could give insights as how to improve the design of hand prostheses or robotic hands by means of obtaining better quality scores. The purpose of this work is to present a framework developed to evaluate the grasp capabilities of a prosthetic hand using a selected set of grasp quality measures, and compare the results with the ones obtained for the human hand using a biomechanical model. Experiments grasping an object with different postures and varying aspects of the prosthetic hand model were performed showing the functionality of the proposed framework to evaluate the grasp quality.

In the context of robot grasping and manipulation, realistic simulation requires accurate modeling of contacts between bodies and, in a practical level, accurate simulation of touch sensors. This paper addresses the problem of simulating a tactile sensor considering soft contacts and full friction description. The developed model consists of a surface contact patch described by a mesh of contact elements. For each element, a full friction description is built considering stick-slip phenomena. The developed sensor model is implemented using OpenRAVE and used to perform typical tasks related to tactile sensors. The performance of the simulated sensor is then compared to a real one. It is also demonstrated how it can be integrated on the simulation of a complete robot grasping system.

We consider the problem of grasp and manipulation planning when the state of the world is only partially observable. Specifically, we address the task of picking up unknown objects from a table top. The proposed approach to object shape prediction aims at closing the knowledge gaps in the robot's understanding of the world. A completed state estimate of the environment can then be provided to a simulator in which stable grasps and collision-free movements are planned. The proposed approach is based on the observation that many objects commonly in use in a service robotic scenario possess symmetries. We search for the optimal parameters of these symmetries given visibility constraints. Once found, the point cloud is completed and a surface mesh reconstructed. Quantitative experiments show that the predictions are valid approximations of the real object shape. By demonstrating the approach on two very different robotic platforms its generality is emphasized.

The adoption of robots for service tasks in natural environments calls for the use of sensors to allow manipulation of objects under imperfect environment knowledge and the use of knowledge transfer from humans. This paper addresses these challenges by proposing a new abstraction architecture for embodiment independent sensor-based control of manipulation. The aim is to address three specific challenges: hardware independent control of manipulation, use of sensors to alleviate problems of complexity and uncertainty of the environment, and ease of transferring knowledge over different embodiments through a hierarchical abstract representation of manipulation skills. The proposed abstraction architecture is demonstrated for hardware independence and failure detection on two different manipulator platforms.

Robust grasping of objects in uncertainty conditions can be achieved with the visual monitoring of the interaction between the robot hand and the object. In this paper we propose a new approach for the visual tracking of a robot hand suitable to observe the interaction between robot and object. It consists on the continuous vision-based recovery of the articular pose of the robot hand. It is based on the principles of virtual visual servoing, which allows to deal with articulated bodies and occlusions. Its suitability is shown by tracking a parallel jaw gripper under different conditions such as self-occlusion, articulated motion and important changes in the point-of-view. The potential applications range from estimating the robot hand articulated pose under poor hand-eye calibration and joint feedback, until detecting deficient contact configurations, incipient slips, etc.

This paper addresses the problem of robot grasping in conditions of uncertainty. We propose a grasp controller that deals robustly with this uncertainty using feedback from different contact-based sensors. This controller assumes a description of grasp consisting of a primitive that only determines the initial configuration of the hand and the control law to be used. We exhaustively validate the controller by carrying out a large number of tests with different degrees of inaccuracy in the pose of the target objects and by comparing it with results of a naive grasp controller.

This paper addresses the problem of grasp planning of unmodelled 3D objects with the aide of vision. The approach proposes the extension of fast 2D contour-based grasp planning algorithms, and the use of images of the target object from different points of view to recover critical 3D information like the size, location and the pose. A complete method is described, and experimental results are presented showing its feasibility.

In this paper we present a framework for grasp planning with a humanoid robot arm and a five-fingered hand. The aim is to provide the humanoid robot with the ability of grasping objects that appear in a kitchen environment. Our approach is based on the use of an object model database that contains the description of all the objects that can appear in the robot workspace. This database is completed with two modules that make use of this object representation: an exhaustive offline grasp analysis system and a real-time stereo vision system. The offline grasp analysis system determines the best grasp for the objects by employing a simulation system, together with CAD models of the objects and the five-fingered hand. The results of this analysis are added to the object database using a description suited to the requirements of the grasp execution modules. A stereo camera system is used for a real-time object localization using a combination of appearance-based and model-based methods. The different components are integrated in a controller architecture to achieve manipulation task goals for the humanoid robot

It is widely accepted that spatial reasoning plays a central role in artificial intelligence, for it has a wide variety of potential applications, e.g., in robotics, geographical information systems, medical analysis and diagnosis. As noticed by many authors, spatial and temporal reasoning have a close connection. In this paper we propose a new, semi-decidable, modal logic for spatial reasoning through directional relations, which is able to express meaningful spatial statements. Spatial propositional neighborhood logic can be polynomially reduced to a decidable temporal logic based on time intervals preserving, at least, valid formulas. Thanks to such a reduction, we are able to reuse a sound and complete tableaux method in order to reason with spatial propositional neighborhood logic; to the best of our knowledge, there are practically no previous attempts of devising automatic reasoning methods for spatial reasoning

This paper presents and analyzes 12 quality measures that characterize robotic grips according to their stability and reliability. The measures are designed to assess three-finger grips of two-dimensional parts performed in a real environment, taking into account both theoretical aspects and unavoidable uncertainties of a grasping action. They build on the existing literature and on physical and mechanical considerations. The measures constitute a feature space that pattern recognition methods can use in order to classify robotic grips according to their quality. Six of the measures depend on the actual finger configuration of the gripper, and they have shown to be critical for better characterization. The kinematics of the Barrett Hand have been used. As a validation step, the measures are merged in two global quality values (with different practical applicability) that can be used to rank feasible candidate grips.

Learning techniques in robotic grasping applications have usually been concerned with the way a hand approaches to an object, or with improving the motor control of manipulation actions. We present an active learning approach devised to face the problem of visually-guided grasp selection. We want to choose the best hand configuration for grasping a particular object using only visual information. Experimental data from real grasping actions is used, and the experience gathering process is driven by an on-line estimation of the reliability assessment capabilities of the system. The goal is to improve the selection skills of the grasping system, minimizing at the same time the cost and duration of the learning process.

This paper presents and analyses ten criteria that assess the quality of a set of three-finger grips suitable for dextrous manipulation on real 2D parts. The set of candidate hand configurations is the result of a previous process of grasp generation from the object image. The proposed criteria include six that depend on the actual finger configuration of the gripper. The kinematics of the Barrett hand has been used. The criteria are merged to give a global quality value that can be used to select the best grip to execute. Experimental results include tests on stability and the effect of parameter variation.

This paper deals with visually guided grasping of unmodeled objects for robots which exhibit an adaptive behavior based on their previous experiences. Nine features are proposed to characterize three-finger grasps. They are computed from the object image and the kinematics of the hand. Real experiments on a humanoid robot with a Barrett hand are carried out to provide experimental data. This data is employed by a classification strategy, based on the k-nearest neighbour estimation rule, to predict the reliability of a grasp configuration in terms of five different performance classes. Prediction results suggest the methodology is adequate.

This paper presents an implemented vision-based strategy for computing three-finger stable grasps on unknown planar objects. Its only input is an image of a real unknown curved object instead of synthetic polygonal models. Grasp regions are identified for achieving tolerance around contact points so that errors in finger positioning as well as a finite finger size are considered. It can find expanding and squeezing grasps involving internal and external regions, including holes. Theoretical constraints for force-closure are met while assuring robustness and real-time performance under real-world conditions.

Research in grasping and manipulation has mainly focused on the theoretical framework of the grasp analysis and synthesis, however there are few works in the practical implementation of these approaches. We propose several methods and criteria for constraining visually-selected planar grasps to the particular kinematics of a three finger gripper, the Barrett Hand. The method relies on the features of an object provided by a vision system, and uses several criteria for classifying the possible grasps according to the configurations of the gripper that fit them. It has been implemented on the UMass Torso and the results are discussed.

A key issue in robotics is the development of the ability to grasp unknown objects. This ability requires a grasp determination mechanism that, based on the analysis of the description of the object, determines how it can be stably grasped. In this paper, a grasp determination method is presented that computes a set of grasps that comply with the force-closure condition. Its input is a set of contours, extracted from the vision data, describing the shape of the object. The internal holes of the object are taken into account, so the algorithm can find grasps on them. The algorithm also finds expansion and squeezing grasps, which are executed by opening and closing the gripper fingers.