Robotic grasping guided by natural language instructions faces challenges due to ambiguities in object descriptions and the need to interpret complex spatial context. Existing visual grounding methods often rely on datasets that fail to capture these complexities, particularly when object categories are vague or undefined. To address these challenges, we make three key contributions. First, we present an automated dataset generation engine for visual grounding in tabletop grasping, combining procedural scene synthesis with template-based referring expression generation, requiring no manual labeling. Second, we introduce the RefGrasp dataset, featuring diverse indoor environments and linguistically challenging expressions for robotic grasping tasks. Third, we propose a visually grounded dexterous grasping framework with continuous grasp generation, validated through extensive real-world robotic experiments. Our work offers a novel approach for language-guided robotic manipulation, providing both a challenging dataset and an effective grasping framework for real-world applications. Project website: https://refer-and-grasp.github.io.

Human-robot object handovers are essential for robots to effectively serve human needs in various domains of human–robot interaction and collaboration, yet remain a significant challenge. Remarkable progress has been made by parallel-jaw gripper robots in grasp generation and motion planning for handovers, while few studies address this issue using anthropomorphic hands, necessitating the ability to handle higher collision probabilities and lower approaching space under occluded situations. In this article, we present a reactive human-to-robot dexterous handover framework for anthropomorphic hands. The closed-loop framework employs an effective collision detection and grasp selection approach to ensure safe and smooth motion in unstructured environments. We implement a handover system using a UR5 robot arm and a Schunk SVH Hand based on the presented framework, which can react to human motion during the handover process, generalize to diverse objects with different 6-DoF poses, and execute suitable grasp configurations. The generalizability, reliability, and efficiency of our method are demonstrated through the handover of 30 novel objects, a system ablation study for submodule evaluation, and a user study assessment involving eight participants.

Grasping is one of the most fundamental skills for humans to interact with objects. However, it remains a challenging problem for anthropomorphic hands, due to the lack of object affordance understanding and high-dimensional grasp planning. In this work, we propose an anthropomorphic hand grasping framework to learn realistic and reasonable grasps in cluttered scenes, which tackles the problem in three items: 1) graspable point segmentation; 2) hand grasp generation and 3) grasp optimization. Specifically, our method generates high-quality hand grasps efficiently without complete object models by learning graspable points, associated grasp configurations from observed point cloud in a parallel manner and optimizing predicted grasps based on hand-object contacts. Simulation experiments show that our model generates physical plausible grasps for the anthropomorphic hand effectively with over 70% success rate. Real-world experiments demonstrate that the model trained in simulation performs satisfactorily in real-world scenarios for unseen objects.

This article investigates the challenge of enabling multifinger hands to perform human-like functional grasping for various intentions. However, accomplishing functional grasping in real robot hands present many challenges, including handling generalization ability for kinematically diverse robot hands, generating intention-conditioned grasps for a large variety of objects, and incomplete perception from a single-view camera. In this work, we first propose a six-step functional grasp synthesis algorithm based on fine-grained contact modeling. With the fine-grained contact-based optimization and learned dense shape correspondence, the algorithm is adaptable to various objects of the same category and a wide range of multifinger hands using few demonstrations. Second, over 10 k functional grasps are synthesized to train our neural network, named DexFG-Net, which generates intention-conditioned grasps based on reconstructed object. Extensive experiments in the simulation and physical grasps indicate that the grasp synthesis algorithm can produce human-like functional grasp with robust stability and functionality, and the DexFG-Net can generate plausible and human-like intention-conditioned grasping postures for anthropomorphic hands.

Grasping in cluttered environments is one of the most fundamental skills in robotic manipulation. Most of the current works focus on estimating grasp poses for parallel-jaw or suction-cup end effectors. However, the study for dexterous anthropomorphic hand grasping in clutter remains a great challenge. In this paper, we propose HGC-Net, a single-shot network that learns to predict dense hand grasp configurations in clutter from single-view point cloud input. Our end-to-end neural network can predict hand grasp proposals efficiently and effectively. To enhance generalization, we built a large-scale synthetic grasping dataset with 179 household objects, 5K cluttered scenes and over 10M hand annotations. Experiments in simulation show that our model can predict dense and robust hand grasps and clear over 78% of unseen objects in clutter without any post-processing and outperform baseline methods by a large margin. Experiments on the real robot platform also demonstrate that the model trained on synthetic data performs well in natural environments. Code is available at https://github.com/yimingli1998/hgc_net.