Robotic grasping guided by natural language instructions faces challenges due to ambiguities in object descriptions and the need to interpret complex spatial context. Existing visual grounding methods often rely on datasets that fail to capture these complexities, particularly when object categories are vague or undefined. To address these challenges, we make three key contributions. First, we present an automated dataset generation engine for visual grounding in tabletop grasping, combining procedural scene synthesis with template-based referring expression generation, requiring no manual labeling. Second, we introduce the RefGrasp dataset, featuring diverse indoor environments and linguistically challenging expressions for robotic grasping tasks. Third, we propose a visually grounded dexterous grasping framework with continuous grasp generation, validated through extensive real-world robotic experiments. Our work offers a novel approach for language-guided robotic manipulation, providing both a challenging dataset and an effective grasping framework for real-world applications. Project website: https://refer-and-grasp.github.io.

In flexible manufacturing, robots need to swiftly adapt to constantly changing production tasks. However, it remains a challenging problem for robots to grasp objects of specific categories through language instructions to complete production tasks in cluttered scenes. To address this issue, this article proposes a language-guided category push–grasp synergy network following a cognitive-decision framework. First, inspired by how humans can understand the world through interactions with the environment, we propose an environment state difference embodied self-supervision method that enables robots to autonomously collect embodied multimodal data and generate ground truths that eliminate annotation errors for cognition network training. Second, we develop a language-guided embodied multimodal object cognition network that fuses color and depth image information, enhancing the object cognition ability of robots in cluttered scenes and enabling dynamic semantic segmentation based on language commands. Finally, we propose an object manipulation space metric to measure the manipulable space of target objects, linking the reward function with metric changes before and after actions, thereby enhancing the system's perception of the manipulation space and improving operational performance. Experiments conducted in both simulated and real-world environments demonstrate that our proposed method outperforms existing state-of-the-art methods and can be generalized for grasping novel objects.

Human-robot object handovers are essential for robots to effectively serve human needs in various domains of human–robot interaction and collaboration, yet remain a significant challenge. Remarkable progress has been made by parallel-jaw gripper robots in grasp generation and motion planning for handovers, while few studies address this issue using anthropomorphic hands, necessitating the ability to handle higher collision probabilities and lower approaching space under occluded situations. In this article, we present a reactive human-to-robot dexterous handover framework for anthropomorphic hands. The closed-loop framework employs an effective collision detection and grasp selection approach to ensure safe and smooth motion in unstructured environments. We implement a handover system using a UR5 robot arm and a Schunk SVH Hand based on the presented framework, which can react to human motion during the handover process, generalize to diverse objects with different 6-DoF poses, and execute suitable grasp configurations. The generalizability, reliability, and efficiency of our method are demonstrated through the handover of 30 novel objects, a system ablation study for submodule evaluation, and a user study assessment involving eight participants.

This article investigates the challenge of enabling multifinger hands to perform human-like functional grasping for various intentions. However, accomplishing functional grasping in real robot hands present many challenges, including handling generalization ability for kinematically diverse robot hands, generating intention-conditioned grasps for a large variety of objects, and incomplete perception from a single-view camera. In this work, we first propose a six-step functional grasp synthesis algorithm based on fine-grained contact modeling. With the fine-grained contact-based optimization and learned dense shape correspondence, the algorithm is adaptable to various objects of the same category and a wide range of multifinger hands using few demonstrations. Second, over 10 k functional grasps are synthesized to train our neural network, named DexFG-Net, which generates intention-conditioned grasps based on reconstructed object. Extensive experiments in the simulation and physical grasps indicate that the grasp synthesis algorithm can produce human-like functional grasp with robust stability and functionality, and the DexFG-Net can generate plausible and human-like intention-conditioned grasping postures for anthropomorphic hands.

Frequent bad weather at sea severely damages the quality of visual images captured by imaging equipment. Ship instance segmentation in adverse weather conditions remains a major challenge because of poor visibility at sea. Existing approaches for instance segmentation are primarily designed for clear days and rarely consider the aforementioned severe weather. Blurred ship objects can easily cause missed ship detection and decrease the instance segmentation performance on ship images, especially in the case of frequent fog at sea. To this end, we propose a ship instance segmentation framework (IRDCLNet) based on Interference Reduction and Dynamic Contour Learning in foggy scenes. The Interference Reduction Module is proposed to reduce the interference caused by fog and solves the problem of missed ship detection. Meanwhile, we present Dynamic Contour Learning to predict the overall contour of the blurred ships to assist in mask prediction. To handle the scarcity of ocean data in foggy weather, we build the Foggy ShipInsseg dataset, which contains 5,739 real and simulated foggy ship images with 10,900 fine instance mask annotations. Experiments on the Foggy ShipInsseg dataset show that our IRDCLNet outperforms the Mask R-CNN and CondInst baselines and achieves the state-of-the-art performance.

Object grasping in cluttered scenes is a widely investigated field of robot manipulation. Most of the current works focus on estimating grasp pose from point clouds based on an efficient single-shot grasp detection network. However, due to the lack of geometry awareness of the local grasping area, it may cause severe collisions and unstable grasp configurations. In this paper, we propose a two-stage grasp pose refinement network which detects grasps globally while fine-tuning low-quality grasps and filtering noisy grasps locally. Furthermore, we extend the 6-DoF grasp with an extra dimension as grasp width which is critical for collisionless grasping in cluttered scenes. It takes a single-view point cloud as input and predicts dense and precise grasp configurations. To enhance the generalization ability, we build a synthetic single-object grasp dataset including 150 commodities of various shapes, and a complex multi-object cluttered scene dataset including 100k point clouds with robust, dense grasp poses and mask annotations. Experiments conducted on Yumi IRB-1400 Robot demonstrate that the model trained on our dataset performs well in real environments and outperforms previous methods by a large margin.

Robots with a parallel-jaw gripper and suction cup is an adaptive and efficient robotic picking system. This paper proposed Policy-Oriented Instance Segmentation (POIS) for ambidextrous robots. POIS can generate a pair of target masks that allows ambidextrous robots to pick in parallel. It takes a depth image and predicts initial mask, center offset, and policy confidence map through three paralleled branches. We incorporate the initial mask with center offset to obtain candidate instances, from which we select masks of target objects for policy execution (decided with policy confidence map). We also provide a dataset that contains 6k synthetic scenes and 100 real scenes for ambidextrous picking. Trained on synthetic scenes, POIS generalizes well in real scene and is capable of handling novel objects in cluttered scenes. Our dataset and video are available at https://bit.ly/3oJj8Tu.

The fast and accurate classification of polarimetric synthetic aperture radar (PolSAR) data in dynamically changing environments is an important and challenging task. In this paper, we propose an Incremental Multi-view Passive-Aggressive Active learning algorithm, named IMPAA, for PolSAR data classification. This algorithm can deal with online two-view multi-class categorization problem by exploiting the relationship between the polarimetric-color and texture feature sets of PolSAR data. In addition, the IMPAA algorithm can handle the dynamic large-scale datasets where not only the amount of data but also the number of classes gradually increases. Moreover, this algorithm only queries the class labels of some informative incoming samples to update the classifier based on the disagreement of different views' predictors and a randomized rule. Experiments on real PolSAR data demonstrate that the proposed method can use a smaller fraction of queried labels to achieve low online classification errors compared with previously known methods.

Object tracking is one of the important tasks for mobile robot, and developing a robust and real-time visual tracking algorithm which can adaptively capture the varying appearance of target under challenging conditions for mobile robot is still an open problem. The main challenges of visual tracking for mobile robot come from variation of target's appearance and disturbance of environment. To cope with these problems, one of the most important topics is how to select the best tracking features. In this paper, we propose a novel adaptive probabilistic tracking method with discriminative feature selection for mobile robot Different from the existing adaptive tracking algorithms which select the discriminative features in a finite feature set, the proposed method treats feature selection as an estimation problem of the best feature tunable parameters in a continuous space. The estimation of the best tunable parameters and object tracking are implemented via different particle filters with novel observation models. A novel target model updating strategy is also proposed to adapt to the varying appearance of target and resist gradual drift. Experiments show the robustness of the proposed method under challenging conditions.

In this letter, we propose a novel saliency model for saliency detection, named sparse-distinctive (SD) saliency model. Different from the existing models that only consider sparsity or distinctness of image, the proposed model computes saliency based on sparsity and distinctness. The basic idea is that sparsity and distinctness contribute to saliency simultaneously and play different roles under different scenes. This sparse-distinctive saliency model is based on some key ideas introduced in this letter and supported by psychological evidence. Experimental results on public benchmark eye-tracking datasets show that considering the sparsity and distinctness for saliency can improve the accuracy of predicting human fixations, and the proposed model outperforms the mainstream models on predicting human fixations.