Grasping transparent objects remains challenging for robotic systems due to their reflective and refractive properties, which distort depth perception and introduce background noise. Unlike humans, who leverage life experience to perceive depth intuitively, robotic algorithms often fail to generalize across different object types. To address this, we propose a novel framework inspired by human perception for grasping transparent objects. Our approach extends features extracted by foundation models to implicitly learn reconstruction strategies for transparent objects without requiring segmentation priors. Crucially, our framework maintains strong performance across all types of objects and scenes, preventing catastrophic forgetting of opaque objects while learning to perceive transparent ones. By integrating affordance information, our method dynamically guides a five-finger dexterous hand to execute diverse grasping strategies based on human intent. To tackle the challenge of annotating transparent objects, we constructed a large-scale synthetic dataset with depth information, affordance data, and automated annotations. Our framework demonstrates strong generalization, achieving a 96% grasp success rate in real-world robotic experiments and proving its broad applicability across varied environments.

Learning from demonstrations faces challenges in generalizing beyond the training data and often lacks collision awareness. This paper introduces Lan-o3dp, a language-guided object-centric diffusion policy framework that can adapt to unseen situations such as cluttered scenes, shifting camera views, and ambiguous similar objects while offering trainingfree collision avoidance and achieving a high success rate with few demonstrations. We train a diffusion model conditioned on 3D point clouds of task-relevant objects to predict the robot's end-effector trajectories, enabling it to complete the tasks. During inference, we incorporate cost optimization into denoising steps to guide the generated trajectory to be collisionfree. We leverage open-set segmentation to obtain the 3D point clouds of related objects. We use a large language model to identify the target objects and possible obstacles by interpreting the user's natural language instructions. To effectively guide the conditional diffusion model using a time-independent cost function, we proposed a novel guided generation mechanism based on the estimated clean trajectories. In the simulation, we showed that diffusion policy based on the object-centric 3D representation achieves a much higher success rate (68.7%) compared to baselines with simple 2D (39.3%) and 3D scene (43.6%) representations across 21 challenging RLBench tasks with only 40 demonstrations. In real-world experiments, we extensively evaluated the generalization in various unseen situations and validated the effectiveness of the proposed zeroshot cost-guided collision avoidance.

Learning dexterous manipulation from few-shot demonstrations is a significant yet challenging problem for advanced, human-like robotic systems. Dense distilled feature fields have addressed this challenge by distilling rich semantic features from 2D visual foundation models into the 3D domain. However, their reliance on neural rendering models such as Neural Radiance Fields (NeRF) or Gaussian Splatting results in high computational costs. In contrast, previous approaches based on sparse feature fields either suffer from inefficiencies due to multi-view dependencies and extensive training or lack sufficient grasp dexterity. To overcome these limitations, we propose Language-ENhanced Sparse Distilled Feature Field (LensDFF), which efficiently distills view-consistent 2D features onto 3D points using our novel language-enhanced feature fusion strategy, thereby enabling single-view few-shot generalization. Based on LensDFF, we further introduce a few-shot dexterous manipulation framework that integrates grasp primitives into the demonstrations to generate stable and highly dexterous grasps. Moreover, we present a real2sim grasp evaluation pipeline for efficient grasp assessment and hyperparameter tuning. Through extensive simulation experiments based on the real2sim pipeline and real-world experiments, our approach achieves competitive grasping performance, outperforming state-of-the-art approaches. See our website for the code and videos: david-s-martinez.github.io/LensDFF.

The deep learning models has significantly advanced dexterous manipulation techniques for multi-fingered hand grasping. However, the contact information-guided grasping in cluttered environments remains largely underexplored. To address this gap, we have developed ContactDexNet, a method for generating multi-fingered hand grasp samples in cluttered settings through contact semantic map. We introduce a contact semantic conditional variational autoencoder network (CoSe-CVAE) for creating comprehensive contact semantic map from object point cloud. We utilize grasp detection method to estimate hand grasp poses from the contact semantic map. Finally, an unified grasp evaluation model PointNetGPD++ is designed to assess grasp quality and collision probability, substantially improving the reliability of identifying optimal grasps in cluttered scenarios. Our grasp generation method has demonstrated remarkable success, outperforming state-of-the-art (SOTA) methods by at least 4.7%, with 81.0% average grasping success rate in real-world single-object grasping using a known hand, and by at least 9.0% when using an unknown hand. Moreover, in cluttered scenes, our method attains a 76.7% success rate, outperforming the SOTA method by 6.3%. We also proposed the multi-modal multi-fingered grasping dataset generation method. Our multi-fingered hand grasping dataset outperforms previous datasets in scene diversity, modality diversity. More details and supplementary materials can be found at https://sites.google.com/view/contact-dexnet.

We introduce a Cable Grasping-Convolutional Neural Network (CG-CNN) designed to facilitate robust cable grasping in cluttered environments. Utilizing physics simulations, we generate an extensive dataset that mimics the intricacies of cable grasping, factoring in potential collisions between cables and robotic grippers. We employ the Approximate Convex Decomposition technique to dissect the non-convex cable model, with grasp quality autonomously labeled based on simulated grasping attempts. The CG-CNN is refined using this simulated dataset and enhanced through domain randomization techniques. Subsequently, the trained model predicts grasp quality, guiding the optimal grasp pose to the robot’s controller for execution. Grasping efficacy is assessed across both synthetic and real-world settings. Given our model’s implicit collision sensitivity, we achieved commendable success rates of 92.3% for known cables and 88.4% for unknown cables, surpassing contemporary state-of-the-art approaches. Supplementary materials can be found at https://leizhang-public.github.io/cg-cnn/.

Despite the substantial progress in deep learning, its adoption in industrial robotics projects remains limited, primarily due to challenges in data acquisition and labeling. Previous sim2real approaches using domain randomization require extensive scene and model optimization. To address these issues, we introduce an innovative physically-based structured light simulation system, generating both RGB and physically realistic depth images, surpassing previous dataset generation tools. We create an RGBD dataset tailored for robotic industrial grasping scenarios and evaluate it across various tasks, including object detection, instance segmentation, and embedding sim2real visual perception in industrial robotic grasping. By reducing the sim2real gap and enhancing deep learning training, we facilitate the application of deep learning models in industrial settings. Project details are available at https://baikaixin-public.github.io/structured_light_3D_synthesizer/

The exploration of robotic dexterous hands utilizing tools has recently attracted considerable attention. A significant challenge in this field is the precise awareness of a tool’s pose when grasped, as occlusion by the hand often degrades the quality of the estimation. Additionally, the tool’s overall pose often fails to accurately represent the contact interaction, thereby limiting the effectiveness of vision-guided, contact-dependent activities. To overcome this limitation, we present the innovative TOOLEE dataset, which, to the best of our knowledge, is the first to feature affordance segmentation of a tool’s end-effector (EE) along with its defined 6D pose based on its usage. Furthermore, we propose the ToolEENet framework for accurate 6D pose estimation of the tool’s EE. This framework begins by segmenting the tool’s EE from raw RGB-D data, then uses a diffusion model-based pose estimator for 6D pose estimation at a category-specific level. Addressing the issue of symmetry in pose estimation, we introduce a symmetry-aware pose representation that enhances the consistency of pose estimation. Our approach excels in this field, demonstrating high levels of precision and generalization. Furthermore, it shows great promise for application in contact-based manipulation scenarios. All data and codes are available on the project website: https://tooleenet-iros2024.github.io/

Multifingered hand dexterous manipulation is quite challenging in the domain of robotics. One remaining issue is how to achieve compliant behaviors. In this work, we propose a human-in-the-loop learning-control approach for acquiring compliant grasping and manipulation skills of a multifinger robot hand. This approach takes the depth image of the human hand as input and generates the desired force commands for the robot. The markerless vision-based teleoperation system is used for the task demonstration, and an end-to-end neural network model (i.e., TeachNet) is trained to map the pose of the human hand to the joint angles of the robot hand in real-time. To endow the robot hand with compliant human-like behaviors, an adaptive force control strategy is designed to predict the desired force control commands based on the pose difference between the robot hand and the human hand during the demonstration. The force controller is derived from a computational model of the biomimetic control strategy in human motor learning, which allows adapting the control variables (impedance and feedforward force) online during the execution of the reference joint angles. The simultaneous adaptation of the impedance and feedforward profiles enables the robot to interact with the environment compliantly. Our approach has been verified in both simulation and real-world task scenarios based on a multifingered robot hand, that is, the Shadow Hand, and has shown more reliable performances than the current widely used position control mode for obtaining compliant grasping and manipulation behaviors.

Grasping unknown objects with multi-fingered hands at high success rates and in real-time is an unsolved problem. Existing methods are limited in the speed of grasp synthesis or the ability to synthesize a variety of grasps from the same observation. We introduce Five-finger Hand Net (FFHNet), an ML model which can generate a wide variety of high-quality multi-fingered grasps for unseen objects from a single view. Generating and evaluating grasps with FFHNet takes only 30ms on a commodity GPU. To the best of our knowledge, FFHNet is the first ML-based real-time system for multi-fingered grasping with the ability to perform grasp inference at 30 frames per second (FPS). For training, we synthetically generate 180k grasp samples for 129 objects. We are able to achieve 91% grasping success for unknown objects in simulation and we demonstrate the model's capabilities of synthesizing high-quality grasps also for real unseen objects.

Currently, task-oriented grasp detection approaches are mostly based on pixel-level affordance detection and semantic segmentation. These pixel-level approaches heavily rely on the accuracy of a 2D affordance mask, and the generated grasp candidates are restricted to a small workspace. To mitigate these limitations, we firstly construct a novel affordance-based grasp dataset and propose a 6-DoF task-oriented grasp detection framework, which takes the observed object point cloud as input and predicts diverse 6-DoF grasp poses for different tasks. Specifically, our implicit estimation network and visual affordance network in this framework could directly predict coarse grasp candidates, and corresponding 3D affordance heatmap for each potential task, respectively. Furthermore, the grasping scores from coarse grasps are combined with heatmap values to generate more accurate and finer candidates. Our proposed framework shows significant improvements compared to baselines for existing and novel objects on our simulation dataset. Although our framework is trained based on the simulated objects and environment, the final generated grasp candidates can be accurately and stably executed in the real robot experiments when the object is randomly placed on a support surface.

Contact-rich manipulation tasks are commonly found in modern manufacturing settings. However, manually designing a robot controller is considered hard for traditional control methods as the controller requires an effective combination of modalities and vastly different characteristics. In this paper, we first consider incorporating operational space visual and haptic information into a reinforcement learning (RL) method to solve the target uncertainty problems in unstructured environments. Moreover, we propose a novel idea of introducing a proactive action to solve a partially observable Markov decision process (POMDP) problem. With these two ideas, our method can either adapt to reasonable variations in unstructured environments or improve the sample efficiency of policy learning. We evaluated our method on a task that involved inserting a random-access memory (RAM) using a torque-controlled robot and tested the success rates of different baselines used in the traditional methods. We proved that our method is robust and can tolerate environmental variations.

Collaborative robots are expected to work alongside humans and directly replace human workers in some cases, thus effectively responding to rapid changes in assembly lines. Current methods for programming contact-rich tasks, particularly in heavily constrained spaces, tend to be fairly inefficient. Therefore, faster and more intuitive approaches are urgently required for robot teaching. This study focuses on combining visual servoing-based learning from demonstration (LfD) and force-based learning by exploration (LbE) to enable the fast and intuitive programming of contact-rich tasks with minimal user efforts. Two learning approaches were developed and integrated into a framework, one relying on human-to-robot motion mapping (visual servoing approach) and the other relying on force-based reinforcement learning. The developed framework implements the noncontact demonstration teaching method based on the visual servoing approach and optimizes the demonstrated robot target positions according to the detected contact state. The developed framework is compared with two most commonly used baseline techniques, i.e., teach pendant-based teaching and hand-guiding teaching. Furthermore, the efficiency and reliability of the framework are validated via comparison experiments involving the teaching and execution of contact-rich tasks. The proposed framework shows the best performance in terms of the teaching time, execution success rate, risk of damage, and ease of use.

In this work, we focus on improving the robot’s dexterous capability by exploiting visual sensing and adaptive force control. TeachNet, a vision-based teleoperation learning framework, is exploited to map human hand postures to a multi-fingered robot hand. We augment TeachNet, which is originally based on an imprecise kinematic mapping and position-only servoing, with a biomimetic learning-based compliance control algorithm for dexterous manipulation tasks. This compliance controller takes the mapped robotic joint angles from TeachNet as the desired goal, computes the desired joint torques. It is derived from a computational model of the biomimetic control strategy in human motor learning, which allows adapting the control variables (impedance and feedforward force) online during the execution of the reference joint angle trajectories. The simultaneous adaptation of the impedance and feedforward profiles enables the robot to interact with the environment in a compliant manner. Our approach has been verified in multiple tasks in physics simulation, i.e., grasping, opening-a-door, turning-a-cap, and touching-a-mouse, and has shown more reliable performances than the existing position control and the fixed-gain-based force control approaches.

An unstable grasp pose can lead to slip, thus an unstable grasp pose can be predicted by slip detection. A regrasp is required afterwards to correct the grasp pose in order to finish the task. In this work, we propose a novel regrasp planner with multi-sensor modules to plan grasp adjustments with the feedback from a slip detector. Then a regrasp planner is trained to estimate the location of center of mass, which helps robots find an optimal grasp pose. The dataset in this work consists of 1 025 slip experiments and 1 347 regrasps collected by one pair of tactile sensors, an RGB-D camera and one Franka Emika robot arm equipped with joint force/torque sensors. We show that our algorithm can successfully detect and classify the slip for 5 unknown test objects with an accuracy of 76.88% and a regrasp planner increases the grasp success rate by 31.0% compared to the state-of-the-art vision-based grasping algorithm.

Advances in exoskeleton technology now enable interacting with rigid objects in a virtual or remote environment using one's hand and fingertips. However, interaction with non-solid materials - such as liquids, sediments and regolith - alongside solids, can greatly extend the versatility of this technology. Rendering rigid objects adequately requires a control loop with high update rates, whereas fluid dynamics equations are computationally expensive. To accommodate this, the fluid dynamics can be simplified - particularly for fluids with high viscosity - resulting in a fast-to-calculate model to enabling haptic rendering of viscous fluids and rigid bodies simultaneously using DLR's Exodex Adam hand exoskeleton. Viscosity as a proprioceptive cue of fluids can be presented to the human through force feedback at multiple points on the human hand - fingers and palm - letting the user interact with a virtual environment in a more natural way and making the experience more immersive. We carry out two user studies to investigate the human perception abilities of virtual fluids rendered with simplified dynamics, and the discernability of different viscosity in virtual fluids compared real fluids. Results show that virtual media can give the user the perception of interacting with a fluid, even with simplified models, at a high update frequency. Furthermore, the material discernibility corresponds well to actual interaction with real viscous fluids. This shows great promise forward for haptic in-hand interaction in fluid and mixed media environments.

Accurate real-time estimation of the pose and configuration of the human hand attached to a dexterous haptic input device is crucial to improve the interaction possibilities for teleoperation and in virtual and augmented reality. In this paper, we present an approach to reconstruct the pose of the human hand and the joint angles of the fingers when wearing a novel fixed-base (grounded) hand exoskeleton. Using a kinematic model of the human hand built from MRI data, we can reconstruct the hand pose and joint angles without sensors on the human hand, from attachment points on the first three fingers and the palm. We test the accuracy of our approach using motion capture as a ground truth. This reconstruction can be used to determine contact geometry and force-feedback from virtual or remote objects in virtual reality or teleoperation.

This paper presents an adaptive and compliant approach-to-grasp strategy for multi-finger robotic hands, to improve the performance of autonomous grasping when encountering object position uncertainties. With the proposed approach-to-grasp strategy, the first robot finger to experience unexpected impact would pause its movement in a compliant manner, and remains in contact with the object to minimize the unplanned motion of the target object. At the same time, the remainder of the fingers continuously, adaptively move toward re-adjusted grasping positions with respect to the first finger in contact with the object, without the need for on-line re-planning or re-grasping. An adaptive grasp control strategy based on spatial virtual spring framework is proposed to achieve local (e.g. not resorting to the robotic arm) in-hand adjustments of the fingers not yet in contact. As such, these fingers can be adaptively driven to the adjusted desired position to accomplish the grasp. Experimental results demonstrate that significantly larger position errors with respect to the hand workspace can be accommodated with the proposed adaptive compliant grasp control strategy. As much as 391% increase in position error area coverage has been achieved. Finally, beyond the quantitative analysis, additional observations during the extensive experiment trials are discussed qualitatively, to help examine several open issues, and further understand the approach-to-grasp phases of the robot hand tasks.

Dexterous multifingered hands are the most complex and versatile variants of robotic end effectors. Compared to simpler grippers and underactuated hands, they should be more capable of grasping and, especially, manipulating different objects. This paper explores the relationship between kinematic design and manipulation performance of robotic hands. Some evaluation criteria frequently used by hand designers to verify kinematic configurations are revisited. The results from these criteria are scrutinized and compared with the evaluation of the manipulation workspace and the ranges of motion of inhand manipulation for a set of predefined objects. Simulations and actual manipulation experiments are carried out with different kinematic configurations on a modular dexterous hand. The results show some disconnection between perceived good designs through common evaluation criteria and their actual, realizable manipulation performance. This work finally gives some insight toward a more holistic approach to design hands that better address grasp and manipulation for the intended tasks and applications.

This paper introduces a new framework for task space telemanipulation. The TASK space grasping and MANipulation (TaskMan) concept utilizes a library of tasks based on gesture commands, which replaces the conventional mapping required between the human hand and the end effector. Task communication between the human machine interface (HMI) and the robot end effector requires two symbiotic but nonidentical state machines on the master and slave side. The task states on two sides are synchronized via a single channel communication, as opposed to multi-channel joint space or Cartesian mapped information. HAND gesture command for grasping and MANipulation (HandyMan) HMI command algorithm is proposed for the recognition of hand gestures, which incorporates a library of intuitive task gestures to be used by the teleoperator wearing a CyberGlove. The task gestures are used to drive the states of the TaskMan state machines. With the proposed concepts, this work has realized teleoperated grasp and manipulation with a 15-DoF robot hand in task space. Full 6-DoF of object manipulation was achieved with different grasp combinations, and demonstrated higher repeatability, success rate and easier operation compared to conventional joint space teleoperation methods.

This paper presents experimental results on the five-finger dexterous robot hand DLR-HIT II, with Cartesian impedance control based on joint torque and nonlinearity compensation for elastic dexterous robot joints. To improve the performance of the impedance controller, system parameter estimations with extended kalman filter and gravity compensation have been investigated on the robot hand. Experimental results show that, for the harmonic drive robot hand with joint toruqe feedback, accurate position tracking and stable torque/force response can be achieved with cartesian and joint impedance controller. In addition, a FPGA-based control architecture with flexible communication is proposed to perform the designed impedance controller.

This paper introduces a study aimed to help quantify the benefits of limited-performance force-feedback user input devices for space telemanipulation with a dexterous robotic arm. A teleoperated robotic hand has been developed for the European Space Agency by the German Aerospace Center (DLR) for a lunar rover prototype. Studies carried out on this telerobotic system investigated several criteria critical to telemanipulation in space: (1) grasping task completion time, (2) grasping task difficulty, (3) grasp quality, and (4) difficulty level for the operator to assess the grasp quality. Several test subjects were allocated to remotely grasp regular and irregular shaped objects, under different combinations of visual- and force-feedback conditions. This work categorized the benefits of visual- and force-feedback in teleoperated grasping through several performance metrics. Furthermore, it has been shown that, with local joint-level impedance control, good grasping performance with rigid hard objects can be achieved, even with limited force-feedback information and low communication bandwidth. On the other hand, a performance ceiling was also found when grasping deformable objects, where the limited force-feedback setup cannot sufficiently reflect the object boundary to the teleoperator.

This paper presents a new developed multisensory five-fingered dexterous robot hand: the DLR/HIT Hand II. The hand has an independent palm and five identical modular fingers, each finger has three DOFs and four joints. All the actuators and electronics are integrated in the finger body and the palm. By using powerful super flat brushless DC motors, tiny harmonic drivers and BGA form DSPs and FPGAs, the whole fingerpsilas size is about one third smaller than the former finger in the DLR/HIT Hand I. By using the steel coupling mechanism, the phalanx distalpsilas transmission ratio is exact 1:1 in the whole movement range. At the same time, the multisensory dexterous hand integrates position, force/torque and temperature sensors. The hierarchical hardware structure of the hand consists of the finger DSPs, the finger FPGAs, the palm FPGA and the PCI based DSP/FPGA board. The hand can communicate with external with PPSeCo, CAN and Internet. Instead of extra cover, the packing mechanism of the hand is implemented directly in the finger body and palm to make the hand smaller and more human like. The whole weight of the hand is about 1.5Kg and the fingertip force can reach 10N.

This paper presents a new developed multisensory five-fingered dexterous robot hand : the DLR/HIT Hand II. The hand has an independent palm and five identical modular fingers, each finger has three DOFs and four joints. All the actuators and electronics are integrated in the finger body and the palm. By using powerful super flat brushless DC motors, tiny harmonic drives and BGA form DSPs and FPGAs, the whole fingerpsilas size is about one third smaller than the former finger in the DLR/HIT Hand I. By using the steel coupling mechanism, the phalanx distalpsilas transmission ratio is exact 1:1 in the whole movement range. At the same time, the multisensory dexterous hand integrates position, force/torque and temperature sensors. The hierarchical hardware structure of the hand consists of the finger DSPs, the finger FPGAs, the palm FPGA and the PCI based DSP/FPGA board. The hand can communicate with external with PPSeCo , CAN and Internet. Instead of extra cover, the packing mechanism of the hand is implemented directly in the finger body and palm to make the hand smaller and more human like. The whole weight of the hand is about 1.5 Kg and the fingertip force can reach 10N.