Cable transmission enables motors of robotic arm to operate lightweight and low-inertia joints remotely in various environments, but it also creates issues with motion coupling and cable routing that can reduce arm's control precision and performance. In this paper, we present a novel motion decoupling mechanism with low-friction to align the cables and efficiently transmit the motor's power. By arranging these mechanisms at the joints, we fabricate a fully decoupled and lightweight cable-driven robotic arm called D3-Arm with all the electrical components be placed at the base. Its 776 mm length moving part boasts six degrees of freedom (DOF) and only 1.6 kg weights. To address the issue of cable slack, a cable-pretension mechanism is integrated to enhance the stability of long-distance cable transmission. Through a series of comprehensive tests, D3-Arm demonstrated 1.29 mm average positioning error and 2.0 kg payload capacity, proving the practicality of the proposed decoupling mechanisms in cable-driven robotic arm.

Fast and efficient collision detection is essential for motion generation in robotics. In this paper, we propose an efficient collision detection framework based on the Signed Distance Field (SDF) of robots, seamlessly integrated with a self-collision detection module. Firstly, we decompose the robot's SDF using forward kinematics and leverage multiple extremely lightweight networks in parallel to efficiently approximate the SDF. Moreover, we introduce support vector machines to integrate the self-collision detection module into the framework, which we refer to as the SDF-SC framework. Using statistical features, our approach unifies the representation of collision distance for both SDF and self-collision detection. During this process, we maintain and utilize the differentiable properties of the framework to optimize collision-free robot trajectories. Finally, we develop a reactive motion controller based on our framework, enabling real-time avoidance of multiple dynamic obstacles. While maintaining high accuracy, our framework achieves inference speeds up to five times faster than previous methods. Experimental results on the Franka robotic arm demonstrate the effectiveness of our approach. Project page: https://sites.google.com/view/icra2025-sdfsc.

Catching flying objects with a cushioning process is a skill commonly performed by humans, yet it remains a significant challenge for robots. In this paper, we present a framework that combines optimization and learning to achieve compliant catching on mobile manipulators (CCMM). First, we propose a high-level capture planner for mobile manipulators (MM) that calculates the optimal capture point and joint configuration. Next, the pre-catching (PRC) planner ensures the robot reaches the target joint configuration as quickly as possible. To learn compliant catching strategies, we propose a network that leverages the strengths of LSTM for capturing temporal dependencies and positional encoding for spatial context (P-LSTM). This network is designed to effectively learn compliant strategies from human demonstrations. Following this, the post-catching (POC) planner tracks the compliant sequence output by the P-LSTM while avoiding potential collisions due to structural differences between humans and robots. We validate the CCMM framework through both simulated and real-world ball-catching scenarios, achieving a success rate of 98.70% in simulation, 92.59% in real-world tests, and a 28.7% reduction in impact torques. The open source code will be released for the reference of the community1.

The number of motors directly influences the dexterity, size, and cost of a robotic hand. In this paper, we present MuxHand, a robotic hand that utilizes a time-division multiplexing motor (TDMM) mechanism. This system enables independent control of 9 cables with just 4 motors, significantly reducing both cost and size while maintaining high dexterity. To enhance stability and smoothness during grasping and manipulation tasks, we integrate magnetic joints into the three 3D-printed fingers. These joints provide impact resistance, resetting capabilities. The three fingers together have a total of 30 degrees of freedom (DOF), 18 of which are passive DOF, allowing the hand to conform closely to the surface of an object during grasping. We conduct a series of experiments to assess the performance parameters of MuxHand, including its grasping and manipulation capabilities. The results show that the TDMM mechanism precisely controls each cable connected to the finger joints, enabling robust grasping and dexterous manipulation. Furthermore, compared to the traditional approach of assigning a motor to each active DOF, the cost is reduced by 42.06%. The maximum load of a single finger reaches 7.0 kg, the maximum load at the finger joint root is 12.0 kg, the maximum driving force at the joint root is 5.0 kg, and the maximum fingertip force is 10.0 N.

Multi-task learning of deformable object manipulation is a challenging problem in robot manipulation. Most previous works address this problem in a goal-conditioned way and adapt goal images to specify different tasks, which limits the multi-task learning performance and can not generalize to new tasks. Thus, we adapt language instruction to specify deformable object manipulation tasks and propose a learning framework. We first design a unified Transformer-based architecture to understand multi-modal data and output picking and placing action. Besides, we have applied the visible connectivity graph to tackle nonlinear dynamics and complex configuration of the deformable object. Both simulated and real experiments have demonstrated that the proposed method is effective and can generalize to unseen instructions and tasks. Compared with the state-of-the-art method, our method achieves higher success rates (87.2% on average) and has a 75.6% shorter inference time. We also demonstrate that our method performs well in real-world experiments. Supplementary videos can be found at https://sites.google.com/view/language-deformable.

It is difficult to guarantee an empty living environment to prevent unexpected contact between the object being manipulated by the robot and unplanned obstacles. In this paper, we propose a planar compliant contact control method for planar manipulation to cope with unexpected contact. We first use sheet gel as a multi-dimensional passive elastic element and combine it with a two-finger gripper to design a multi-dimensional elastic gripper. Subsequently, we explore the lumped parameter model for the force-displacement relationship of gel deformation and combine the model with the high impedance motion of robots to design an elastic interaction controller. The controller not only actively adjusts the deformation of the gel to provide the desired contact force and torque depending on contact, but also performs avoidance by following the surface of obstacles. Finally, we design and deploy several planar compliant contact experiments to validate the proposed method and demonstrate the unexpected contact response in human-robot co-packing. The results show that our method enables the robot to remain compliant in the face of unexpected contact caused by unplanned obstacles, which provides a guarantee for safe manipulation. Physics experiments can be viewed in the attached video.

The acquisition of transparent 3D shapes will facilitate many multimedia and computer vision tasks, such as game/movie production and virtual enrioment applications. In this work, we propose a novel method for detailed reconstruction of transparent objects by exploiting polarimetric cues. Most of existing transparent shapes reconstruction methods usually lack sufficient constraints and suffer from the over-smooth problem. Hence, we introduce polarization information as a complementary cue. Specifically, we employ the implicit representation for object's geometry with a neural network, while the polarization render is capable of differentiably rendering the object's polarization images from given illumination configuration. However, direct comparison of rendered polarization images to the real-world captured images will have additional errors due to the transmission in the transparent object. To make the polarimetric cues technically feasible on transparent shapes reconstruction, the concept of reflection percentage which represents proportion of the reflection component is introduced as the weight of the polarization loss. Based on controllable environment setup, we build a polarization dataset containing several solid and smooth transparent objects to verify our method. Experimental results show that our method is capable of recovering detailed shapes and improving reconstruction quality of transparent objects.

This paper presents a learning-based method for transparent surface estimation from a single view polarization image. Existing shape from polarization(SfP) methods have the difficulty in estimating transparent shape since the inherent transmission interference heavily reduces the reliability of physics-based prior. To address this challenge, we propose the concept of physics-based prior confidence, which is inspired by the characteristic that the transmission component in the polarization image has more noise than reflection. The confidence is used to determine the contribution of the interfered physics-based prior. Then, we build a network(TransSfP) with multi-branch architecture to avoid the destruction of relationships between different hierarchical inputs. To train and test our method, we construct a dataset for transparent shape from polarization with paired polarization images and ground-truth normal maps. Extensive experiments and comparisons demonstrate the superior accuracy of our method. Our cdataset and code are publicly available at https://github.com/shaomq2187/TransSfP

This paper presents a learning based motion planning method for robotic manipulation, aiming to solve the asymptotically-optimal motion planning problem with nonlinear kinematics in a complex environment. The core of the proposed method is based on a novel neural network model, i.e., graph wasserstein autoencoder (GraphWAE) network, which is used to represent the implicit sampling distributions of the configuration space (C-space) for sampling-based planning algorithms. Through learning the implicit distributions, we can guide the planning process to search or extend in the desired region to reduce the collision checks dramatically for fast and high-quality motion planning. The theoretical analysis and proofs are given to demonstrate the probabilistic completeness and asymptotic optimality of the proposed method. Numerical simulations and experiments are conducted to validate the effectiveness of the proposed method through a series of planning problems from 2D, 6D and 12D robot C-spaces in the challenging scenes. Results indicate that the proposed method can achieve better planning performance than the state-of-the-art planning algorithms. Note to Practitioners—The motivation of this work is to develop a fast and high-quality asymptotically optimal motion planning method for practical applications such as autonomous driving, robotic manipulation and others. Due to the time consumption caused by collision detection, current planning algorithms usually take much time to converge to the optimal motion path especially in the complicated environment. In this paper, we present a neural network model based on GraphWAE to learn the biasing sampling distributions as the sample generation source to further reduce or avoid collision checks of sampling-based planning algorithms. The proposed method is general and can be also deployed in other sampling-based planning algorithms for improving planning performance in different robot applications.

The grasping of transparent objects is challenging but of significance to robots. In this article, a visual–tactile fusion framework for transparent object grasping in complex backgrounds is proposed, which synergizes the advantages of vision and touch, and greatly improves the grasping efficiency of transparent objects. First, we propose a multiscene synthetic grasping dataset named SimTrans12 K together with a Gaussian-mask annotation method. Next, based on the TaTa gripper, we propose a grasping network named transparent object-grasping convolutional neural network for grasping position detection, which shows good performance in both synthetic and real scenes. Inspired by human grasping, a tactile calibration method and a visual–tactile fusion classification method are designed, which improve the grasping success rate by 36.7% compared with direct grasping and the classification accuracy by 39.1%. Furthermore, the tactile height sensing module and the tactile position exploration module are added to solve the problem of grasping transparent objects in irregular and visually undetectable scenes. The experimental results demonstrate the validity of the framework.

Large-area and high-precision tactile sensing information can not only improve the stability of robot grasping but also compensate for the lack of visual information in specific environments such as turbid underwater, dimness, and smoke. In this paper, we devise a universal jamming gripper with high-quality tactile sensing capability. The gripper adopts the particle jamming mechanism for grasping, and simultaneously uses a built-in camera to detect the deformation of its surface to obtain tactile information. To make the inside of the gripper transparent, glass beads and liquid with the same refractive index are applied as the internal filling. Besides, special treatments are taken to improve the tactile perception resolution of the gripper. The design perfectly merges visual-based tactile sensing into the traditional universal jamming gripper without changing its original gripping performance, making it possible for simultaneous grasping and sensing. To verify the tactile perception and grasping ability of the gripper in specific environments, we design two underwater experiments for grasping and pipe leak detection based on tactile information. Both have achieved a success rate not less than 95%, which demonstrates the effectiveness of the proposed gripper for manipulation in low visibility environments.

In this paper, we propose a novel real-time 6D object pose estimation framework based on Point-wise Attention Keypoints Voting Network (PAV-Net). Compared with previous methods that use all features indiscriminately, we evaluate and integrate the visible points features before estimation to deal with the unstructured and uneven properties of point-wise features. Specifically, we first locate the object roughly by object detection and transfer the captured point cloud coordinates to the local center. Then we extract point-wise features from RGB images and point clouds respectively and perform semantic segmentation. Finally, the point-wise features are screened and integrated with the help of the attention keypoints voting to predict the accurate keypoint coordinates, and the 6D object pose can be obtained within keypoints fitting. The proposed method can effectively avoid external interference and improve the efficiency of influential point features utilization by point-wise attention voting so that the framework only needs a simple feature extraction network support to have better real-time performance. Extensive experiments confirm this conclusion and show that the performance of proposed framework on LineMOD and YCB-Video datasets is superior to other real-time pose estimation methods at the same speed.

Object rearranging is one of the most common deformable manipulation tasks, where the robot needs to rearrange a deformable object into a goal configuration. Previous studies focus on designing an expert system for each specific task by model-based or data-driven approaches and the application scenarios are therefore limited. Some research has been attempting to design a general framework to obtain more advanced manipulation capabilities for deformable rearranging tasks, with lots of progress achieved in simulation. However, transferring from simulation to reality is difficult due to the limitation of the end-to-end CNN architecture. To address these challenges, we design a local GNN (Graph Neural Network) based learning method, which utilizes two representation graphs to encode keypoints detected from images. Self-attention is applied for graph updating and cross-attention is applied for generating manipulation actions. Extensive experiments have been conducted to demonstrate that our framework is effective in multiple 1-D (rope, rope ring) and 2-D (cloth) rearranging tasks in simulation and can be easily transferred to a real robot by fine-tuning a keypoint detector.

Rearranging deformable objects is a long-standing challenge in robotic manipulation for the high dimensionality of configuration space and the complex dynamics of deformable objects. We present a novel framework, Graph-Transporter, for goal-conditioned deformable object rearranging tasks. To tackle the challenge of complex configuration space and dynamics, we represent the configuration space of a deformable object with a graph structure and the graph features are encoded by a graph convolution network. Our framework adopts an architecture based on Fully Convolutional Network (FCN) to output pixel-wise pick-and-place actions from only visual input. Extensive experiments have been conducted to validate the effectiveness of the graph representation of deformable object configuration. The experimental results also demonstrate that our framework is effective and general in handling goal-conditioned deformable object rearranging tasks.

Finger dexterity and tactile perception are key capabilities for humans to manipulate objects within hand, as well as robots. Inspired by the thumb-forefinger dexterous manipulative movement, we devised a novel robotic finger with an active rotational tactile sensor (i.e. TacRot), and mounted the finger on a parallel-jaw gripper. By processing the high-resolution images of the vision-based tactile sensor, we achieved depth reconstruction of the surface and localization of the contact area. To improve gripping flexibility and stability, we applied a self-adaptive grasping strategy with real-time contact detection feedback, which performed 94% success rate in experiment. Based on the rotational actuator at the fingertip, we proposed two in-hand manipulation primitives: (1) pivot: fingertips co-rotating for object reorientation; (2) twist: fingertips contra-rotating for object spin. The primitives are theoretically analyzed and experimentally verified in two practical tasks: pivoting a paper cup under vertical constraints and twisting a screw with spin angle estimation. Our design and experiments demonstrate a feasible way to enhance the active tactile manipulation ability for common parallel-jaw grippers.

Although computer vision has the advantages of long detection distance and large amount of information, it also has certain limitations for complex scenes such as dimness, reflections, and smoke. In order to solve the problem of robot grasping in these scenes, we designed a novel gripper that can search, identify and grasp objects based on tactile information. The gripper can effectively grasp the objects in real life, and can sense the shape and posture of the objects through the touch. We proposed a lifting finger structure that allows the gripper to switch between sensing and grasping modes. We applied visual-tactile detection methods to obtain tactile information and propose a feature extraction algorithm based on U-net. We designed a method of grasping the center of mass of the object contour, and the success rate of the grasping can reach 85%. In addition, we also designed experiments to show the feasibility of object searching and grasping by tactile information when visual information is not available.

Reasonable grasping for unknown objects is an interesting and important problem for autonomous robots in unstructured environment. Current grasping methods for unknown objects mostly focus on precision and stability. Until now there are few specific studies or reports describing reasonable grasp of unknown objects for service robots such as home service robots and nursing robots. In the paper we proposed a reasonable grasping method for unknown objects based on hierarchical decomposition point cloud models using a vision sensor. As a complex task, vision-based grasp is composed of a series solution of a mixture of subproblems. Therefore, we adopt an improved superquadrics fitting algorithm with the improved cuckoo search strategy (S-ICS) to achieve restoration and segmentation of incomplete point cloud data of unknown objects in a single visual angle. Then a reasonable region decision method based on hierarchical decomposition models is proposed to evaluate the reasonableness of grasped positions of unknown objects. Finally, we use a simulation to verify the effectiveness of the proposed method. Moreover, we also perform an extensive real-world grasping experiment on a set of unknown objects in daily use. The results also verify the effectiveness of our approach.