Robot scooping is a challenging and important task in robotic tool manipulation research due to the complex relationship between the robot, the tool, and target objects/environment. Taking into account different tools, different target objects and varying environments, the required scooping manipulation strategy usually varies greatly. Even considering a specific type of spoon, the question of how to obtain a policy model that requires less demonstration data but shows better generalization capabilities deserves further exploration. In this paper, we propose a progressive learning framework for general robot scooping tasks, which requires a limited number of demonstrations but shows promising generalization capability. We first learn a scooping policy via human demonstrations with a specific setup. We then use this as a pre-train model for reinforcement learning in a curriculum manner to achieve a scooping strategy that is generalizable to different task setups. Finally, we evaluate the capabilities of the policy with a series of experiments both in simulation and on a real robot.

Robotic hand tool use has garnered significant attention from robotics researchers, because it enhances dexterity beyond the limitations imposed by manipulators with fixed tool configurations and human-involved manual tool changes. Despite extensive research, current methodologies predominantly focus on imitating human hand trajectories, often neglecting the pivotal role of tool-environment interaction. This study addresses this gap by exploring the task of cucumber peeling as a case study to implement contact-based demonstration strategies in robotic tool use. Our approach concentrates on the subtle tool contact behaviors that manifest through contact dynamics. Specifically, we select appropriate tool stiffness for the peeling tasks, which is captured via a handheld teaching device equipped with optical tactile sensors. Subsequently, object-level stiffness control strategies are employed to emulate these behaviors using a three-fingered robotic hand. Experimental results from real-world cucumber peeling trials substantiate our methodology, illustrating that the robotic hand can adjust contact through finger movements, thereby achieving humanlike peeling efficiency without necessitating alterations to the tool structure. This study not only demonstrates the feasibility of sophisticated tool use by robotic hands, but also highlights the critical importance of integrating tactile feedback to refine interaction with the environment.

Robots and prostheses are increasingly designed with curvilinear surfaces for functional, aesthetic, aerodynamic, and safety reasons. Electronic skins (e-skins) capable of sensing contact location and pressure across complex, non-developable surfaces are essential for empowering next-generation robots with tactile awareness. This will facilitate safe and natural human-machine interactions while enhancing object manipulation capabilities. Despite the evident advantages of conformal e-skins, current fabrication methods face significant challenges in realizing their full potential. In this paper, we introduce thermoforming as a technique to efficiently fabricate tactile sensitive e-skins that conform to curvilinear surfaces. The performance, repeatability and uniformity of the sensors are characterized in detail. We also present a custom calibration pipeline where accurate digital replicas of conformal e-skins are generated for use in simulations. Finally, we demonstrate the benefits of 3D e-skins in a tool manipulation task.

Cable-driven continuum robots are widely used for endoluminal intervention because of their dexterity and shape conforming steerability. However, body contact between the continuum robot and its surrounding anatomy is unavoidable, which imposes a potential safety risk, including vessel wall damage or even perforation. This paper presents an approach for body contact estimation of continuum robots with tension-profile sensing of actuation fibers. First, tension-sensing optical fibers with multiple inscribed fiber Bragg grating (FBG) sensors are used for both actuation and in-situ sensing of the continuum robot. Second, a beam theory-based mechanical model considering segmental differences, multiple fiber interactions and external force interactions is established, followed by robust estimation of contact positions and forces. Finally, detailed simulations are conducted to validate the accuracy and effectiveness of the proposed method. Experiments on a notched continuum robot are carried out, and the results show that the proposed approach can effectively recover in-situ segmental actuation forces without the need of explicit modeling of the friction between the fibers and guiding channels. The method enables the estimation of the number of contact points, as well as contact positions and contact forces along the body of the continuum robot.

This paper proposes a novel efficient multi-phase trajectory generation algorithm for dynamic dexterous manipulation tasks, such as throwing, catching, dynamic regrasping, and dynamic handover, which can be decomposed into multiple manipulation primitives, including sticking, rolling, approaching, separating, colliding, and grasping. Each manipulation primitive is formulate as a free-terminal optimal control problem (OCP), aimed at computing the optimal pose (position and orientation) trajectories of the object and the robot subject to the pose and force linkage constraints between them and the expected force maintenance at contact. A single-arm regrasping task and a dual-arm dynamic handover task are conducted to demonstrate the effectiveness of the proposed algorithm.

Grasping of objects using multi-fingered robotic hands often fails due to small uncertainties in the hand motion control and the object's pose estimation. To tackle this problem, we propose a grasping adjustment strategy based on tactile seroving. Our technique employs feedback from a sensorized multi-fingered robotic hand to collaboratively servo the fingers and palm to achieve the desired grasp. We demonstrate the performance of our method through simulation and physical experiments by having a robot grasp different objects under conditions of variable uncertainty. The results show that our approach achieved a higher success rate and tolerated greater uncertainty than an open-looped grasp.

Intercepting an object in flight through nonpre-hensile manipulation is a challenging problem, which is aimed at catching and stopping a flying object using little contacts without completely restraining its relative motion to the robot. This paper presents a two-stage optimal trajectory generation method to tackle this problem. At the pre-catching stage, optimal position and attitude trajectories of the robot's end-effector to approach the object are generated by a variational method. At the post-catching stage, the end-effector's trajectories are generated to optimally eliminate the translational and rotational motion of the object and a convex-MPC algorithm combined with admittance control is used to realize the trajectory tracking. A series of simulations and experiments have been conducted to verify the effectiveness of the proposed method.

Most robotic systems designed for mass manufacturing are optimized for a specific type of product. They generally lack the ability to adapt to low-volume customized products. In this article, we present a system based on a modular design for manufacturing personalized medical stent graft implants. The concept is based on learning-by-demonstration by integrating real-time 3-D vision, multirobot collaboration, and personalization to guide the robots to learn and execute tasks continuously with adaptation to different implant geometry. The system is optimized to generate customized and collision-free paths for efficient object manipulation and task completion. We show that the system is generalizable to different stent graft designs and the proposed multirobot system can seamlessly work together with high efficiency without collisions. The results have also suggested its usability for other manipulation tasks, especially for flexible production of customized products where bimanual or multirobot cooperation is required.Note to Practitioners—The motivation of this article is the problem of automatic sewing of personalized stent grafts (a tailor-made artificial vessel). Existing personalized stent grafts are mostly hand sewn, which is time consuming and often undersupplied. Automating such process can significantly improve the production and this requires a sewing system that can handle different designs. This article suggests a new schema to design a robotic system to handle personalized designs. The first methodology is modularized design to separate the task into a repetitive part and a personalized part, each handled by a module. The second methodology is to find the best relative pose between the modules such that the robots can complete their task within their working space and with minimum motion. This ensures that a stent graft can be sewn feasibly and with the lowest cost. Computational results show this approach can find optimal solution for different personalized stent grafts and preliminary on robot experiment verifies that this approach is feasible. Please note that this approach is not limited to sewing personalized stent graft. The schema can be applied to solve similar problem of customized product motion planning and system design.

Reinforcement Learning (RL) methods have been widely applied for robotic manipulations via sim-to-real transfer, typically with proprioceptive and visual information. However, the incorporation of tactile sensing into RL for contact-rich tasks lacks investigation. In this paper, we model a tactile sensor in simulation and study the effects of its feedback in RL-based robotic control via a zero-shot sim-to-real approach with domain randomization. We demonstrate that learning and controlling with feedback from tactile sensor arrays at the gripper, both in simulation and reality, can enhance grasping stability, which leads to a significant improvement in robotic manipulation performance for a door opening task. In real-world experiments, the door open angle was increased by 45% on average for transferred policies with tactile sensing over those without it.

In this paper, we present a novel approach for surgical task segmentation. A segmentation policy learns the correlations between features and segmentation points from manually labeled data. The most correlated features and rules for segmenting them are identified and learned. These form a complete set of segmentation policy. The proposed approach is developed to segment new but similar tasks through transfer learning. It is verified through applying the segmentation rule learned from the labeled data to segment other tasks. The performance of the proposed algorithm was evaluated by comparing the results against the ground truths. Experimental results demonstrate that our approach can achieve high segmentation rates with an accuracy of between 68.8% - 81.8%.

This paper presents a multirobot system for manufacturing personalized medical stent grafts. The proposed system adopts a modular design, which includes a (personalized) mandrel module, a bimanual sewing module, and a vision module. The mandrel module incorporates the personalized geometry of patients, while the bimanual sewing module adopts a learning-by-demonstration approach to transfer human hand-sewing skills to the robots. The human demonstrations were first observed by the vision module and then encoded using a statistical model to generate the reference motion trajectories. During autonomous robot sewing, the vision module plays the role of coordinating multirobot collaboration. Experimental results show that the robots can adapt to generalized stent designs. The proposed system can also be used for other manipulation tasks, especially for flexible production of customized products and where bimanual or multirobot cooperation is required.

This paper presents a vision-based learning-by-demonstration approach for multi-robot manipulation. With this method, a vision system is involved in both the task demonstration and reproduction stages, and the speed and accuracy of the task reproduction are adapted according to the context of the demonstration. An expert first demonstrates how to use tools to perform a task, while the tool motion is observed using a vision system. The demonstrations are then encoded using a statistical model to generate a reference motion trajectory. Equipped with the same tools and the learned model, the robot is guided by vision to reproduce the task. The task performance was evaluated in terms of both accuracy and speed. However, simply increasing the robot's speed could decrease the reproduction accuracy. To this end, a dual-rate Kalman filter is employed to compensate for latency between the robot and vision system. More importantly, the robot speed is adapted according to the learned motion model. We demonstrate the effectiveness of our approach by performing two tasks: a trajectory reproduction task and a bimanual sewing task. We show that using our vision-based approach, the robots can conduct effective learning by demonstrations and perform accurate and fast task reproduction. The proposed approach is generalisable to other manipulation tasks, where bimanual or multi-robot cooperation is required.

This paper presents an intelligent sewing system for personalized stent graft manufacturing, a challenging sewing task that is currently performed manually. Inspired by medical suturing robots, we have adopted a single-sided sewing technique using a curved needle to perform the task of sewing stents onto fabric. A motorized surgical needle driver was attached to a 7 d.o.f robot arm to manipulate the needle with a second robot controlling the position of the mandrel. A learning-from-demonstration approach was used to program the robot to sew stents onto fabric. The demonstrated sewing skill was segmented to several phases, each of which was encoded with a Gaussian Mixture Model. Generalized sewing movements were then generated from these models and were used for task execution. During execution, a stereo vision system was adopted to guide the robots and adjust the learnt movements according to the needle pose. Two experiments are presented here with this system and the results show that our system can robustly perform the sewing task as well as adapt to various needle poses. The accuracy of the sewing system was within 2mm.

A fundamental problem with dual-arm robotic control is to find the coordinated motion resolution under high kinematic redundancy and intrinsic constraints of each robot. To solve this problem, this paper presents a multi-tasking, co-operative control framework, in which potential task conflicts and robot joint constraints are properly handled. Based on the relative Jacobian formulation, singularity-robust inverse kinematics and the scheme of null space distributing exceeded joint velocity, this work contributes by introducing a framework to handle multi-tasking conflicts both in task and joint space for dual-arm robots. Detailed validation of the proposed framework is first conducted by using a simulated dual-arm robot, followed by a demonstration on two 7-dof Kuka lightweight manipulators in a bimanual stent graft manufacturing task.

Real time planning strategy is crucial for robots working in dynamic environments. In particular, robot grasping tasks require quick reactions in many applications such as human-robot interaction. In this paper, we propose an approach for grasp learning that enables robots to plan new grasps rapidly according to the object's position and orientation. This is achieved by taking a three-step approach. In the first step, we compute a variety of stable grasps for a given object. In the second step, we propose a strategy that learns a probability distribution of grasps based on the computed grasps. In the third step, we use the model to quickly generate grasps. We have tested the statistical method on the 9 degrees of freedom hand of the iCub humanoid robot and the 4 degrees of freedom Barrett hand. The average computation time for generating one grasp is less than 10 milliseconds. The experiments were run in Matlab on a machine with 2.8GHz processor.