The 6-degree-of-freedom (DoF) grasp method based on point clouds has shown significant potential in enabling robots to grasp target objects. However, most existing methods are based on the point clouds (2.5-D points) generated from single-view depth images. These point clouds only have one surface side of the object, providing incomplete geometry information, which misleads the grasping algorithm to judge the shape of the target object, resulting in low grasping accuracy. Humans can accurately grasp objects from a single view by leveraging their geometry experience to estimate object shapes. Inspired by humans, we propose a novel 6-DoF grasping framework that converts the point completion results as object shape features to train the 6-DoF grasp network. Here, point completion can generate approximately complete points from the 2.5-D points similar to the human geometry experience, and converting them into shape features is the way to utilize it to improve grasp efficiency. Furthermore, due to the gap between the network generation and actual execution, we integrate a score filter into our framework to select more executable grasp proposals for the real robot. This enables our method to maintain a high grasp quality in any camera viewpoint. Extensive experiments demonstrate that utilizing complete point features enables the generation of significantly more accurate grasp proposals, and the inclusion of a score filter greatly enhances the credibility of real-world robot grasping. Our method achieves a 17.8% success rate, higher than the state-of-the-art method in real-world experiments. Code and videos are available at https://github.com/ChengYaofeng/PCF-Grasp

With the growing popularity of electric vehicles, the demand for robot-based unmanned automatic charging has become both urgent and challenging. Two key challenges need to be addressed: how to efficiently locate the charging port, and how to compliantly insert the connector into the port. In this paper, we propose an incremental learning method based on the broad learning system to address the visual positioning error of the charging port. This method allows the robot to transfer and generalize the search skills learned in simulation to real-world scenarios. As a result, the robot can rapidly locate the charging port in real-world environments without the need for complex contact state modeling, time-consuming data collection, or model retraining. Subsequently, a biomimetic admittance controller is designed to enable the robot to adapt its compliant behavior online during the plugging process. Finally, experiments are performed on a UR robot to verify the effectiveness of our method.

In contact-rich tasks such as polishing and drilling, inevitable physical interactions often lead to task deviations due to interference, typically resulting in excessive contact forces and eventual task failure. To tackle these challenges, we propose an innovative human-guided visual-impedance control framework. Specifically, we first introduce an interaction model within image feature space, which models the dynamics of human-robot-environment interactions. Subsequently, human operation skills are characterized through human-guided wrenches, and acts on visual features through a projection matrix, thus integrating human-guided wrenches with visual-impedance interaction dynamics. Finally, leveraging this framework, we develop a novel variable-stiffness visual-impedance control strategy. The impedance parameters are optimized online via Quadratic Program, ensuring that the end-tool contact force converges to desired value while adhering to safety constraints. The validity of the proposed framework was established through polish experiments.

Due to non-destructive testing (NDT) techniques being both expensive and inconvenient in dynamic detection scenarios, innovative alternatives are urgently needed to address cost-efficiency and deployment challenges. We first design TacRoller, a tactile sensor roller for automated characterization of surface defects in composite materials, to address the dilemma. It collects tactile images of defects on the composite’s plies by capturing changes caused by deformation of the outer elastomer through the internal camera. It reduces the cost of inspection by 80% to 90% compared to NDT equipment like radiographic testing while ensuring detection efficiency. It takes 58.86 seconds to complete a 35 cm×18 cm × 0.5 mm dry-woven fabric. Moreover, we collect a total of 2,744 images of samples of dry-woven fabric unidirectional prepreg through TacRoller to form a dataset, including wrinkles, foreign objects and debris (FODs), broken fibre, voids and healthy textures. Subsequently, we propose a multi-order gated aggregation (MOGA)-U-Net to tackle critical challenges of noise sensitivity and multi-scale defect recognition in tactile images, enabling robust segmentation and multi-category classification tasks. The results show that the MOGA-U-Net achieves a test dice coefficient of 76.0% and classification accuracy of 98.9%, outperforming DeepLabV3 and other benchmarks. By providing a scalable and effective NDT substitute, our system realises autonomous defect identification and classification on composites surface, thus improving quality control in the production of composites.

Supernumerary robotic limbs (SRLs) can significantly enhance human manipulation capability. However, it is difficult to achieve a safe and friendly interaction for collaboration tasks in complicated and dynamic external environments, and it easily leads to safety issues between a human user and the SRL robot. To address the aforementioned issues, this paper designs a new type of SRL and proposes a teleoperated control method for human-robot safe interaction. Specifically, inspired by the principle that human arms can autonomously adjust their stiffness to safely interact with the external environment, this paper proposes an SRL interaction control method based on the variable stiffness of the human upper limb. Specifically, the proposed variable stiffness control parameters can be adaptively updated based on the characteristics of surface electromyography (sEMG) signals from the human upper limb. The effectiveness of the proposed framework is validated through experimental results.

Physical human-robot interaction (pHRI) plays an important role in robotic. In order for a human operator to be able to easily adapt to interact with a robot, a minimal interaction force in pHRI should be achieved. In this paper, a pHRI framework is proposed to allow the robot to regulate its trajectory adaptively for minimizing the interaction force with small position-tracking errors. The trajectory of the robot is first adjusted by the interaction force which is updated by the performance evaluation index. Then, the human hand motion is predicted based on the autoregressive (AR) model to further adapt the trajectory. Thirdly, an adaptive impedance control method is developed to update the stiffness in the robot impedance controller using surface electromyography (sEMG) signals for robot compliant interaction with the environment. This method allows the human operator to interact with the robot by the interaction force, the hand motion and muscle contraction. By investigating the performance of the proposed method, the interaction force is decreased and a good position tracking accuracy is achieved. Comparative experiments demonstrate the enhanced performance of the proposed method. Note to Practitioners—This paper focuses on developing a novel method that can allow the robot to compliantly interact with the human operator while simultaneously taking into account the trajectory-tracking accuracy and the interaction force in pHRI scenarios. The proposed method has a large application potential in a variety of pHRI tasks, such as human-robot collaborative transporting, curing, assembly, cutting, and so on. In addition, the proposed method can allow the human operator to physically interact with the robot in an easier and more intuitive manner, by taking advantage of human motion prediction and adaptive impedance control. Therefore, it is also potentially utilized for rehabilitation and assistive robots, and robot learning skills from human physical demonstration.

Imitation learning allows us to directly encode manipulation skills based on human demonstrations, facilitating rapid transfer of skills without any expert knowledge. Autonomous dynamic systems (DS) offer reliable stability and time-independence though sacrificing part of accuracy, and are increasingly attractive as an encoding method. As unstructured environments become more challenging, skill trajectories become more complex, and various disturbances are encountered, existing state-of-the-art encoding methods struggle to adapt to these complex tasks. This paper introduces a novel robust DS-based framework for learning skills in complex tasks, which consists of trajectory regularization, adaptive segmentation, skill modeling, and skill organization based on new task requirements. It achieves a task-level generalization so that the operator only needs to focus on the semantic deconstruction of a task. Additionally, we propose an online modulation policy for the skill decision engine to address two types of disturbances: enhancing convergence speed for large-scale disturbances and improving fitting capability for small-scale disturbances while still keeping stability. To evaluate the effectiveness of the proposed framework, we conduct various comparison experiments in simulation and a real-world sugar-scooping task to assess the generalization performance and the ability of resistance to disturbances.Note to Practitioners—Imitation learning for complex tasks is crucial to the development of robot intelligence. However, achieving a balance between maintaining generalization accuracy and robustness remains a challenging problem that necessitates continuous exploration in the field of imitation learning. The purpose of this paper is to propose a robust imitation learning framework from human demonstration, which includes preprocessing, learning and generalization, that can be applied in industrial production or daily life. Considering appropriate trajectory segmentation and self-organization strategies can effectively improve the generalization accuracy by prior research, it is necessary to introduce them into our framework. Most importantly, we design novel disturbance-resistant online modulation strategies from both task-level and motion-level aspects. To validate the effectiveness of our approach, we conduct simulations and coffee scooping experiments. The results show that skills acquired through demonstration can reliably, accurately, and safely perform tasks even in uncertain environments. This paper is a systematic and pioneering attempt to implement.

Imitation learning is an important method for the human-robot skill transfer. However, ensuring that skills learned through imitation remain effective in different environments is a challenge. This article addresses the challenge by proposing a stable autonomous dynamic system that can effectively handle obstacles and disturbances while maintaining trajectory accuracy. We introduce an energy-approximated dynamic subattractor (EADA) method that enhances disturbance resistance by dynamically selecting subattractors through Neum (an energy function derived from demonstration data). By combining velocity modulation algorithms with EADA, the system achieves global stability, precise obstacle avoidance, autonomous trajectory recovery, and rapid response. The proposed framework effectively handles complex scenarios, including environments with multiple obstacles, dynamic obstacles, and disturbances. We validate the proposed approach through simulations on the LASA dataset and real-world robotic experiments (both single-arm and dual-arm robots), demonstrating its effectiveness in achieving smooth and accurate obstacle avoidance trajectories with generalization capability.

Supernumerary robotic limb (SRL) is recognized as being at the forefront of robotics innovation, aimed at augmenting human capabilities in complex working environments. Despite their potential to significantly enhance operational efficiency, the integration of SRL for dynamic and intricate tasks presents challenges in teleoperation, precise positioning, and dynamic balance control. To address challenges in initiating control when targets or the SRL’s end-effector are outside the camera’s visual range, a coarse teleoperation strategy is implemented. This strategy utilizes the inertial measurement unit (IMU) and the extended Kalman filter (EKF), enabling basic orientation and movement toward the target area without reliance on visual cues. Challenges in achieving fine-tuned control for accurate task completion, particularly in visual navigation and precise positioning of the SRL’s end-effector, are addressed by integrating object detection via YOLOX with the tangential artificial potential field (T-APF) method for exact path planning. This integration significantly enhances the system’s ability to fine-tune the placement of end-effector. The challenge of conducting balance tasks without force sensors is tackled by adopting a dual-spring model combined with autoregressive (AR) predictive modeling, enabling effective balance support through anticipatory motion adjustments. Experiments have demonstrated the system’s enhanced positional accuracy and maintained synchronization with human movements, underscoring the effectiveness of the integrated approach in facilitating complex human-robot collaborative tasks.

Robotic rigid contact-rich manipulation in an unstructured dynamic environment requires an effective resolution for smart manufacturing. As the most common use case for the intelligence industry, a lot of studies based on reinforcement learning (RL) algorithms have been conducted to improve the performances of single peg-in-hole assembly. However, existing RL methods are difficult to apply to multiple peg-in-hole issues due to more complicated geometric and physical constraints. In addition, previously limited solutions for multiple peg-in-hole assembly are hard to transfer into real industrial scenarios flexibly. To effectively address these issues, this work designs a novel and more challenging multiple peg-in-hole assembly setup by using the advantage of the Industrial Metaverse. We propose a detailed solution scheme to solve this task. Specifically, multiple modalities, including vision, proprioception, and force/torque, are learned as compact representations to account for the complexity and uncertainties and improve the sample efficiency. Furthermore, RL is used in the simulation to train the policy, and the learned policy is transferred to the real world without extra exploration. Domain randomization and impedance control are embedded into the policy to narrow the gap between simulation and reality. Evaluation results demonstrate the effectiveness of the proposed solution, showcasing successful multiple peg-in-hole assembly and generalization across different object shapes in real-world scenarios.

One promising function of interactive robots is to provide a specific interaction force to human users. For example, rehabilitation robots are expected to promote patients' recovery by interacting with them with a prescribed force. However, motion uncertainties of different individuals, which are hard to predict due to the varying motion speed and noises during motion, degrade the performance of existing control methods. This article proposes a method to learn a desired reference trajectory for a robot based on dynamic motion primitives (DMPs) and iterative learning (IL). By controlling the robot to follow the generated desired reference trajectory, the interaction force can achieve a desired value. In our proposed approach, DMPs are first employed to parameterize the demonstration trajectories of the human user. Then, a recursive least square (RLS)-based estimator is developed and combined with the Adam optimization method to update the trajectory parameters so that the desired reference trajectory of the robot is iteratively obtained by resolving the DMPs. Since the proposed method parameterizes the trajectories depending on the phase variable, it removes the essential assumption of traditional IL methods that the iteration period should be invariant, and thus, has improved robustness compared with the existing methods. Experiments are performed using an interactive robot to validate the effectiveness of our proposed scheme.

This paper proposes a whole-body learning from demonstration (LfD) framework that enables differential drive mobile manipulators to learn coordination working and disturbance rejection. First, an efficient kinesthetic teaching method is devised based on the weighted least-norm (WLN) inverse kinematics solution and an admittance controller, which facilitates human users to guide the mobile manipulator to perform tasks. Second, we propose a whole-body LfD framework through Gaussian Process, which endows the mobile manipulator's skill learning process with features of large-scale convergence, coordination working and disturbance rejection, after just a few human demonstrations. The proposed learning framework also allows for human-in-the-loop correction when the whole-body is conducting a task. Finally, the effectiveness of the proposed framework is verified via two simulations and a pick-and-place experiment. Supplementary video for this paper is available in github††https://github.com/yuqiang-yang/SMC2023-Video.

Multifingered hand dexterous manipulation is quite challenging in the domain of robotics. One remaining issue is how to achieve compliant behaviors. In this work, we propose a human-in-the-loop learning-control approach for acquiring compliant grasping and manipulation skills of a multifinger robot hand. This approach takes the depth image of the human hand as input and generates the desired force commands for the robot. The markerless vision-based teleoperation system is used for the task demonstration, and an end-to-end neural network model (i.e., TeachNet) is trained to map the pose of the human hand to the joint angles of the robot hand in real-time. To endow the robot hand with compliant human-like behaviors, an adaptive force control strategy is designed to predict the desired force control commands based on the pose difference between the robot hand and the human hand during the demonstration. The force controller is derived from a computational model of the biomimetic control strategy in human motor learning, which allows adapting the control variables (impedance and feedforward force) online during the execution of the reference joint angles. The simultaneous adaptation of the impedance and feedforward profiles enables the robot to interact with the environment compliantly. Our approach has been verified in both simulation and real-world task scenarios based on a multifingered robot hand, that is, the Shadow Hand, and has shown more reliable performances than the current widely used position control mode for obtaining compliant grasping and manipulation behaviors.

How to enable robotic compliant manipulation has become a critical problem in the robotics field. Inspired by a biomimetic adaptive control strategy, this article presents a novel representation model named human-like compliant movement primitives (Hl-CMPs) which could allow a robot to learn human-like compliant behaviors. The state-of-the-art approaches can hardly learn complete compliant profiles for a specific task. Comparatively, our model can encode task-specific parametric movement trajectories, correspondingly associated with dynamic trajectories including both impedance and feedforward force profiles. The compliant profiles are learned based on a biomimetic control strategy derived from the human motor learning in the muscle space, enabling the robot to simultaneously learn the impedance and the force while executing the movement trajectories obtained from human demonstration. Furthermore, both the kinematic and the dynamic profiles are learned in the parametric space, thus enabling the representation of a skill using corresponding parameters (i.e, task-specific parameters). Hl-CMps can allow the robot to automatically learn compliant behaviors in an online manner after kinematic demonstration. Our approach is validated by an insertion task and a cutting task based on a KUKA LBR iiwa robot.

In this work, we focus on improving the robot’s dexterous capability by exploiting visual sensing and adaptive force control. TeachNet, a vision-based teleoperation learning framework, is exploited to map human hand postures to a multi-fingered robot hand. We augment TeachNet, which is originally based on an imprecise kinematic mapping and position-only servoing, with a biomimetic learning-based compliance control algorithm for dexterous manipulation tasks. This compliance controller takes the mapped robotic joint angles from TeachNet as the desired goal, computes the desired joint torques. It is derived from a computational model of the biomimetic control strategy in human motor learning, which allows adapting the control variables (impedance and feedforward force) online during the execution of the reference joint angle trajectories. The simultaneous adaptation of the impedance and feedforward profiles enables the robot to interact with the environment in a compliant manner. Our approach has been verified in multiple tasks in physics simulation, i.e., grasping, opening-a-door, turning-a-cap, and touching-a-mouse, and has shown more reliable performances than the existing position control and the fixed-gain-based force control approaches.

Transferring human stiffness regulation strategies to robots enables them to effectively and efficiently acquire adaptive impedance control policies to deal with uncertainties during the accomplishment of physical contact tasks in an unstructured environment. In this article, we develop such a physical human-robot interaction system which allows robots to learn variable impedance skills from human demonstrations. Specifically, the biological signals, i.e., surface electromyography are utilized for the extraction of human arm stiffness features during the task demonstration. The estimated human arm stiffness is then mapped into a robot impedance controller. The dynamics of both movement and stiffness are simultaneously modeled by using a model combining the hidden semi-Markov model and the Gaussian mixture regression. More importantly, the correlation between the movement information and the stiffness information is encoded in a systematic manner. This approach enables capturing uncertainties over time and space and allows the robot to satisfy both position and stiffness requirements in a task with modulation of the impedance controller. The experimental study validated the proposed approach.

Robots are often required to generalize the skills learned from human demonstrations to fulfil new task requirements. However, skill generalization will be difficult to realize when facing with the following situations: the skill for a complex multistep task includes a number of features; some special constraints are imposed on the robots during the process of task reproduction; and a completely new situation quite different with the one in which demonstrations are given to the robot. This work proposes a new framework to facilitate robot skill generalization. The basic idea lies in that the learned skills are first segmented into a sequence of subskills automatically, then each individual subskill is encoded and regulated accordingly. Specifically, we adapt each set of the segmented movement trajectories individually instead of the whole movement profiles, thus, making it more convenient for the realization of skill generalization. In addition, human limb stiffness estimated from surface electromyographic signals is considered in the framework for the realization of human-to-robot variable impedance control skill transfer, as well as the generalization of both movement trajectories and stiffness profiles. Experimental study has been performed to verify the effectiveness of the proposed framework.

It has been established that the transfer of human adaptive impedance is of great significance for physical human-robot interaction (pHRI). By processing the electromyography (EMG) signals collected from human muscles, the limb impedance could be extracted and transferred to robots. The existing impedance transfer interfaces rely only on visual feedback and, thus, may be insufficient for skill transfer in a sophisticated environment. In this paper, physical haptic feedback mechanism is introduced to result in muscle activity that would generate EMG signals in a natural manner, in order to achieve intuitive human impedance transfer through a designed coupling interface. Relevant processing methods are integrated into the system, including the spectral collaborative representation-based classifications method used for hand motion recognition; fast smooth envelop and dimensionality reduction algorithm for arm endpoint stiffness estimation. The tutor's arm endpoint motion trajectory is directly transferred to the robot by the designed coupling module without the restriction of hands. Haptic feedback is provided to the human tutor according to skill learning performance to enhance the teaching experience. The interface has been experimentally tested by a plugging-in task and a cutting task. Compared with the existing interfaces, the developed one has shown a better performance.