Supernumerary robotic limbs (SRLs) can significantly enhance human manipulation capability. However, it is difficult to achieve a safe and friendly interaction for collaboration tasks in complicated and dynamic external environments, and it easily leads to safety issues between a human user and the SRL robot. To address the aforementioned issues, this paper designs a new type of SRL and proposes a teleoperated control method for human-robot safe interaction. Specifically, inspired by the principle that human arms can autonomously adjust their stiffness to safely interact with the external environment, this paper proposes an SRL interaction control method based on the variable stiffness of the human upper limb. Specifically, the proposed variable stiffness control parameters can be adaptively updated based on the characteristics of surface electromyography (sEMG) signals from the human upper limb. The effectiveness of the proposed framework is validated through experimental results.

Physical human-robot interaction (pHRI) plays an important role in robotic. In order for a human operator to be able to easily adapt to interact with a robot, a minimal interaction force in pHRI should be achieved. In this paper, a pHRI framework is proposed to allow the robot to regulate its trajectory adaptively for minimizing the interaction force with small position-tracking errors. The trajectory of the robot is first adjusted by the interaction force which is updated by the performance evaluation index. Then, the human hand motion is predicted based on the autoregressive (AR) model to further adapt the trajectory. Thirdly, an adaptive impedance control method is developed to update the stiffness in the robot impedance controller using surface electromyography (sEMG) signals for robot compliant interaction with the environment. This method allows the human operator to interact with the robot by the interaction force, the hand motion and muscle contraction. By investigating the performance of the proposed method, the interaction force is decreased and a good position tracking accuracy is achieved. Comparative experiments demonstrate the enhanced performance of the proposed method. Note to Practitioners—This paper focuses on developing a novel method that can allow the robot to compliantly interact with the human operator while simultaneously taking into account the trajectory-tracking accuracy and the interaction force in pHRI scenarios. The proposed method has a large application potential in a variety of pHRI tasks, such as human-robot collaborative transporting, curing, assembly, cutting, and so on. In addition, the proposed method can allow the human operator to physically interact with the robot in an easier and more intuitive manner, by taking advantage of human motion prediction and adaptive impedance control. Therefore, it is also potentially utilized for rehabilitation and assistive robots, and robot learning skills from human physical demonstration.

As robots grow increasingly intelligent and utilize data from various sensors, relying solely on unimodal data sources is becoming inadequate for their operational needs. Consequently, integrating multimodal data has emerged as a critical area of focus. However, the effective combination of different data modalities poses a considerable challenge, especially in complex and dynamic settings where accurate object recognition and manipulation are essential. In this paper, we introduce a novel framework integrating with Multimodal Information for Grasping Synthesis with vision-language models (MIG) designed to improve robotic grasping capabilities. This framework incorporates visual data, textual information, and human-derived prior knowledge. We start by creating target object masks based on this prior knowledge, which are then used to segregate the target objects from their surroundings in the image. Subsequently, we employ language cues to refine the visual representations of these objects. Finally, our system executes precise grasping actions using visual and textual data synthesis, thus facilitating more effective and contextually aware robotic grasping. We carry out experiments using the OCID-VLG dataset. We observe that our methodology surpasses current state-of-the-art (SOTA) techniques, delivering improvements of 9.91% and 5.70% for top-1 and top-5 predictions in grasp accuracy. Moreover, when apply to the reconstructed Grasp-MultiObject dataset, our approach demonstrates even more substantial enhancements, achieving gains of 17.63% and 22.76% over SOTA methods for top-1 and top-5 predictions, respectively. Note to Practitioners—As robotic systems evolve, the challenge of enabling them to function effectively in complex environments has become increasingly apparent. This paper introduces a solution that integrates multiple sources of data—visual, textual, and human knowledge—to enhance robotic grasping capabilities. The practical problems addressed include the limitations of current unimodal systems that struggle with accurate object recognition and manipulation in dynamic settings, such as warehouses or assembly lines. Our framework, MIG, demonstrates significant improvements in grasp accuracy, making it suitable for tasks where precision is critical. While our results show promise, particularly in controlled experiments, there are limitations to consider. The framework’s performance may vary in unstructured real-world environments due to factors like occlusion or varying lighting conditions. Future work should focus on refining the system for real-time application and exploring additional sensory inputs to enhance robustness. By addressing these challenges, we aim to make this approach more applicable across industries, paving the way for smarter, more adaptable robotic solutions in everyday tasks.

Supernumerary robotic limb (SRL) is recognized as being at the forefront of robotics innovation, aimed at augmenting human capabilities in complex working environments. Despite their potential to significantly enhance operational efficiency, the integration of SRL for dynamic and intricate tasks presents challenges in teleoperation, precise positioning, and dynamic balance control. To address challenges in initiating control when targets or the SRL’s end-effector are outside the camera’s visual range, a coarse teleoperation strategy is implemented. This strategy utilizes the inertial measurement unit (IMU) and the extended Kalman filter (EKF), enabling basic orientation and movement toward the target area without reliance on visual cues. Challenges in achieving fine-tuned control for accurate task completion, particularly in visual navigation and precise positioning of the SRL’s end-effector, are addressed by integrating object detection via YOLOX with the tangential artificial potential field (T-APF) method for exact path planning. This integration significantly enhances the system’s ability to fine-tune the placement of end-effector. The challenge of conducting balance tasks without force sensors is tackled by adopting a dual-spring model combined with autoregressive (AR) predictive modeling, enabling effective balance support through anticipatory motion adjustments. Experiments have demonstrated the system’s enhanced positional accuracy and maintained synchronization with human movements, underscoring the effectiveness of the integrated approach in facilitating complex human-robot collaborative tasks.

Abstract In robot-assisted rehabilitation, the performance of robotic assistance is dependent on the human user’s dynamics, which are subject to uncertainties. In order to enhance the rehabilitation performance and in particular to provide a constant level of assistance, we separate the task space into two subspaces where a combined scheme of adaptive impedance control and trajectory learning is developed. Human movement speed can vary from person to person and it cannot be predefined for the robot. Therefore, in the direction of human movement, an iterative trajectory learning approach is developed to update the robot reference according to human movement and to achieve the desired interaction force between the robot and the human user. In the direction normal to the task trajectory, human’s unintentional force may deteriorate the trajectory tracking performance. Therefore, an impedance adaptation method is utilized to compensate for unknown human force and prevent the human user drifting away from the updated robot reference trajectory. The proposed scheme was tested in experiments that emulated three upper-limb rehabilitation modes: zero interaction force, assistive and resistive. Experimental results showed that the desired assistance level could be achieved, despite uncertain human dynamics.

In this work, a human motion intention prediction method based on an autoregressive (AR) model for teleoperation is developed. Based on this method, the robot’s motion trajectory can be updated in real time through updating the parameters of the AR model. In the teleoperated robot’s control loop, a virtual force model is defined to describe the interaction profile and to correct the robot’s motion trajectory in real time. The proposed human motion prediction algorithm acts as a feedforward model to update the robot’s motion and to revise this motion in the process of human–robot interaction (HRI). The convergence of this method is analyzed theoretically. Comparative studies demonstrate the enhanced performance of the proposed approach. Note to Practitioners—In general, the robot trajectory is predetermined and it does not consider the influence of the interaction profiles in terms of position and interaction force between the human and the robot. In addition, it is hard to quantify the influence of interaction profile for the robot trajectory. For teleoperation, an AR-based model is proposed to predict the trajectory of the human and then to update the trajectory of the robot. The developed method includes the following aspects: 1) the robot trajectory can be regulated based on the interaction profiles; 2) the feedforward model can estimate the trajectory of the human to achieve the purpose of human intention recognition in advance for the robot; and 3) the proposed method can be potentially utilized for telerehabilitation, microsurgery, and so on.

Due to rich spectral information, hyperspectral images (HSIs) have been widely used in various fields. However, limited by imaging systems, the low spatial resolution of HSIs has become an important problem. In this article, for enhancing the spatial resolution, Interactformer is proposed to interact with global and local features extracted by Transformer and 3D convolutional neural network (CNN) branches. Within the Transformer branch, a separable self-attention module with linear complexity is designed to solve the problem that traditional self-attention mechanisms suffer from large memory costs due to quadratic complexity. In the 3D CNN branch, the spectral attention module and 3D convolution are applied jointly to better protect the spectral correlation among spectral bands and facilitate local feature extraction of HSIs. The interactive attention unit between the two parallel branches is designed to interact with local and global feature information adaptively. Compared with state-of-the-art super-resolution (SR) methods, the proposed method reconstructs better HSI in simulated SR experiments, real SR experiments, and classification experiments, which prove that Interactformer can effectively improve the spatial resolution while preserving the spectral information.

In human-robot collaborative transportation and sawing tasks, the human operator physically interacts with the robot and directs the robot's movement by applying an interaction force. The robot needs to update its control strategy to adapt to the interaction with the human and to minimize the interaction force. To this end, we propose an integrated algorithm of robot's trajectory adaptation and adaptive impedance control to minimize the interaction force in physical humanrobot interaction (pHRI) and to guarantee the performance of the collaboration tasks. We firstly utilize the information of the interaction force to regulate the robot's reference trajectory. Then, an adaptive impedance controller is developed to ensure automatic adaptation of the robot's impedance parameters. While one can reduce the interaction force by using either trajectory adaptation or adaptive impedance control, we investigate the task performance when combining both. Experimental results on a planar robotic platform verify the effectiveness of the proposed method.

Due to the lack of transparent and friendly human-robot interaction (HRI) interface, as well as various uncertainties, it is usually a challenge to remotely manipulate a robot to accomplish a complicated task. To improve the teleoperation performance, we propose a new perception mechanism by integrating a novel learning method to operate the robots in the distance. In order to enhance the perception of the teleoperation system, we utilize a surface electromyogram signal to extract the human operator's muscle activation. As a response to the changes in the external environment, as sensed through haptic and visual feedback, a human operator naturally reacts with various muscle activations. By imitating the human behaviors in task execution, not only motion trajectory but also arm stiffness adjusted by muscle activation, it is expected that the robot would be able to carry out the repetitive tasks autonomously or uncertain tasks with improved intelligence. To this end, we develop a robot learning algorithm based on probability statistics under an integrated framework of the hidden semi-Markov model (HSMM) and the Gaussian mixture method. This method is employed to obtain a generative task model based on the robot's trajectory. Then, Gaussian mixture regression based on HSMM is applied to correct the robot trajectory with the reproduced results from the learned task model. The execution procedures consist of a learning phase and a reproduction phase. To guarantee the stability, immersion, and maneuverability of the teleoperation system, a variable gain control method that involves electromyography (EMG) is introduced. Experimental results have demonstrated the effectiveness of the proposed method.

Teleoperated robot systems are able to support humans to accomplish their tasks in many applications. However, the performance of teleoperation largely depends on motor functionality and human operator's skill, especially when a human operator is short of skill training. In order to adapt to various unstructured environments for the robot system and the human operator, in this paper, a teleoperation scheme using integrated tremor attenuation with a variable gain control algorithm involving surface electromyogram is proposed to achieve personalized control performance and to reduce reliance on operator's skill. For attenuating tremor, a filter based on support vector machine is developed to guarantee normal operation. This filter depends on the machine learning scheme and does not rely on a priori filter parameters. Semiphysical experiments have been performed to demonstrate the effectiveness of the proposed methods.