Integrating Large Language Models (LLMs) into modern robotic systems presents significant computational and energy constraint challenges, particularly for human-centered robotic applications. This paper presents a novel hardware optimization technique for deploying LLMs on resource-constrained embedded devices, achieving an up to 77% reduction in computational latency through an FPGA implementation in comparison to other popular embedded computing devices (e.g., CPU and GPUs). Additionally, we demonstrate our methodology by deploying a LLaMA 2-7B model on a Unitree Go2 robotic dog integrated with the proposed FPGA platform. The proposed optimization framework preserves real-time interaction capabilities while significantly reducing computational and energy overhead, facilitating efficient natural language processing for human-robot interaction in safety-critical and dynamic environments. Experimental results demonstrate that the FPGA-based LLaMA 2-7B implementation achieves up to 6.06-fold and 1.95-fold higher throughput compared to baseline CPU and GPU implementations while maintaining comparable inference accuracy. Furthermore, the proposed FPGA design surpasses existing state-of-the-art FPGA implementations, delivering a 30% improvement in computational efficiency.

Physical human-robot interaction (pHRI) plays an important role in robotic. In order for a human operator to be able to easily adapt to interact with a robot, a minimal interaction force in pHRI should be achieved. In this paper, a pHRI framework is proposed to allow the robot to regulate its trajectory adaptively for minimizing the interaction force with small position-tracking errors. The trajectory of the robot is first adjusted by the interaction force which is updated by the performance evaluation index. Then, the human hand motion is predicted based on the autoregressive (AR) model to further adapt the trajectory. Thirdly, an adaptive impedance control method is developed to update the stiffness in the robot impedance controller using surface electromyography (sEMG) signals for robot compliant interaction with the environment. This method allows the human operator to interact with the robot by the interaction force, the hand motion and muscle contraction. By investigating the performance of the proposed method, the interaction force is decreased and a good position tracking accuracy is achieved. Comparative experiments demonstrate the enhanced performance of the proposed method. Note to Practitioners—This paper focuses on developing a novel method that can allow the robot to compliantly interact with the human operator while simultaneously taking into account the trajectory-tracking accuracy and the interaction force in pHRI scenarios. The proposed method has a large application potential in a variety of pHRI tasks, such as human-robot collaborative transporting, curing, assembly, cutting, and so on. In addition, the proposed method can allow the human operator to physically interact with the robot in an easier and more intuitive manner, by taking advantage of human motion prediction and adaptive impedance control. Therefore, it is also potentially utilized for rehabilitation and assistive robots, and robot learning skills from human physical demonstration.

Tele-echography has emerged as a promising and effective solution, leveraging the expertise of sonographers and the autonomy of robots to perform ultrasound scanning for patients residing in remote areas, without the need for in-person visits by the sonographer. Designing effective and natural human-robot interfaces for tele-echography remains challenging, with patient safety being a critical concern. In this article, we develop a teleoperation system for robot-assisted sonography with two different interfaces, a haptic device-based interface and a low-cost 3D Mouse-based interface, which can achieve continuous and intuitive telemanipulation by a leader device with a small workspace. To achieve compliant interaction with patients, we design impedance controllers in Cartesian space to track the desired position and orientation for these two teleoperation interfaces. We also propose comprehensive evaluation metrics of robot-assisted sonography, including subjective and objective evaluation, to evaluate tele-echography interfaces and control performance. We evaluate the ergonomic performance based on the estimated muscle fatigue and the acquired ultrasound image quality. We conduct user studies based on the NASA Task Load Index to evaluate the performance of these two human-robot interfaces. The tracking performance and the quantitative comparison of these two teleoperation interfaces are conducted by the Franka Emika Panda robot. The results and findings provide guidance on human-robot collaboration design and implementation for robot-assisted sonography.Note to Practitioners—Robot-assisted sonography has demonstrated efficacy in medical diagnosis during clinical trials. However, deploying fully autonomous robots for ultrasound scanning remains challenging due to various constraints in practice, such as patient safety, dynamic tasks, and environmental uncertainties. Semi-autonomous or teleoperation-based robot sonography represents a promising approach for practical deployment. Previous work has produced various expensive teleoperation interfaces but lacks user studies to guide teleoperation interface selection. In this article, we present two typical teleoperation interfaces and implement a continuous and intuitive teleoperation control system. We also propose a comprehensive evaluation metric for assessing their performance. Our findings show that the haptic device outperforms the 3D Mouse, based on operators’ feedback and acquired image quality. However, the haptic device requires more learning time and effort in the training stage. Furthermore, the developed teleoperation system offers a solution for shared control and human-robot skill transfer. Our results provide valuable guidance for designing and implementing human-robot interfaces for robot-assisted sonography in practice.

Human-robot interaction (HRI) is a crucial component in the field of robotics, and enabling faster response, higher accuracy, as well as smaller human effort, is essential to improve the efficiency, robustness, and applicability of HRI-driven tasks. In this article, we develop a novel neuroadaptive admittance control with human motion intention (HMI) estimation and output error constraint for natural and stable interaction. First, the interaction force information of the robot is utilized to predict the HMI and the stiffness in the admittance model is dynamically updated based on surface electromyography (sEMG) signals of the human upper limb to achieve human-like compliance. Then, based on the designed error transformation mechanism, an innovative prescribed performance control (PPC) is proposed that allows the trajectory error to converge to the given constraint range within a predefined time for any bounded initial conditions, thus enabling the robot to maintain a comprehensive performance of moving in the desired direction as guided by the human. Also, an adaptive neural network (NN) is employed to compensate for the uncertainty of robotics systems to improve the tracking accuracy further. According to the Lyapunov stability analysis criterion, our approach ensures that all states of the closed-loop system remain globally uniformly ultimately bounded. Finally, a series of real-world robot experiments demonstrate the effectiveness of the proposed framework.

Supernumerary robotic limb (SRL) is recognized as being at the forefront of robotics innovation, aimed at augmenting human capabilities in complex working environments. Despite their potential to significantly enhance operational efficiency, the integration of SRL for dynamic and intricate tasks presents challenges in teleoperation, precise positioning, and dynamic balance control. To address challenges in initiating control when targets or the SRL’s end-effector are outside the camera’s visual range, a coarse teleoperation strategy is implemented. This strategy utilizes the inertial measurement unit (IMU) and the extended Kalman filter (EKF), enabling basic orientation and movement toward the target area without reliance on visual cues. Challenges in achieving fine-tuned control for accurate task completion, particularly in visual navigation and precise positioning of the SRL’s end-effector, are addressed by integrating object detection via YOLOX with the tangential artificial potential field (T-APF) method for exact path planning. This integration significantly enhances the system’s ability to fine-tune the placement of end-effector. The challenge of conducting balance tasks without force sensors is tackled by adopting a dual-spring model combined with autoregressive (AR) predictive modeling, enabling effective balance support through anticipatory motion adjustments. Experiments have demonstrated the system’s enhanced positional accuracy and maintained synchronization with human movements, underscoring the effectiveness of the integrated approach in facilitating complex human-robot collaborative tasks.

In this paper, we present the TacShade: a newly designed 3D-printed soft optical tactile sensor. The sensor is developed for shape reconstruction under the inspiration of sketch drawing that uses the density of sketch lines to draw light and shadow, resulting in the creation of a 3D-view effect. TacShade, building upon the strengths of the TacTip, a single-camera tactile sensor of large in-depth deformation and being sensitive to edge and surface following, improves the structure in that the markers are distributed within the gap of papillae pins. Variations in light, dark and grey effects can be generated inside the sensor under the external contact interactions. The contours of the contacting objects are outlined by white markers, while the contact depth characteristics can be indirectly obtained from the distribution of black pins and white markers, creating a 2.5D visualization. Based on the imaging effect, we improve the Shape from Shading (SFS) algorithm to process tactile images, enabling a coarse but fast reconstruction for the contact objects. Two experiments are performed. The first verifies TacShade’s ability to reconstruct the shape of the contact objects through one image for object distinction. The second experiment shows the shape reconstruction capability of TacShade for a large panel with ridged patterns based on the location of robots and image splicing technology.

Tactile sensing is significant for robotics since it can obtain physical contact information during manipulation. To capture multimodal contact information within a compact framework, we designed a novel sensor called ViTacTip, which seamlessly integrates both tactile and visual perception capabilities into a single, integrated sensor unit. ViTacTip features a transparent skin to capture fine features of objects during contact, which can be known as the see-through-skin mechanism. In the meantime, the biomimetic tips embedded in ViTacTip can amplify touch motions during tactile perception. For comparative analysis, we also fabricated a ViTac sensor devoid of biomimetic tips, as well as a TacTip sensor with opaque skin. Furthermore, we develop a Generative Adversarial Network (GAN)-based approach for modality switching between different perception modes, effectively alternating the emphasis between vision and tactile perception modes. We conducted a performance evaluation of the proposed sensor across three distinct tasks: i) grating identification, ii) pose regression, iii) contact localization and force estimation. In the grating identification task, ViTacTip demonstrated an accuracy of 99.72%, surpassing TacTip, which achieved 94.60%. It also exhibited superior performance in both pose and force estimation tasks with the minimum error of 0.08 mm and 0.03N, respectively, in contrast to ViTac’s 0.12 mm and 0.15N. Results indicate that ViTacTip outperforms single-modality sensors.

Neuroscience research has demonstrated the sig-nificance of modulating stiffness during human task performance. Similarly, endowing robots with such capability is expected. However, existing methods for robot teleoperation require operators to simultaneously control position and stiffness, resulting in high workload and task inefficiency. On the other hand, learning from demonstration (LfD) offers a feasible approach for autonomously generating stiffness. Therefore, this paper proposes a robot teleoperation system that combines the advantages of teleoperation and LfD. Teleoperation enables precise positioning guided by human operators, while LfD can transfer human stiffness skills to robots. A teleoperation-oriented stiffness-adaptive Gaussian Mixture Model/Gaussian Mixture Regression method is proposed to learn human multi-dimensional stiffness and reproduce robot stiffness on a Riemannian manifold. To enhance generalization and cooperate with teleoperation, reference points and position-driven output are introduced. Furthermore, a teleoperation strategy for both the single-leader-single-follower configuration and the single-leader-dual-follower configuration are designed, which allows operators to control either one or two robot arms with a single leader device. Finally, the effectiveness of our method is verified through a plugging-in task and a continuous flipping task, demonstrating that the proposed system is capable of performing tasks that demand high positioning accuracy and stiffness adjustment. A supplementary video for this paper is available in GitHub**https://github.com/setowenGit/TOSA-GMM-GMR-Video.

In this paper, a distributed observer-based prescribed performance control method is proposed for using a multi-robot teleoperation system to manipulate a common deformable object. To achieve a stable position-tracking effect and realize the desired cooperative operational performance, we first define a new hybrid error matrix for both the relative distances and absolute positions of robots and then decompose the matrix into two new error terms for cooperative and independent robot control. Then, we improve the Kelvin-Voigt (K-V) contact model based on the new error terms. Because the center position and deformation of the object cannot be measured, the object dynamics are then expressed by the relative distances of robots and an equivalent impedance term. Each robot incorporates an observer to estimate contact force and object dynamics based on its own measurements. To address the position errors caused by biases in force estimation and realize the position-tracking effect of each robot, we improve the barrier Lyapunov functions (BLFs) by incorporating the errors into system control. which allows us to achieve a predefined position-tracking effect. We conduct an experiment to verify the proposed controller’s ability in a dual-telerobot cooperative manipulation task, even when the object is subjected to unknown disturbances.Note to Practitioners—This article is inspired by the limitations of multi-telerobot manipulation with a deformable object, where the deformation of the object cannot be measured directly. Meanwhile, force sensors, especially 6-axis force sensors, are very expensive. To realize the purpose that objects manipulated by multiple robots match the same state as operated on the leader side, we propose an object-centric teleoperation framework based on the estimates of contact forces and object dynamics and the improved barrier Lyapunov functions (BLFs). This framework contributes to two aspects in practice: 1) propose a control diagram for deformable object co-teleoperation of multi-robots for unmeasurable object’s centre position and deformation; 2) propose an improved BLFs controller based on the estimation of contact force and robot dynamics. The estimation errors are considered and transferred using an equivalent impedance to be integrated into the Lyapunov function to minimize both force and motion-tracking errors. The experimental results verify the effectiveness of the proposed method. The developed framework can be used in industrial applications with a similar scenario.

Tactile sensors can be used for motion detection and object perception in robot manipulation. The contact detection within the camera's visual inspection area has been well-developed, but perception outside the field of view of the camera is overlooked. In this paper, we present a new tendon-linked tactile sensor, MechTac, to achieve perceptions inside and outside the field of view. The MechTac is an evolution of the typical TacTip sensor with the following two advantages. 1) The ability to provide perception outside the field of view. This is achieved by using a network of braided tendons to transfer deformation from the blind perception regions (TacSide) to the visual areas (TacTip). 2) The tactility of the TacSide and TacTip is reflected by the movements of multiple papillae pins and visible markers on the pin tips on the inner surface of the TacTip. The pins and markers are differentially sensitive to various touch features, which is similar to the differentiated perceptual ability of humans. TacTip is more sensitive to small touches, corresponding to the fingertip, while the TacSide is less sensitive but has a larger perceptual area, corresponding to the middle part of the finger. Moreover, we propose a new deep learning method to decompose the mixed information affected by the TacSide and the TacTip. A modified DenseNet121 was specifically designed for object perception at the TacTip and localization at the TacSide. The experimental results show that prediction accuracy reaches about 98% for object perception or localization and over 99% for the case requiring two functions.

Abstract In this paper, composite dynamic movement primitives (DMPs) based on radial basis function neural networks (RBFNNs) are investigated for robots’ skill learning from human demonstrations. The composite DMPs could encode the position and orientation manipulation skills simultaneously for human-to-robot skills transfer. As the robot manipulator is expected to perform tasks in unstructured and uncertain environments, it requires the manipulator to own the adaptive ability to adjust its behaviours to new situations and environments. Since the DMPs can adapt to uncertainties and perturbation, and spatial and temporal scaling, it has been successfully employed for various tasks, such as trajectory planning and obstacle avoidance. However, the existing skill model mainly focuses on position or orientation modelling separately; it is a common constraint in terms of position and orientation simultaneously in practice. Besides, the generalisation of the skill learning model based on DMPs is still hard to deal with dynamic tasks, e.g., reaching a moving target and obstacle avoidance. In this paper, we proposed a composite DMPs-based framework representing position and orientation simultaneously for robot skill acquisition and the neural networks technique is used to train the skill model. The effectiveness of the proposed approach is validated by simulation and experiments.

Learning a stable autonomous dynamic system (ADS) encoding human motion rules has been shown as an effective way for demonstration learning. However, the stability guarantee may sacrifice the demonstration learning accuracy. This article solves the issue by learning a stability certificate, represented by a neural energy function, on the demonstration set. We propose a polarlike space analysis approach to derive parameter constraints to guarantee the unique-minimum property of the neural energy function, which is essential for it to be a cogent stability certificate. Then, the neural energy function is learned to capture the demonstration preferences via constrained optimization algorithms. With the learned neural energy function, a globally asymptotically stable ADS with predefined position constraint is further formulated. We also quantitatively analyze the generalization ability of the learned ADS by utilizing the substantial flexibility of the neural energy function. The effectiveness of the proposed approach is validated on the LASA dataset and two representative robotic experiments.

Previous works have developed impedance control to increase safety and improve performance in contact tasks, where the robot is in physical interaction with either an environment or a human user. This article investigates impedance learning for a robot guided by a human user while interacting with an unknown environment. We develop automatic adaptation of robot impedance parameters to reduce the effort required to guide the robot through the environment, while guaranteeing interaction stability. For nonrepetitive tasks, this novel adaptive controller can attenuate disturbances by learning appropriate robot impedance. Implemented as an iterative learning controller, it can compensate for position dependent disturbances in repeated movements. Experiments demonstrate that the robot controller can, in both repetitive and nonrepetitive tasks: first, identify and compensate for the interaction, second, ensure both contact stability (with reduced tracking error) and maneuverability (with less driving effort of the human user) in contact with real environments, and third, is superior to previous velocity-based impedance adaptation control methods.

Learning from demonstration (LfD) is a promising method for robots to learn and generalize human-like skills. It has the advantages of high programming efficiency, easy optimization, and non-professionals can also operate. There is a lot of research work that learn motion trajectories and stiffness curves from human demonstrations simutaneously to make the robot compliant, but previous work rarely consider the changes of environment. In this article, we propose an adaptive stiffness method that enables the robot to learn motion and stiffness trajectories from a single demonstration. When the environment changes, it can spontaneously tune the stiffness according to environmental feedback to ensure the smoothness of the task. Thus the robot has the ability to adapt to environmental changes. We first proved the theoretical feasibility of the method, and then we conducted physical experiments on the Baxter robot to verify the effectiveness of the proposed method.