Unmanned surface vehicle (USV) formation navigation in constrained harbor environments presents challenges for maritime Internet of Things (IoT) applications, including surveillance, inspection, and rapid response missions. This paper addresses the trajectory planning problem for high-speed penetration, specifically focusing on minimizing oscillations in the reference trajectories. We propose S-Flocking, an integrated framework that reduces abrupt transitions in behavioral vector synthesis through the sigmoid-based transformation. For global path planning, we propose the Bisection-based Rewiring RRT* (B-RRT*) algorithm, which combines bidirectional search with dual backtracking optimization and bisection-based node insertion strategy to enhance computational efficiency and path quality. For reactive local planning, we introduce a dual-layer avoidance strategy (DAS) that integrates COLREGs-compliant maneuvers with 1-vs-1 emergency mechanism, resolving conflicts between formation-keeping and avoidance in dynamic encounters. Comprehensive simulations demonstrate that S-Flocking generates smoother reference trajectories with effective formation cohesion, thereby providing a robust solution for autonomous cooperative navigation in partially unknown dynamic maritime environments.

Determining the reasonable grasping order and reducing the interference to surrounding objects are critical for robotic grasping under mutually-stacked scenes. However, existing manipulation relationship reasoning methods generally predict object relationships from detection perspective, and require dense evaluation of all object pairs, leading to restricted reasoning accuracy caused by detection bias and relationship class imbalance. To solve this problem, we propose an EFficient Visual Manipulation Relationship Reasoning Network, called EF-VMR2N, with relationship attention and sparse graph. Specifically, relationship attention constructs two new attention mechanisms to holistically leverage the correlation characteristics between visual features and semantic information among object pairs by highlighting reasoning-specific features. Sparse graph infers the mutual relationships within the scene using fixed number of strongly-correlated triplet sets to improve the efficiency and accuracy of object pair evaluation. Extensive experiments on the VMRD and REGRAD datasets both show that the proposed EF-VMR2N achieves SOTA performance in terms of four evaluation metrics (mAP, OR, OP and IA), and compared with object detection, the improvements on manipulation reasoning are more remarkable. The stacked objects grasping under real-world scenarios further proves the effectiveness of the proposed method. Video: https://github.com/LiMing336/VideoDemonstration.

This paper presents a cooperative control framework for dual-arm robots that integrates vision-language models (VLMs) with online reinforcement learning (RL) to enhance autonomy and adaptability in complex manipulation tasks. The proposed framework adopts a hierarchical architecture: at the top level, the VLM interprets natural language instructions and visual image to generate task plans; at the middle level, an online RL module refines manipulation policies and ensures adaptive decision-making under environmental uncertainty; and at the bottom level, compliant control based on trajectory planning and impedance regulation enables safe and robust execution. In the feedback, YOLOv5 is used to detect the object, GraspNet is used to obtain the optimal grasp pose, and CLIP (Contrastive Language-Image Pre-Training) is used to judge whether task is completed. Simulations and real-world experiments validate the effectiveness of the proposed method. The dual-arm robot successfully performed various cooperative tasks such as grasping, bottle-cap unscrewing, water pouring, and box carrying, achieving an increase in the task success rate from 43% to 100% with online adaptive learning and training. These results demonstrate that the proposed framework effectively bridges high-level reasoning with low-level control, providing a scalable solution for future applications in service robotics, industrial automation, and human-robot collaboration.

Dual-arm robotic manipulation of liquid or biochemical reagents poses critical challenges due to high-dimensional configuration spaces, stringent task-specific end-effector orientation requirements to prevent spillage, frequent inter-arm collisions, and cluttered experimental environments. This paper introduces CLEAR-MP (Clearance Learning-Based Efficient Motion Planning for Dual-Arm Robots under End-Effector Orientation Constraints), a modular framework that integrates multiple innovations: a decoupled learning-driven collision estimation module–comprising aPairwise Link Clearance Networkfor self-collision and aClearance Inference Networkfor environmental obstacles, aLearning-Driven Bidirectional Parallel Search Strategyfor accelerated tree expansion, parallel Cartesian batch sampling for efficient candidate generation, fast inverse-kinematics mapping, andLearning-Guided Batch Shortcut Optimizationto refine trajectories. Together, these components generate smooth, safety-certified paths with substantially reduced planning time and path length. Extensive simulations and real-robot experiments show that CLEAR-MP achieves an average path length of 2.391 m, average planning time of 3.529 s, outperforming state-of-the-art baselines by over 50% in computation and 40% in trajectory quality while maintaining strong generalization without retraining.

Most existing robotic fruit harvesting systems rely on mechanically complex end-effectors that lack delicate tactile dexterity and fail to replicate the sophisticated sensory-motor coordination of human pickers. To enable safe, reliable, and efficient automated harvesting of delicate fruits, this paper presents a novel bio-inspired humanoid TacTip gripper for precision strawberry harvesting. Inspired by human thumbfinger opposition, we design an asymmetric TacTip gripper that integrates a Thumb tactile sensor with a built-in fingernail for stem cutting and a supporting Pillow tactile sensor. We further develop a hybrid SSIM-CNN perception framework that fuses real-time structural similarity index measure (SSIM) from both fingertips with convolutional neural network (CNN) features, enabling precise closed-loop grasp-state detection and gentle force adjustment. In addition, a segmented dynamic system (DS) motion planner decomposes the harvesting task into approach, cut, and place phases, generating reactive, smooth, and biologically plausible trajectories. Experimental results on both laboratory setups and live potted strawberry plants demonstrate reliable full-cycle harvesting with high success rates and minimal fruit damage. The proposed system provides a practical and effective solution for automated delicate fruit harvesting.

How to guarantee the effective planning trajectory of autonomous driving for mobile vehicles is the main challenge. This study provides a multi-constraint trajectory planning technique to construct safe, smooth, and dynamically viable trajectories in complicated situations. Firstly, a two-stage sampling path generation algorithm is proposed to obtain a cluster of candidate paths, considering road geometry, vehicle kinematics, and static obstacle avoidance constraints. Secondly, a cost function is designed to select the optimal path based on smoothness, consistency with the reference path, and distance to obstacles. Finally, a speed planning model is developed using convex optimization to allocate speed profiles for each path point with dynamic constraints, including time efficiency, boundary conditions, and dynamic obstacle avoidance. Experimental results on mobile vehicles demonstrate the effectiveness and stability of the proposed trajectory planning algorithm in real-world environments and its ability to handle various typical driving scenarios. The success rate of trajectory planning in the experiments was 94%, with an average planning time of 37.3ms.

Dual-arm humanoid robots are poised to transform industrial manufacturing automation in human-centric environments. However, unlocking this potential requires a unified framework that can simultaneously handle coupled bimanual coordination, versatile physical interaction, and safety. We introduce Passive Model Predictive Cooperative Interaction Control (P-MPCIC), a framework that co-optimizes task performance and interaction safety under a formal passivity guarantee. P-MPCIC integrates model predictive control for the bimanual subsystem within a whole-body architecture and uses a coupling matrix to enforce synchronization objectives across relative motion and force distribution. For interaction prediction, the framework incorporates a composite robot-environment model that combines parallel and series impedance dynamics, yielding a linear state-space predictor. Passivity is enforced as a constraint on the energy balance at the interaction port, preventing destabilizing energy generation from the controller. We verify the framework’s core principles through planar simulations and demonstrate its practical effectiveness on a 7-DoF dual-arm humanoid.

In this paper, an adaptive force-motion control framework with orientation compliance is present for redundant manipulators in physical interaction with unknown surfaces. The proposed framework includes control task space definition and double-closed-loop control based on external force loop approach. Firstly, a specification matrix is designed merely through force feedback to ensure the control task space defined in orthogonal spaces. Then, an orientation compliance controller and a force-motion close-loop controller are constructed in the outer-loop control of external force feedback loop approach. Secondly, the output of outer-loop control task, along with boundary constraints and optimization indexes is formulated as a nolinear dynamic programming problem. Next a finite-time convergence neural network based inner-loop controller is proposed for this category of dynamic programming problem and its stability and convergence analysis are given. Simulations verify the convergence and effectiveness of the proposed framework. The real-world experiments show that the Mean Integral of the Absolute Error of the proposed control framework is reduced by 77.26% compared with constant impedance control.

In this paper, a fixed-time adaptive deferred constrained control strategy is proposed for a flexible single-link manipulator system with input saturation. Fuzzy Neural Networks are utilized to estimate the unknown dynamics of the flexible manipulator system as well as the errors caused by input saturation. To address output constraints imposed within a prescribed time period, a time-shift function and an adjusted barrier function are introduced. The system’s stability is rigorously proven using the direct Lyapunov method. Finally, numerical simulations and experimental results are presented to validate the effectiveness and superiority of the proposed control approach.

In the field of semantic segmentation, the high computational cost of deep models poses a major barrier to deployment on edge devices. Among various efficiency-oriented methods, knowledge distillation has emerged as a promising technique for transferring knowledge from large models to lightweight networks. However, current knowledge distillation methods for efficient semantic segmentation still face two key challenges: (1) they often rely on large offline pre-trained teacher networks that remain fixed during training, and (2) they lack joint optimization mechanisms that enable effective teacher-student interaction in pixel-wise dense prediction. As a result, mutual learning strategies originally designed for image-level classification often fail to capture the fine-grained consistency required for semantic segmentation. To address these two challenges, we propose a novel training framework termed Teacher-Student Interactive Cycle (TSIC), which performs efficient semantic segmentation. Specifically, TSIC integrates a lightweight student network into a prompt-based foundation model as a prompted segmentor to assist an online-trained teacher. The student provides coarse mask prompts to guide the teacher, while the teacher offers fine-grained supervision through posterior probabilities and intermediate feature maps. This loop enables joint online optimization without relying on offline pre-trained teachers and fosters effective bidirectional communication. Extensive experiments conducted on several benchmark datasets, including Cityscapes, Pascal VOC, CamVid, and ADE20k, demonstrate the effectiveness of TSIC. Compared to previous methods, TSIC achieves superior segmentation mIoU in most scenarios. Our code will be made publicly available at https://github.com/CV-ShuchangLyu/TSIC.

In this study, we develop an event-triggered predefined-time sensorless prescribed and personalized compliant performance control scheme for teleoperation systems. In the absence of force/torque sensors, a predefined-time torque behavior estimator (PTTBE) is designed, and its estimated values are applied to both the admittance structure and the control law. Then, a variable stiffness parameter related to the operator's surface electromyography (sEMG) signal is incorporated into the admittance structure. By integrating the PTTBE, predefined-time sliding manifold, predefined-time performance function, and event-triggered mechanism involving time-scaling, error-scaling, and muscle activation-scaling functions, the PTTBE-based event-triggered predefined-time control (PTTBE-ETPTC) scheme is proposed. This scheme ensures that not only does the tracking error converge to a residual set within a predefined time regardless of the system's initial state, but also that the error constraints are not violated at any time. Compared with existing tracking control methods, the introduction of a variable stiffness parameter admittance structure, along with an event-triggered mechanism related to predefined-time parameters and a variable capable of reflecting the operator's intention, greatly enhances the system's flexibility, enabling a favorable balance between tracking performance for free motion and compliant performance for interaction/contact situations while reducing the control frequency. Simulations and experiments are carried out to demonstrate the effectiveness and practicality of the developed PTTBE-ETPTC scheme.

The 6-degree-of-freedom (DoF) grasp method based on point clouds has shown significant potential in enabling robots to grasp target objects. However, most existing methods are based on the point clouds (2.5-D points) generated from single-view depth images. These point clouds only have one surface side of the object, providing incomplete geometry information, which misleads the grasping algorithm to judge the shape of the target object, resulting in low grasping accuracy. Humans can accurately grasp objects from a single view by leveraging their geometry experience to estimate object shapes. Inspired by humans, we propose a novel 6-DoF grasping framework that converts the point completion results as object shape features to train the 6-DoF grasp network. Here, point completion can generate approximately complete points from the 2.5-D points similar to the human geometry experience, and converting them into shape features is the way to utilize it to improve grasp efficiency. Furthermore, due to the gap between the network generation and actual execution, we integrate a score filter into our framework to select more executable grasp proposals for the real robot. This enables our method to maintain a high grasp quality in any camera viewpoint. Extensive experiments demonstrate that utilizing complete point features enables the generation of significantly more accurate grasp proposals, and the inclusion of a score filter greatly enhances the credibility of real-world robot grasping. Our method achieves a 17.8% success rate, higher than the state-of-the-art method in real-world experiments. Code and videos are available at https://github.com/ChengYaofeng/PCF-Grasp

In minimally invasive surgery, surgical instruments are typically inserted through small incisions in the patient’s body, which serve as the remote center of motion (RCM). In robot-assisted minimally invasive surgery, developing control algorithms that comply with RCM constraints is highly challenging due to the nonlinear nature of robot motion models and the stringent precision requirements necessary for patient safety. This article introduces an enhanced zeroing neural network (EZNN) model for controlling redundant manipulators while maintaining RCM constraints. The proposed model eliminates the need for pseudoinverse matrix computations and features an explicit dynamic form. It guarantees finite-time convergence and demonstrates robustness through the application of nonlinear activation functions (AFs). These properties are rigorously validated using Lyapunov theory. Simulation and experimental results indicate that the EZNN model surpasses Jacobian-based algorithms and recurrent neural networks (RNNs) in terms of efficiency and stability, all while ensuring adherence to RCM constraints.

It is challenging to employ a quadruped robot for real-time mapping and positioning in a large range of scenes. The significant vibration and instability of the quadruped robot during mobility, as well as the quantity of computation required to convey a wide variety of complex landscapes, result in unsatisfactory drawing construction accuracy and inefficient real-time performance. Therefore, we propose an accurate robust spinning LiDAR SLAM (ARS-SLAM) algorithm for a quadruped robot under the large-scale scene. The tightly coupled iterative Kalman filter in FAST-LIO2 is introduced into the front end of the cartographer framework to improve the accuracy and robustness of robot pose estimation. To reduce the computational complexity of the original cartographer framework, a pose threshold optimization algorithm was introduced to effectively remove redundant information from loop detection and improve computational efficiency and real-time performance. We tested the system's performance against the most advanced point-cloud-based methods, LIO-SAM and FAST-LIO2, on a large dataset of large science parks and underground parking lots, and the results show that the proposed system achieves the same or better accuracy and real-time performance.

Reliable 6D pose estimation is crucial for robotic tasks but presents significant challenges in environments with occlusion. Recent approaches tend to directly predict pose parameters of object with deep neural networks, lacking the modeling ability of non-adjacent and complex relationships of surface points in occluded scenarios. To solve this problem, we propose a novel occlusion-aware 6D pose estimation framework, which uses depth-guided graph neural network (GNN) to model potential relationships from RGBD input. Two semantic information, which are mask and binary code of object, are adaptively fused to extract 2D-3D correspondence related features in an effective manner. Both enhanced graph features and fused semantic information contribute to the performance improvement of pose estimation with occlusion. Extensive experiments indicate that our approach outperforms comparative methods by 1.2% and 1.9% on LMO and YCBV datasets (up to 30% for certain objects) and its validity is also verified under real-world pose estimation test.

With the growing popularity of electric vehicles, the demand for robot-based unmanned automatic charging has become both urgent and challenging. Two key challenges need to be addressed: how to efficiently locate the charging port, and how to compliantly insert the connector into the port. In this paper, we propose an incremental learning method based on the broad learning system to address the visual positioning error of the charging port. This method allows the robot to transfer and generalize the search skills learned in simulation to real-world scenarios. As a result, the robot can rapidly locate the charging port in real-world environments without the need for complex contact state modeling, time-consuming data collection, or model retraining. Subsequently, a biomimetic admittance controller is designed to enable the robot to adapt its compliant behavior online during the plugging process. Finally, experiments are performed on a UR robot to verify the effectiveness of our method.

In contact-rich tasks such as polishing and drilling, inevitable physical interactions often lead to task deviations due to interference, typically resulting in excessive contact forces and eventual task failure. To tackle these challenges, we propose an innovative human-guided visual-impedance control framework. Specifically, we first introduce an interaction model within image feature space, which models the dynamics of human-robot-environment interactions. Subsequently, human operation skills are characterized through human-guided wrenches, and acts on visual features through a projection matrix, thus integrating human-guided wrenches with visual-impedance interaction dynamics. Finally, leveraging this framework, we develop a novel variable-stiffness visual-impedance control strategy. The impedance parameters are optimized online via Quadratic Program, ensuring that the end-tool contact force converges to desired value while adhering to safety constraints. The validity of the proposed framework was established through polish experiments.

The development of large Vision-Language-Action (VLA) models has enhanced the robot’s ability to manipulate objects in unseen scenarios based on language instructions. While existing VLAs have demonstrated promise in various scenarios, they still struggle with effective multi-modal data feature extraction and lack a closed-loop inference framework. In this paper, we propose an advanced VLA model. Unlike previous works that repurpose VLM for action prediction using simple action quantization, we componentized the VLA architecture with a specialized action module conditioned on the model output and a critic module for inference. We demonstrate the performance improvement of diffusion action transformers in modeling continuous temporal actions, with the critic module applied during inference to form a closed-loop model. Extensive experiments on real robots demonstrate that our model significantly outperforms existing methods, with the ability to handle complex, high-precision tasks and generalize to unseen objects and backgrounds.

Due to non-destructive testing (NDT) techniques being both expensive and inconvenient in dynamic detection scenarios, innovative alternatives are urgently needed to address cost-efficiency and deployment challenges. We first design TacRoller, a tactile sensor roller for automated characterization of surface defects in composite materials, to address the dilemma. It collects tactile images of defects on the composite’s plies by capturing changes caused by deformation of the outer elastomer through the internal camera. It reduces the cost of inspection by 80% to 90% compared to NDT equipment like radiographic testing while ensuring detection efficiency. It takes 58.86 seconds to complete a 35 cm×18 cm × 0.5 mm dry-woven fabric. Moreover, we collect a total of 2,744 images of samples of dry-woven fabric unidirectional prepreg through TacRoller to form a dataset, including wrinkles, foreign objects and debris (FODs), broken fibre, voids and healthy textures. Subsequently, we propose a multi-order gated aggregation (MOGA)-U-Net to tackle critical challenges of noise sensitivity and multi-scale defect recognition in tactile images, enabling robust segmentation and multi-category classification tasks. The results show that the MOGA-U-Net achieves a test dice coefficient of 76.0% and classification accuracy of 98.9%, outperforming DeepLabV3 and other benchmarks. By providing a scalable and effective NDT substitute, our system realises autonomous defect identification and classification on composites surface, thus improving quality control in the production of composites.

Robotic hand tool use has garnered significant attention from robotics researchers, because it enhances dexterity beyond the limitations imposed by manipulators with fixed tool configurations and human-involved manual tool changes. Despite extensive research, current methodologies predominantly focus on imitating human hand trajectories, often neglecting the pivotal role of tool-environment interaction. This study addresses this gap by exploring the task of cucumber peeling as a case study to implement contact-based demonstration strategies in robotic tool use. Our approach concentrates on the subtle tool contact behaviors that manifest through contact dynamics. Specifically, we select appropriate tool stiffness for the peeling tasks, which is captured via a handheld teaching device equipped with optical tactile sensors. Subsequently, object-level stiffness control strategies are employed to emulate these behaviors using a three-fingered robotic hand. Experimental results from real-world cucumber peeling trials substantiate our methodology, illustrating that the robotic hand can adjust contact through finger movements, thereby achieving humanlike peeling efficiency without necessitating alterations to the tool structure. This study not only demonstrates the feasibility of sophisticated tool use by robotic hands, but also highlights the critical importance of integrating tactile feedback to refine interaction with the environment.

Since its inception, impedance control has emerged as a fundamental framework for robotic interaction control. Recent advancements in geometric impedance control have demonstrated certain advantages over traditional Cartesian impedance control. However, existing geometric impedance control approaches generally lack force regulation capabilities or rigorous stability guarantees. In this paper, we propose a safety-aware geometric force-impedance controller that addresses these limitations. By incorporating an energy tank mechanism, the proposed approach enables precise force tracking while preserving full compatibility with the impedance behavior. Furthermore, an energy injection and freezing mechanism is introduced, allowing dynamic regulation of energy exchange between the tank and the robotic system. Notably, the proposed method eliminates the need for an offline estimation of the initial energy stored in the tank, facilitating real-time adjustments of force controller parameters. To validate the effectiveness of the proposed framework, we conduct extensive polishing experiments on a real robotic platform. The results demonstrate the capability of the proposed controller to achieve stable and precise force regulation.

Impedance control with fixed parameters lacks the flexibility to adapt to dynamic and uncertain environments, which may not meet the task requirements during robot-environment interaction. In this paper, a novel impedance learning control method is proposed to enhance robotic adaptability in unknown environments. First, an adaptive gradient learning strategy is designed to optimize step size in iterative learning process, leveraging historical gradients for dynamic adjustment. Then, a data-efficient adaptive model based on Streaming Sparse Gaussian Process (SSGP) is employed to accelerate impedance learning convergence. Additionally, it also can reduce computational complexity and improve generalization by online removing redundant data points, which utilizes prior data to estimate impedance parameters. The simulation results have demonstrated that the proposed method outperforms traditional iterative learning control approaches in convergence speed and generalization, verifying the feasibility and validity of the proposed method.

Multimodal sentiment analysis and depression detection are pivotal for advancing human-computer interaction, yet significant challenges remain. First, the limited extraction of global contextual information within individual modalities risks the loss of modal-specific features. Second, existing methods often prioritize unaligned textual interactions, neglecting critical inter-modal discrepancies. To address these issues, we propose the Scale-Selectable Global and Discrepancy Learning Network (SSGDL), an innovative framework that integrates two core modules: the Cross-Shaped Dynamic Scale Attention Module (CSDSA) and the Primary-Secondary modal Discrepancy Learning Module (PS-MDL). The CS-DSA dynamically selects scales and employs cross-shaped attention to capture comprehensive global context and intricate internal correlations, effectively producing a fused modal representation. Meanwhile, the PS-MDL designates the fused modal as primary and utilizes cross-attention mechanisms to learn discrepancy representations between it and other modalities (textual, acoustic, and visual). By leveraging intermodal discrepancies, SSGDL achieves a more nuanced and holistic understanding of emotional content. Extensive experiments on three benchmark multimodal sentiment analysis datasets (MOSI, MOSEI, SIMS) and a depression detection dataset (AVEC2019) demonstrate that SSGDL consistently outperforms state-of-theart approaches, setting a new benchmark for multimodal affective computing.

Robot learning has attracted an ever-increasing attention by automating complex tasks, reducing errors, and increasing production speed and flexibility, which leads to significant advancements in manufacturing intelligence. However, its low training efficiency, limited real-time feedback, and challenges in adapting to untrained scenarios hinder its applications in smart manufacturing. Introducing a human role in the training loop, a practice known as human-in-the-loop (HITL) robot learning, can improve the performance of robots by leveraging human prior knowledge. Nonetheless, the exploration of HITL robot learning within the context of human-centric smart manufacturing remains in its infancy. This study provides a holistic literature review for understanding HITL robot learning within an industrial context from a human-centric perspective. A united structure is presented to encompass different aspects of human intelligence in HITL robot learning, highlighting perception, cognition, behavior, and notably, empathy. Then, the typical applications in manufacturing scenarios are analyzed to expand the research landscape for smart manufacturing. Finally, it introduces the empirical challenges and future directions for HITL robot learning in the next industrial revolution era.

With medical ultrasound becoming one of the most prevalent examination methods, robotic ultrasound systems offer the potential to simplify the scanning process and relieve professional sonographers from repetitive and tedious tasks. Despite recent advances, enabling robots to autonomously perform ultrasound examinations remains a challenge, mainly due to the difficulty in representing and generalizing professional ultrasound skills. In this paper, we present a comprehensive framework for learning autonomous ultrasound skills from freehand demonstrations in clinical settings. Our proposed framework consists of two key stages: offline learning and online adaptation. During the offline learning stage, ultrasound skills are encapsulated into a low-dimensional probabilistic model using a self-supervised architecture. The multimodal signals include ultrasound images, probe orientations, and contact forces. During the online adaptation stage, the model predicts the optimal actions either by direct regression or by using local exploration schemes. We perform clinical demonstrations with 24 volunteers and collect 120 experiences. Our benchmark includes 5 different tasks, including intra-patient, inter-patient, inter-sex, inter-age, and inter-obesity tasks. Both one-step and sequence-based predictions are achieved by using different variants of our framework. Customized and generic representation learning backbones are tested and analyzed. In conclusion, our autonomous ultrasound framework is flexible and robust, and potentially enriches the options for freehand/robotic ultrasound applications. Note to Practitioners—This paper is motivated by the problem of learning multimodal manipulation skills from human demonstrations, with a specific focus on freehand ultrasound skills. Our multimodal fusion framework is effective and compatible with some popular image representation backbones. Our adaptive methods and the variants have satisfactory prediction accuracy, with flexibility achieved by adjusting a few factors. We collect high-quality freehand demonstrations from ultrasound examinations in clinical settings. The data is openly available to facilitate the reproducibility of our work and to support the development of autonomous ultrasound strategies based on imitation learning.

Ultrasound-based medical examination usually requires establishing proper contact between an ultrasound probe and a human body that ensures the quality of ultrasound images. The scanning skills are quite challenging for a robot to learn primarily due to the complex coupling between the applied force profile and the resulting ultrasound image quality. While reinforcement learning appears as a powerful tool for learning complex robot skills, the deployment of these algorithms in medical robots demands special attention due to the evident safety concerns that arise from physical probe-tissue interactions. In this paper, we explicitly consider external constraints on the force magnitude when searching for the optimal policy parameters to enhance safety during ultrasound-guided robotic interventions. In particular, we study policy optimization under the framework of a constrained Markov decision process. The resulting gradient-based policy update is then subject to the involved constraints, which can be readily addressed by the primal-dual interior-point technique. In addition, upon the observation that policy update requires consecutive policies to be close to each other to have stable and robust performance with reinforcement learning algorithms, we design the learning rate of policy gradient from an imitation perspective. The performance of the proposed constraint-aware policy optimization method is validated with experiments of robotic ultrasound imaging for spinal diagnosisNote to Practitioners—This paper was motivated by the problem of safely learning the optimal interaction force strategy to facilitate robotic ultrasound imaging. Existing approaches to robotic ultrasound imaging usually empirically set a constant value for the scanning force, despite the fact the force strategy plays an important role in the quality of the ultrasound images. This paper suggests the usage of reinforcement learning to identify the optimal interaction force due to the complex acoustic coupling between the force and the ultrasound image quality. Specifically, we propose constraint-aware reinforcement learning in view of the safety-critical issues as a result of physical human-probe interaction. We then conduct a theoretical analysis of the proposed safe reinforcement learning, including monotonic improvement and policy value bound under mild assumptions. Preliminary real experiments on ultrasound imaging of the spine of a phantom for scoliosis assessment suggest that the proposed approach can safely learn the optimal scanning force without violating the prescribed force threshold. In the future, we would like to apply our approach to learning the optimal scanning force on different organs of interest of human subjects.

This paper proposes a specific and effective real-time sequence planning method using robot manipulators to complete complex assembly tasks. Many previous studies developed different traversal methods to obtain the optimal assembly sequence. Besides, a number of algorithms were proposed to enhance flexibility when the conditions or rules were changed in various sequence optimization problems. However, these state-of-the-art (STOA) methods necessarily require modifications when task details are changed. Consequently, to further improve the generalization ability and improve the performance of the sequence optimization, a neural combinatorial optimization algorithm combined with a self-learning strategy is proposed for assembly sequence planning. In addition, obstacle avoidance and the non-collision constraints between workpieces in the assembly process are considered. According to the experiment results, the new method is superior to the STOA methods in terms of optimization efficiency. More importantly, the proposed method has satisfactory generalization ability for different assembly tasks.Note to Practitioners—This paper studies assembly sequence planning problems for different real-world applications in industrial and home service fields. Many assembly sequence planning solutions have been widely utilized before. However, the generalization ability of the previous methods is not satisfactory since the re-adjust process is required when the workpiece number or collision condition changes in different tasks.Motivated by the above reasons, this paper develops a learning-based assembly sequence planning solution to resolve complex assembly problems without parameter re-adjustment processes. Users can directly apply the developed workpiece identification and localization method to obtain the sensing information. Then, the newly designed collision-free cost function should be programmed as the core of the assembly sequence optimization. Next, the proposed neural combinatorial optimization (NCO) with the sensing information and target configuration as inputs can provide the optimal assembly sequence by self-learning. The learned NCO-based method can be directly applied to diverse planning tasks, even with different workpiece numbers. Users can also refer to the experimental examples in this paper for the extension of the proposed method to their own applications.

Stable dynamic systems enable robotic systems to plan and execute complex geometric motions in unstructured environments. However, in certain scenarios, such as precision assembly tasks, the constrained operational workspace of robotic arms, along with the presence of obstacles, may lead to unintended collisions. Furthermore, motion precision plays a crucial role in determining the success rate of such tasks. To address these challenges, we propose a novel Safe-Critical Dynamic System (SC-DS) framework. The SC-DS framework consists of a stable dynamic system and an obstacle avoidance controller. The stable dynamic system is formulated in a parametric nonlinear form, which enhances performance in terms of accuracy. Additionally, control barrier functions (CBF), corresponding to complex constraint spaces, are learned from demonstration data. By utilizing these learned CBFs, an obstacle avoidance controller is designed to ensure that the system trajectory remains within the learned safety boundaries. Moreover, the controller adaptively extends the effective range of control inputs, thus mitigating replication errors due to input limitations. Experimental results, both in simulation and with a physical robot, demonstrate that the SC-DS framework effectively reproduces trajectories with both stability and safety, outperforming existing methods in terms of overall task performance.Note to Practitioners—This study is motivated by the need to develop a safer and more precise skill-learning framework for practical applications, such as service robots and assembly robots. We propose the SC-DS framework, which integrates the challenges of uncertain environments into the learning of precise motion skills, ensuring both accuracy and safety in trajectory generation. This framework is particularly suitable for applications requiring strict performance and safety standards. By incorporating safety constraints directly into the learning process, our method provides a robust solution that effectively addresses robot-environment interactions, including obstacles, disturbances, and varying conditions. Our research enhances the reliability of dynamic system-based learning frameworks and offers practical tools for real-world applications, ensuring robots perform tasks efficiently while maintaining safety. Practitioners can leverage this framework to improve safety and maintain high-precision skill learning, ensuring effective handling of real-world challenges.

Deep Venous Thrombosis (DVT) is a common vascular disease with blood clots inside deep veins, which may block blood flow or even cause a life-threatening pulmonary embolism. A typical exam for DVT using ultrasound (US) imaging is by pressing the target vein until its lumen is fully compressed. However, the compression exam is highly operator-dependent. To alleviate intra-and inter-variations, we present a robotic US system with a novel hybrid force motion control scheme ensuring position and force tracking accuracy, and soft landing of the probe onto the target surface. In addition, a path-based virtual fixture is proposed to realize easy human-robot interaction for repeat compression operation at the lesion location. To ensure the biometric measurements obtained in different examinations are comparable, the 6D scanning path is determined in a coarse-to-fine manner using both an external RGBD camera and US images. The RGBD camera is first used to extract a rough scanning path on the object. Then, the segmented vascular lumen from US images are used to optimize the scanning path to ensure the visibility of the target object. To generate a continuous scan path for developing virtual fixtures, an arc-length based path fitting model considering both position and orientation is proposed. Finally, the whole system is evaluated on a human-like arm phantom with an uneven surface. The code (https://github.com/dianyeHuang/RobDVTUS) and intuitive demonstration video (https://www.youtube.com/ watch?v=3xFyqU1rV8c) can be publicly accessed.Note to Practitioners—Robotic ultrasound (US) systems have attracted attention for various applications in the past decades. However, the existing studies are not mature and intelligent enough for some challenging applications, such as DVT exam, which requires rich contact interaction between patients and clinicians. To tackle with this challenge, this study presents a novel human-centric robotic DVT exam program using the technique of virtual fixture. The coarse-to-fine path planning module ensures the repeatability of US acquisitions carried out at different times. During DVT exam, the proposed continuous 6D path virtual fixture can guide clinicians to freely move the probe along the scan path while limiting the probe motion in other directions. In order to perform the compress-release exam, a decoupled position/force controller is developed to precisely generate the contact force conveyed by clinicians and to restrict the probe motion along the probe centerline. We believe such a robot-assisted system is a promising solution to take both advantages of robots about the accuracy and repeatability and human operators about the advanced physiological knowledge.

A promising ultrasound (US) image acquisition requires experienced sonographers holding the probe with proper force and pose to ensure an excellent acoustic coupling. To enable a robotic ultrasound system (RUSS) to acquire the sonographers’ skills from ultrasound image demonstrations, this paper proposes a cutting-edge framework that integrates an expert technique discrimination network and a robotic strategy generation network to learn expert scanning skills. In this framework, the expert technique discrimination network focuses on learning expert scanning techniques from the pre- and post-frame ultrasound images. Furthermore, to acquire expert scanning skills and obtain a standard image view, we design a knowledge-based algorithm grounded on inverse reinforcement learning (IRL) to generate a series of scanning policies concerning the expert technique discrimination network. Both simulations and experiments are conducted to validate the effectiveness of the proposed framework by comparing it with MI-GPSR and PTR. The scanning success rate and trajectory tracking error of the algorithm in the simulation environment are 68% and 12.0782, respectively, while in the phantom environment are 94% and 11.8367. The results demonstrate good performance in the task of imitating expert techniques for autonomous scanning. Note to Practitioners—The motivation for this work originates from the need for ultrasound scanning tasks, such as carotid plaque and thyroid scans, to follow specific procedures. In clinical practice, sonographers require extensive learning and training to acquire these scanning skills. Traditional autonomous robotic ultrasound research often focuses on achieving the final standard view, neglecting the logical flow of the scanning process. Moreover, current studies on imitation learning for robotic ultrasound typically require not only ultrasound images from expert demonstrations but also additional data like probe position, adding complexity and potential errors to the data collection process. The proposed method addresses this by designing a framework that learns and comprehends expert techniques solely from image demonstrations. This endows RUSS with the ability to perform a human-like ultrasound scanning task. This work can be applied in the field of autonomous ultrasound robotics to assist sonographers in achieving more precise scans.

Considering the increasingly widespread use of multilegged robots in various fields, the design of their bilateral or multilateral teleoperation system is facing new challenges, such as foot slippage, tele-driving skills, and operators experience, which can induce activity of its teleoperation and poor maneuverability. With the assistance of an experienced trainer and the development of a multi-user cooperative teleoperation system, it is more conducive to tele-driving the multilegged robot in real field environment. Accordingly, in this study, a novel trilateral shared control architecture based on the velocity and force coordination of a dual-user haptic tele-training system for a hexapod robot subjected to soft terrain is proposed. Its methodology simultaneously considers the online transfer strategy of the task authority adjustment between dual users and the dynamic compensation approach for the active exogenous disturbance. In our system, the dominance factor adjusts the control authority of dual users according to the velocity and force command-tracking performance of the trilateral system. The closed-loop stability of the system is established by its passivity, and the force applied by the environment termination is determined to be approximately equal to the sum of the forces felt by two operators. The experiments of the proposed dual-user haptic training system for the hexapod robot demonstrate that it results in a stable trilateral teleoperation with a satisfactory tracking performance. Note to Practitioners—Multi-legged mobile robots have been widely used in daily and scientific scenarios, including industrial manufacturing, field exploration, and domestic service. However, as tasks and requirements become larger and more complex, the difficulty of development increases. To overcome these issues, this paper proposes a trilateral tele-training architecture for teleoperating the movement of a hexapod robot on deformable terrain, with complications such as foot slippage. The shared control method incorporates primarily two novel technical contributions, which refer to an adaptive authority adjustment method to allocate control between trainer and trainee operators, and a passivity controller (an adaptive damper) informed by observed loss in the remote robot’s velocity (both of which can be executed simultaneously). The former is implemented with a varied dominance term. Two different shared control methods are proposed, one which essentially relies on the errors in provided position inputs between the two leaders and the follower (position-error-based, PEB), and one that examines torques applied to the operators compared with environmental interaction force (direct force reflection, DFR). In addition, the practitioners who research in fields where efficient control of cooperative teleoperation for multi-legged robots subject to outdoor environment can benefit from these proposed methods. In future research, we will further address the dynamic teleoperation of hexapod robot and time delays.

This paper presents a framework for learning and transferring robot tool-use skills based on Dynamic Movement Primitives (DMPs) for robot fine manipulation. DMPs and their enhanced methods are employed to acquire a specific tool-use skill applicable to tools with similar sizes, shapes, and uses. However, the acquired skills may not be transferable to other scenarios and tools with variations. The new framework introduces two new types of skills based on DMPs: Object Operating (O2) skill and Tool Flipping (TF) skill. The O2 skill enables robots to handle tools for manipulating objects to achieve desired effects. The learning process for the O2 skill considers limitations imposed by tools and the environment during human demonstrations. Distinguishing between whether constraints can be modelled or not, we propose both a model-based and a constraint-based method to separate a constraint-irrelevant (CI) skill and the constrained conditions. The CI skill is generalized using a novel method called constrained -DMP lite, enabling adaptation to new tasks with special tools. The TF skill addresses situations where tools must generate an action to alter contacting positions on both objects and tools while avoiding conflicts during movement. Finally, the TF and O2 skills are generalized to be applied in creating a continuous action chain. We conduct several experiments to compare and analyze the advantages and disadvantages of the proposed methods with other approaches in terms of generalizability and calculation complexity.Note to Practitioners—Strengthening robot tool-use ability has been a hot research topic in recent years because these tools can extend the reachability and enhance the flexibility of robots. The previous research on DMPs has been utilized for learning tool-use skills. However, the learned skills few considered the tools’ special use regulations, therefore the skill of using a tool is hard to transfer to another tool-use case. This paper explores tool-use skill learning and transfer between different tools by developing a framework based on the DMPs for this problem. The framework consists of two kinds of skills: O2 skill and TF skill with different purposes as well as a series of newly developed algorithms, such as constrained -DMP lite, a model-based and a constraint-based CI skill learning methods. These methods can separate the constraints from human demonstrations of using tools to achieve a CI skill and generalize the CI skill according to the constraints generated from a new tool-use manipulation task. We verify the effectiveness of the proposed framework through some typical tool-use experiments, including pushing objects, cutting and obstacle avoidance in actuality. The development of this framework can be used in industrial and house working scenarios.

Supernumerary robotic limbs (SRLs) can significantly enhance human manipulation capability. However, it is difficult to achieve a safe and friendly interaction for collaboration tasks in complicated and dynamic external environments, and it easily leads to safety issues between a human user and the SRL robot. To address the aforementioned issues, this paper designs a new type of SRL and proposes a teleoperated control method for human-robot safe interaction. Specifically, inspired by the principle that human arms can autonomously adjust their stiffness to safely interact with the external environment, this paper proposes an SRL interaction control method based on the variable stiffness of the human upper limb. Specifically, the proposed variable stiffness control parameters can be adaptively updated based on the characteristics of surface electromyography (sEMG) signals from the human upper limb. The effectiveness of the proposed framework is validated through experimental results.

Physical human-robot interaction (pHRI) plays an important role in robotic. In order for a human operator to be able to easily adapt to interact with a robot, a minimal interaction force in pHRI should be achieved. In this paper, a pHRI framework is proposed to allow the robot to regulate its trajectory adaptively for minimizing the interaction force with small position-tracking errors. The trajectory of the robot is first adjusted by the interaction force which is updated by the performance evaluation index. Then, the human hand motion is predicted based on the autoregressive (AR) model to further adapt the trajectory. Thirdly, an adaptive impedance control method is developed to update the stiffness in the robot impedance controller using surface electromyography (sEMG) signals for robot compliant interaction with the environment. This method allows the human operator to interact with the robot by the interaction force, the hand motion and muscle contraction. By investigating the performance of the proposed method, the interaction force is decreased and a good position tracking accuracy is achieved. Comparative experiments demonstrate the enhanced performance of the proposed method. Note to Practitioners—This paper focuses on developing a novel method that can allow the robot to compliantly interact with the human operator while simultaneously taking into account the trajectory-tracking accuracy and the interaction force in pHRI scenarios. The proposed method has a large application potential in a variety of pHRI tasks, such as human-robot collaborative transporting, curing, assembly, cutting, and so on. In addition, the proposed method can allow the human operator to physically interact with the robot in an easier and more intuitive manner, by taking advantage of human motion prediction and adaptive impedance control. Therefore, it is also potentially utilized for rehabilitation and assistive robots, and robot learning skills from human physical demonstration.

Dynamic motion primitives (DMPs) have been widely used in robotics and automation systems because of their rapid deployment capability. Previous research has concentrated on extending coupled dynamic movement primitives (CDMPs) to manipulate rigid objects using a dual-arm robot. However, manipulating deformable objects may fail due to issues with the workspace, manipulability caused by the robot’s layout, and the uncertain dynamics of deformable objects. This research proposes a unique system that combines Learning from Demonstration (LfD) to optimise robot layouts based on human experience and robot manipulability. It also employs a modified CDMPs (MCDMPs) method to manipulate deformable objects. The MCDMPs include a new term to ensure that the deformed object can change its configuration during manipulation. Furthermore, another term is introduced to track the desired trajectory of the deformed object, which is crucial for transporting tasks. We conducted simulations using a mass-spring-damper system for cooperative manipulation to validate the proposed approach. We also employed a dual-arm robot platform to transport a deformed ball with disturbance. The simulation and experimental findings indicate that our method performs well in trajectory tracking and configuration change. Note to Practitioners—This research paper addresses the need for faster deployment of dual-arm robots in various applications, such as industrial and services. We must consider the robot’s installation layout and task programming for rapid deployment. To optimise layouts for tasks quickly, we introduce an optimisation framework which considers the comprehensive performance of dual arms and uses demonstration sampling and learning generalisation. To address the manipulation of deformable objects by dual-arm robots, we use an MCDMPs method that employs the barrier Lyapunov function (BLF) to track changed configurations. Additionally, we introduce a term for reference trajectory tracking component for the manipulated object.

Teleoperation and Learning from Demonstration (LfD) are complementary paradigms for robotic control, yet their synergistic potential remains underexplored. This work proposes a unified framework that dynamically adjusts teleoperation precision using task-specific priors from LfD while optimizing demonstration learning through teleoperation scaling. By analyzing human operator signals (e.g., muscle activity), our method autonomously scales robot movements to balance the precision requirements and execution efficiency during teleoperation. Conversely, teleoperation data informs the adaptive arrangement of motion primitives in LfD, improving trajectory accuracy in critical task phases. Experiments show that the proposed method can improve the efficiency of teleoperation tasks and the accuracy of task learning. This bidirectional synergy offers a practical pathway to enhance both human-robot interaction and skill learning, particularly in applications demanding variable precision, such as assembly and surgical robotics.

Tele-echography has emerged as a promising and effective solution, leveraging the expertise of sonographers and the autonomy of robots to perform ultrasound scanning for patients residing in remote areas, without the need for in-person visits by the sonographer. Designing effective and natural human-robot interfaces for tele-echography remains challenging, with patient safety being a critical concern. In this article, we develop a teleoperation system for robot-assisted sonography with two different interfaces, a haptic device-based interface and a low-cost 3D Mouse-based interface, which can achieve continuous and intuitive telemanipulation by a leader device with a small workspace. To achieve compliant interaction with patients, we design impedance controllers in Cartesian space to track the desired position and orientation for these two teleoperation interfaces. We also propose comprehensive evaluation metrics of robot-assisted sonography, including subjective and objective evaluation, to evaluate tele-echography interfaces and control performance. We evaluate the ergonomic performance based on the estimated muscle fatigue and the acquired ultrasound image quality. We conduct user studies based on the NASA Task Load Index to evaluate the performance of these two human-robot interfaces. The tracking performance and the quantitative comparison of these two teleoperation interfaces are conducted by the Franka Emika Panda robot. The results and findings provide guidance on human-robot collaboration design and implementation for robot-assisted sonography.Note to Practitioners—Robot-assisted sonography has demonstrated efficacy in medical diagnosis during clinical trials. However, deploying fully autonomous robots for ultrasound scanning remains challenging due to various constraints in practice, such as patient safety, dynamic tasks, and environmental uncertainties. Semi-autonomous or teleoperation-based robot sonography represents a promising approach for practical deployment. Previous work has produced various expensive teleoperation interfaces but lacks user studies to guide teleoperation interface selection. In this article, we present two typical teleoperation interfaces and implement a continuous and intuitive teleoperation control system. We also propose a comprehensive evaluation metric for assessing their performance. Our findings show that the haptic device outperforms the 3D Mouse, based on operators’ feedback and acquired image quality. However, the haptic device requires more learning time and effort in the training stage. Furthermore, the developed teleoperation system offers a solution for shared control and human-robot skill transfer. Our results provide valuable guidance for designing and implementing human-robot interfaces for robot-assisted sonography in practice.

The challenges inherent in long-horizon tasks in robotics persist due to the typical inefficient exploration and sparse rewards in traditional reinforcement learning approaches. To address these challenges, we have developed a novel algorithm, termed hlexplicit-implicit subgoal planning (EISP), designed to tackle long-horizon tasks through a divide-and-conquer approach. We utilize two primary criteria, feasibility and optimality, to ensure the quality of the generated subgoals. EISP consists of three components: a hybrid subgoal generator, a hindsight sampler, and a value selector. The hybrid subgoal generator uses an explicit model to infer subgoals and an implicit model to predict the final goal, inspired by way of human thinking that infers subgoals by using the current state and final goal as well as reason about the final goal conditioned on the current state and given subgoals. Additionally, the hindsight sampler selects valid subgoals from an offline dataset to enhance the feasibility of the generated subgoals. While the value selector utilizes the value function in reinforcement learning to filter the optimal subgoals from subgoal candidates. To validate our method, we conduct four long-horizon tasks in both simulation and the real world. The obtained quantitative and qualitative data indicate that our approach achieves promising performance compared to other baseline methods. These experimental results can be seen on the website https://sites.google.com/view/vaesi.

Imitation learning allows us to directly encode manipulation skills based on human demonstrations, facilitating rapid transfer of skills without any expert knowledge. Autonomous dynamic systems (DS) offer reliable stability and time-independence though sacrificing part of accuracy, and are increasingly attractive as an encoding method. As unstructured environments become more challenging, skill trajectories become more complex, and various disturbances are encountered, existing state-of-the-art encoding methods struggle to adapt to these complex tasks. This paper introduces a novel robust DS-based framework for learning skills in complex tasks, which consists of trajectory regularization, adaptive segmentation, skill modeling, and skill organization based on new task requirements. It achieves a task-level generalization so that the operator only needs to focus on the semantic deconstruction of a task. Additionally, we propose an online modulation policy for the skill decision engine to address two types of disturbances: enhancing convergence speed for large-scale disturbances and improving fitting capability for small-scale disturbances while still keeping stability. To evaluate the effectiveness of the proposed framework, we conduct various comparison experiments in simulation and a real-world sugar-scooping task to assess the generalization performance and the ability of resistance to disturbances.Note to Practitioners—Imitation learning for complex tasks is crucial to the development of robot intelligence. However, achieving a balance between maintaining generalization accuracy and robustness remains a challenging problem that necessitates continuous exploration in the field of imitation learning. The purpose of this paper is to propose a robust imitation learning framework from human demonstration, which includes preprocessing, learning and generalization, that can be applied in industrial production or daily life. Considering appropriate trajectory segmentation and self-organization strategies can effectively improve the generalization accuracy by prior research, it is necessary to introduce them into our framework. Most importantly, we design novel disturbance-resistant online modulation strategies from both task-level and motion-level aspects. To validate the effectiveness of our approach, we conduct simulations and coffee scooping experiments. The results show that skills acquired through demonstration can reliably, accurately, and safely perform tasks even in uncertain environments. This paper is a systematic and pioneering attempt to implement.

This paper investigates the problem of practical prescribed-time fuzzy bipartite time-varying formation control for nonlinear multiagent systems. To achieve practical prescribed-time control, a bounded and continuous prescribed-time evolution function is constructed, and a new stability lemma is derived, proving that the proposed controller avoids singularities and ensures synchronization errors converge to a neighborhood independent of initial conditions within the prescribed time. Considering cooperative-competitive interactions among the agents, a fuzzy bipartite time-varying formation controller is designed to achieve formation control within a specified time while keeping all closed-loop signals bounded. Furthermore, to reduce communication costs, an adaptive self-triggered mechanism is introduced. This mechanism maps the prescribed time to a dynamic triggering function, enabling the controller to adjust triggering conditions based on the user-defined convergence time, thereby balancing speed and communication costs. Finally, the effectiveness of the proposed method is validated through a simulation case involving five unmanned ground vehicles.

As robots grow increasingly intelligent and utilize data from various sensors, relying solely on unimodal data sources is becoming inadequate for their operational needs. Consequently, integrating multimodal data has emerged as a critical area of focus. However, the effective combination of different data modalities poses a considerable challenge, especially in complex and dynamic settings where accurate object recognition and manipulation are essential. In this paper, we introduce a novel framework integrating with Multimodal Information for Grasping Synthesis with vision-language models (MIG) designed to improve robotic grasping capabilities. This framework incorporates visual data, textual information, and human-derived prior knowledge. We start by creating target object masks based on this prior knowledge, which are then used to segregate the target objects from their surroundings in the image. Subsequently, we employ language cues to refine the visual representations of these objects. Finally, our system executes precise grasping actions using visual and textual data synthesis, thus facilitating more effective and contextually aware robotic grasping. We carry out experiments using the OCID-VLG dataset. We observe that our methodology surpasses current state-of-the-art (SOTA) techniques, delivering improvements of 9.91% and 5.70% for top-1 and top-5 predictions in grasp accuracy. Moreover, when apply to the reconstructed Grasp-MultiObject dataset, our approach demonstrates even more substantial enhancements, achieving gains of 17.63% and 22.76% over SOTA methods for top-1 and top-5 predictions, respectively. Note to Practitioners—As robotic systems evolve, the challenge of enabling them to function effectively in complex environments has become increasingly apparent. This paper introduces a solution that integrates multiple sources of data—visual, textual, and human knowledge—to enhance robotic grasping capabilities. The practical problems addressed include the limitations of current unimodal systems that struggle with accurate object recognition and manipulation in dynamic settings, such as warehouses or assembly lines. Our framework, MIG, demonstrates significant improvements in grasp accuracy, making it suitable for tasks where precision is critical. While our results show promise, particularly in controlled experiments, there are limitations to consider. The framework’s performance may vary in unstructured real-world environments due to factors like occlusion or varying lighting conditions. Future work should focus on refining the system for real-time application and exploring additional sensory inputs to enhance robustness. By addressing these challenges, we aim to make this approach more applicable across industries, paving the way for smarter, more adaptable robotic solutions in everyday tasks.

Imitation learning is an important method for the human-robot skill transfer. However, ensuring that skills learned through imitation remain effective in different environments is a challenge. This article addresses the challenge by proposing a stable autonomous dynamic system that can effectively handle obstacles and disturbances while maintaining trajectory accuracy. We introduce an energy-approximated dynamic subattractor (EADA) method that enhances disturbance resistance by dynamically selecting subattractors through Neum (an energy function derived from demonstration data). By combining velocity modulation algorithms with EADA, the system achieves global stability, precise obstacle avoidance, autonomous trajectory recovery, and rapid response. The proposed framework effectively handles complex scenarios, including environments with multiple obstacles, dynamic obstacles, and disturbances. We validate the proposed approach through simulations on the LASA dataset and real-world robotic experiments (both single-arm and dual-arm robots), demonstrating its effectiveness in achieving smooth and accurate obstacle avoidance trajectories with generalization capability.

In this article, a novel energy-efficient control method for waypoint tracking of underwater gliders is designed. The method can be divided into a planning layer and a control layer. In the planning layer, a novel steady/unsteady gliding depth intervals-based dead-reckoning is proposed to predict depth-averaged current velocity with lower consumption. Also a novel heading and depth modification strategy based on line-of-sight is proposed to implement waypoint tracking planning. In the control layer, heading control is implemented by two event-triggered extended state observers (ET-ESOs) and an event-triggered backstepping heading controller (ET-BHC). The ET-ESOs intermittently estimate the real states input to the ET-BHC, and the ET-BHC intermittently outputs the control signal. Simulation and sea trial results show that the UG achieves tracking waypoints, and in addition, energy efficiency is significantly improved.

Human-robot interaction (HRI) is a crucial component in the field of robotics, and enabling faster response, higher accuracy, as well as smaller human effort, is essential to improve the efficiency, robustness, and applicability of HRI-driven tasks. In this article, we develop a novel neuroadaptive admittance control with human motion intention (HMI) estimation and output error constraint for natural and stable interaction. First, the interaction force information of the robot is utilized to predict the HMI and the stiffness in the admittance model is dynamically updated based on surface electromyography (sEMG) signals of the human upper limb to achieve human-like compliance. Then, based on the designed error transformation mechanism, an innovative prescribed performance control (PPC) is proposed that allows the trajectory error to converge to the given constraint range within a predefined time for any bounded initial conditions, thus enabling the robot to maintain a comprehensive performance of moving in the desired direction as guided by the human. Also, an adaptive neural network (NN) is employed to compensate for the uncertainty of robotics systems to improve the tracking accuracy further. According to the Lyapunov stability analysis criterion, our approach ensures that all states of the closed-loop system remain globally uniformly ultimately bounded. Finally, a series of real-world robot experiments demonstrate the effectiveness of the proposed framework.

This article proposes a spatial learning control system for robots to achieve a desired behavior during interacting with unknown environments. In contacting with the environment, the force is estimated by a force observer, so sensing devices are not required. Motivated by the human interaction versatility, the reference trajectory of the robot is updating with a learning law such that the interacting force can be maintained at a desired level. Compared with the trajectory iteration algorithm based on time domain, which requires maintaining a fixed motion speed for each iteration, the proposed method can remove this limitation and have better feasibility. The adaptive controller with neural networks can compensate the uncertain dynamics of the system and ensure the control accuracy. Through Lyapunov's theory, the system is proved to be stable, and all the states are bounded. Comparative simulations and experiments are conducted on a robot platform to verify the effectiveness of the proposed method.

In this study, a novel fault-tolerant broad fuzzy learning control scheme is proposed for a flexible single-link manipulator with input delay and output constraints. The broad fuzzy neural network is utilized to effectively approximate the time delay of the controller and compensate for the unknown nonlinear uncertainties. By applying the Lyapunov direct method and barrier Lyapunov function, the semi-global uniformly ultimately boundedness (SGUUB) of the system is demonstrated, which guarantees that all system states converge to zero within the specified limitation. Finally, the simulation and experiment results, compared with those of different neural networks, manifest the validity of the proposed control method.

Pouring from one beaker to another is crucial in the preparation of coatings within material synthesis. In this study, a collaborative robot is utilized to imitate the behavior of experimenters in order to accomplish pouring tasks across various scenarios. Given that these tasks involve complex position-attitude relationships and the presence of obstacles, traditional rigid programming methods are hardly employed. Instead, learning from demonstration is incorporated to transfer experimenters’ pouring skills, which encompasses three phases: teaching, learning, and reproduction. We propose a vision-based dynamic movement primitives approach to generalize the skill based on visual feedback. Utilizing real-time visual information feedback regarding liquid level and beaker size, the teaching trajectory, which involves coupled position and attitude relationships, is generalized to adapt dynamically to differing experimental requirements. In experiments, we utilized the Kinect V2 camera and the Kinova Jaco2 manipulator to assess the efficacy of the proposed method.

Learning an autonomous dynamic system (ADS) encoding human motion rules has been shown as an effective way for human motion skills transfer. However, most existing approaches focus on goal-directed motion skills transfer, and the study on periodic motion skills transfer is rare. One popular approach for periodic motion skills transfer is learning periodic dynamic movement primitive (DMP); however, periodic DMP is sensitive to spatial disturbances due to the introduction of the phase parameters. To solve this issue, this brief presents a novel approach to learn an ADS with a stable limit cycle without introducing phase parameters. First, a data-driven Lyapunov function (energy function) is learned, such that one of its level surfaces is consistent with periodic human demonstration trajectories. Then, an ADS is learned by sequentially solving energy function-related constrained optimization problems. With a proper design of constraint functions, we can ensure that the trajectory generated by the ADS will converge to an energy function-level surface, of which the shape is similar to periodic human demonstration trajectories. Experiments are conducted to show the effectiveness of the proposed approach (PA).

In the realm of semantic segmentation tasks, knowledge distillation (KD) has emerged as a prominent strategy, leveraging the transfer of mature knowledge from large teacher networks to enhance the performance of smaller student networks. However, existing methods often rely heavily on high-quality yet cumbersome teacher networks, leading to a complex training process. To address this challenge, we introduce a novel approach termed self-training driven attention-guided self-mimicking online ensemble network. Our proposed method begins by employing intermediate channel-joint attention maps to guide image augmentation. Both the original and augmented images are then input into the networks. Leveraging intermediate feature maps and predictive predictions generated from the two images, we employ KD to uncover invariant features. To further harness representation potential through learning from credible predictions, we introduce a self-training mechanism. This mechanism utilizes an exponential moving average (EMA)-teacher network constructed using the exponential moving average technique to generate feature maps and predicted posterior probabilities. The knowledge of the EMA-teacher is subsequently transferred to the student network through distillation. Extensive experiments and visualization analyses conducted on multiple benchmark datasets, including Cityscapes, Pascal VOC, CamVid, and ADE20k, validate the effectiveness of self-training driven attention-guided self-mimicking network (ST-ASMNet). The interpretability of our method is further validated through visualization and analysis. Our code will be publicly available.

This brief proposes a novel neuron dynamic-growing (NDG) strategy for radial basis function neural networks (RBF NNs). Only one neuron is selected in advance relying on the system initial states, and other neurons are dynamically generated based on the designed threshold for the distance between the current NN input and the closest neuron. Compared with the RBF NN using neuron fixed evenly spaced strategy (NFES), the improved RBF NN has two major advantages: one is to extremely reduce the number of neurons, especially for the high dimensional NN inputs; and the other is to provide a theoretical criteria for the choice of NN structure parameters including the neuron center and the compact set size. To guarantee the dynamic learning ability of the improved RBF NN, the persistent excitation (PE) is verified strictly by subtly constructing the threshold and the center of newly added neurons. Simulation and experimental results illustrate that the improved RBF NN integrated into the existing dynamic learning control effectively enhances the transient control performance, reduces the computational burden, and saves data storage space.

The stability criterion is critical for the design of legged robots' motion planning and control algorithms. If these algorithms cannot theoretically ensure legged robots' stability, we need many trials to identify suitable parameters for stable locomotion. However, most existing stability criteria are tailored to robots driven solely by legs and cannot be applied to thruster-assisted legged robots. Here, we propose a stability criterion for a thruster-assisted underwater hexapod robot by finding maximum and minimum allowable thruster forces and comparing them with the current thrusts to check its stability. On this basis, we propose a method to increase the robot's stability margin by adjusting the value of thrusts. This process is called stability enhancement. The criterion uses the optimization method to transform multiple variables such as attitude, velocity, acceleration of the robot body, and the angle and angular velocity of leg joints into one kind of variable (thrust) to judge the stability directly. In addition, the stability enhancement method is straightforward to implement because it only needs to adjust the thrusts. These provide insights into how multiclass forces such as inertia force, fluid force, thrust, gravity, and buoyancy affect the robot's stability.

This article introduces CIMAP, a high-performance motion planning algorithm for robotic manipulators in complex environments, based on the clearance inference network (CIN). CIMAP incorporates a batch collision estimation module powered by CIN, which efficiently predicts collisions by dividing the manipulator’s workspace into voxels and estimating clearances between the manipulator and surrounding obstacles. The algorithm also features a batch adaptive bidirectional expansion mechanism, enabling the simultaneous extension of multiple nodes within joint space. Leveraging CIN for batch collision estimation, CIMAP accelerates the discovery of feasible paths. Additionally, CIMAP includes a phased path optimization mechanism that identifies local shortcuts through CIN, improving path efficiency. A geometric collision checker ensures safety, performing necessary repairs when required. To assess CIMAP’s effectiveness in continuous motion planning, we compared its performance against four existing algorithms (CN-RRT, B-RRT, GB-RRT*, and NPB-RRT*-DC) across various obstacle scenarios. Experimental results demonstrate that CIMAP achieves an average motion planning time of under 0.7 s, improving planning efficiency by at least 89% compared to the baseline algorithms, while maintaining shorter path lengths.

This study investigates a multitrajectory tracking control strategy for an unknown 2-DOF helicopter system, integrating deterministic learning (DL) and reinforcement learning (RL). Initially, DL theory is applied to identify the local unknown dynamics of a 2-DOF helicopter system using radial basis function neural networks (RBFNNs). Subsequently, the identified dynamic knowledge is expressed and stored using constant RBFNNs. To mitigate the issue of partial knowledge failure due to deviations between the actual and learned trajectories, we introduce a RL framework for dynamic compensation. Finally, a composite control strategy incorporating both nominal and auxiliary components is designed to achieve multitrajectory tracking control. The stability of the closed-loop system is analyzed and demonstrated using the Lyapunov direct method. The simulation and experimental results demonstrate the effectiveness of the proposed control strategy.

With the growing demand for robots in emerging fields, such as smart medical and home services, their ability to interact with soft environments has received increased attention. Nevertheless, an overlooked issue is that the inadequate description of soft environments using a linear model may significantly diminish the accuracy of interaction control. In this article, a neural-network-based impedance control framework is proposed for robots to physically interact with soft environments and optimize interaction performance. Specifically, a nonlinear definition of soft environments is introduced based on the Hunt–Crossley (HC) model, with parameter identification utilizing a data-driven technique. Regarding system performance evaluated by a cost function, the determination of interaction behavior described by the impedance model is transformed into an optimal control problem. Moreover, to address model uncertainties, the original optimal control problem is redefined using a modified cost function with a constructed auxiliary system. Then, a critic network is employed to approximate the nonlinear optimal solution, thereby avoiding complicated mathematical derivations. Finally, the effectiveness of the proposed impedance adaptation strategy is validated through both simulations and experiments. Numerical results indicate that both the convergent cost and total cost are significantly reduced based on the proposed method compared to the linear-model-based impedance control, particularly for materials with viscoelastic properties, achieving a reduction of up to 30%.

The assurance of faster transient response rate, higher steady-state tracking accuracy, and global adaptability are crucial for enhancing the efficiency and robustness of manipulators during operation. This article explores a novel fuzzy adaptive predefined-time controller based on global prescribed performance for robotic systems with unknown dynamics and bounded disturbances. First, a predefined time error transformation function (PTETF) is developed and combined with a barrier function based on constant value constraints for control design, which not only significantly simplifies the derivation process of the proposed predefined time prescribed performance control (PTPPC), but also equips it with the ability of global constraints on the trajectory tracking error. Then, we utilize the fuzzy logic system (FLS) with a single-parameter update mechanism to compensate for the dynamic uncertainty of manipulators, thereby reducing computational complexity and cost. In addition, a fixed time disturbance observer (FxTDOB) is introduced to alleviate the effect of nonparametric disturbances on the tracking performance. Further, by integrating the predefined time theory with Lyapunov method to ensure that all state signals of the controlled system converge in a predefined time. Finally, numerical simulations and practical experiments are carried out to demonstrate the effectiveness of the proposed framework.

In this paper, we present the TacShade: a newly designed 3D-printed soft optical tactile sensor. The sensor is developed for shape reconstruction under the inspiration of sketch drawing that uses the density of sketch lines to draw light and shadow, resulting in the creation of a 3D-view effect. TacShade, building upon the strengths of the TacTip, a single-camera tactile sensor of large in-depth deformation and being sensitive to edge and surface following, improves the structure in that the markers are distributed within the gap of papillae pins. Variations in light, dark and grey effects can be generated inside the sensor under the external contact interactions. The contours of the contacting objects are outlined by white markers, while the contact depth characteristics can be indirectly obtained from the distribution of black pins and white markers, creating a 2.5D visualization. Based on the imaging effect, we improve the Shape from Shading (SFS) algorithm to process tactile images, enabling a coarse but fast reconstruction for the contact objects. Two experiments are performed. The first verifies TacShade’s ability to reconstruct the shape of the contact objects through one image for object distinction. The second experiment shows the shape reconstruction capability of TacShade for a large panel with ridged patterns based on the location of robots and image splicing technology.

Recognizing and grasping novel-category objects remains a crucial yet challenging problem in real-world robotic applications. Despite its significance, limited research has been conducted in this specific domain. To address this, we seamlessly propose a novel framework that integrates open-vocabulary learning into the domain of robotic grasping, empowering robots with the capability to adeptly handle novel objects. Our contributions are threefold. Firstly, we present a large-scale benchmark dataset specifically tailored for evaluating the performance of open-vocabulary grasping tasks. Secondly, we propose a unified visual-linguistic framework that serves as a guide for robots in successfully grasping both base and novel objects. Thirdly, we introduce two alignment modules designed to enhance visual-linguistic perception in the robotic grasping process. Extensive experiments validate the efficacy and utility of our approach. Notably, our framework achieves an average accuracy of 71.2% and 64.4% on base and novel categories in our new dataset, respectively. Our code and dataset are available at https://github.com/cv516Buaa/OVGNet.

Neuroscience research has demonstrated the sig-nificance of modulating stiffness during human task performance. Similarly, endowing robots with such capability is expected. However, existing methods for robot teleoperation require operators to simultaneously control position and stiffness, resulting in high workload and task inefficiency. On the other hand, learning from demonstration (LfD) offers a feasible approach for autonomously generating stiffness. Therefore, this paper proposes a robot teleoperation system that combines the advantages of teleoperation and LfD. Teleoperation enables precise positioning guided by human operators, while LfD can transfer human stiffness skills to robots. A teleoperation-oriented stiffness-adaptive Gaussian Mixture Model/Gaussian Mixture Regression method is proposed to learn human multi-dimensional stiffness and reproduce robot stiffness on a Riemannian manifold. To enhance generalization and cooperate with teleoperation, reference points and position-driven output are introduced. Furthermore, a teleoperation strategy for both the single-leader-single-follower configuration and the single-leader-dual-follower configuration are designed, which allows operators to control either one or two robot arms with a single leader device. Finally, the effectiveness of our method is verified through a plugging-in task and a continuous flipping task, demonstrating that the proposed system is capable of performing tasks that demand high positioning accuracy and stiffness adjustment. A supplementary video for this paper is available in GitHub**https://github.com/setowenGit/TOSA-GMM-GMR-Video.

This paper presents a neural optimization-based fixed-time adaptive control scheme for robot systems with unknown dynamics and input saturation. During the process of information exploration, security and control efficiency issues always exist due to the complexity of the system. In this regard, a performance index function is constructed to optimize control performance, and a nonlinear auxiliary compensation system is developed to solve the saturation effect of the actuator. By solving the Hamilton–Jacobi–Bellman (HJB) equation and utilizing fixed-time theory, a fixed-time optimization control scheme is designed within the framework of adaptive dynamic programming. The objective of this scheme is to achieve both optimal performance and rapid convergence. Secondly, universal approximators, namely neural network (NNs), are employed to handle unknown uncertainties through the actor-critic-identifier structure. Among them, the critic network evaluates system performance, the actor network implements control actions, and the identifier network estimates unknown dynamics. Additionally, under the Lyapunov stability criterion and optimization theory, a stability analysis is conducted to demonstrate the feasibility of the devised neuro-optimal fixed-time control scheme and guarantee the convergence of all signals within a fixed-time. Finally, simulations are performed to further validate the effectiveness of the developed control method.

6D pose estimation with individual modality encounters difficulties due to the limitations of modalities, such as RGB information on textureless objects and depth on reflective objects. This can be improved by exploiting the complementarity between modalities. Most of the previous methods only consider the correspondence between point clouds and RGB images and directly extract the features of the corresponding two modalities for fusion, which ignore the information of the modality itself and are negatively affected by erroneous background information when introducing more features for fusion. To enhance the complementarities between multiple modalities, we propose a neighbor-based cross-modalities attention mechanism for multi-modal 6D pose estimation. Neighbors represent that the RGB features of multiple neighbor are applied for fusion, which expands the receptive field. The cross-modalities attention mechanism leverages the similarities between the different modal features to help modal feature fusion, which reduces the negative impact of incorrect background information. Moreover, we design some features between the rendered image and the original image to obtain the confidence of pose estimation results. Experimental results on LM, LM-O and YCB-V datasets demonstrate the effectiveness of our methods. Video is available at https://www.youtube.com/watch?v=ApNBcX6NEGs.Note to Practitioners—Introducing the information of surrounding points during multi-modal fusion improves the performance of 6D pose estimation. For example, the RGB image corresponding to some point clouds on the object may lack rich texture features while the neighbors exist. However, most methods of modal fusion based on RGBD for 6D pose estimation only simply consider the corresponding between RGB images and point clouds for feature fusion, which may bring redundant information or the wrong background information when introducing neighbor information. In this paper, we propose a cross-modal attention mechanism based on neighbor information. By introducing the information of the modality itself to obtain the weight of the neighbor information of another modality in the encoding and decoding stages, the receptive field is expanded and the complementarities between different modalities are enhanced. The experiment shows our effectiveness. In addition, we provide a pose confidence estimator for predicted pose results. Specifically, the rendered image with the predicted pose and the real image are applied to extract features for the decision tree. The experimental results show that the result of the wrong estimation can be eliminated with high accuracy and recall. The 6D pose confidence can provide a reference for real-world grasping. However, the current method can only estimate objects with known models. In the future, we will consider applying the method to unseen objects.

In this paper, a distributed observer-based prescribed performance control method is proposed for using a multi-robot teleoperation system to manipulate a common deformable object. To achieve a stable position-tracking effect and realize the desired cooperative operational performance, we first define a new hybrid error matrix for both the relative distances and absolute positions of robots and then decompose the matrix into two new error terms for cooperative and independent robot control. Then, we improve the Kelvin-Voigt (K-V) contact model based on the new error terms. Because the center position and deformation of the object cannot be measured, the object dynamics are then expressed by the relative distances of robots and an equivalent impedance term. Each robot incorporates an observer to estimate contact force and object dynamics based on its own measurements. To address the position errors caused by biases in force estimation and realize the position-tracking effect of each robot, we improve the barrier Lyapunov functions (BLFs) by incorporating the errors into system control. which allows us to achieve a predefined position-tracking effect. We conduct an experiment to verify the proposed controller’s ability in a dual-telerobot cooperative manipulation task, even when the object is subjected to unknown disturbances.Note to Practitioners—This article is inspired by the limitations of multi-telerobot manipulation with a deformable object, where the deformation of the object cannot be measured directly. Meanwhile, force sensors, especially 6-axis force sensors, are very expensive. To realize the purpose that objects manipulated by multiple robots match the same state as operated on the leader side, we propose an object-centric teleoperation framework based on the estimates of contact forces and object dynamics and the improved barrier Lyapunov functions (BLFs). This framework contributes to two aspects in practice: 1) propose a control diagram for deformable object co-teleoperation of multi-robots for unmeasurable object’s centre position and deformation; 2) propose an improved BLFs controller based on the estimation of contact force and robot dynamics. The estimation errors are considered and transferred using an equivalent impedance to be integrated into the Lyapunov function to minimize both force and motion-tracking errors. The experimental results verify the effectiveness of the proposed method. The developed framework can be used in industrial applications with a similar scenario.

Traditional deep learning-based visual imitation learning techniques require a large amount of demonstration data for model training, and the pre-trained models are difficult to adapt to new scenarios. To address these limitations, we propose a unified framework using a novel progressive learning approach comprised of three phases: i) a coarse learning phase for concept representation, ii) a fine learning phase for action generation, and iii) an imaginary learning phase for domain adaptation. Overall, this approach leads to a one-shot domain-adaptive imitation learning framework. We use robotic pouring as an example task to evaluate its effectiveness. Our results show that the method has several advantages over contemporary end-to-end imitation learning approaches, including an improved success rate for task execution and more efficient training for deep imitation learning. In addition, the generalizability to new domains is improved, as demonstrated here with novel backgrounds, target containers, and granule combinations in the experiment. We believe that the proposed method is broadly applicable to various industrial or domestic applications that involve deep imitation learning for robotic manipulation, and where the target scenarios are diverse and human demonstration data is limited. For project video, please check our website:https://sites.google.com/view/imitation-learning-tase2022. Note to Practitioners—The motivation of this paper is to develop a progressive learning framework, which can be used for both service and industrial robots to learn from human demonstrations, and then transfer the learned skill to different scenarios with ease. We use the robotic pouring task as an example to demonstrate the effectiveness of our proposed method, since pouring is an essential skill for service robots to assist humans’ daily lives, and can benefit robot automation in wet-lab industries. The aim of this research is to enable robots to obtain visuomotor skills (such as the pouring skill), and accomplish the tasks with a high success rate using our proposed progressive learning method. We conducted experiments to show that the proposed method has good performance, high data efficiency and evident generalizability. This is significant for intelligent robots working in various practical applications.

This article investigates an adaptive neural network (NN) control technique with fixed-time tracking capabilities, employing composite learning, for manipulators under constrained position error. The first step involves integrating the composite learning method into the NN to address the dynamic uncertainties that inevitably arise in manipulators. A composite adaptive updating law of NN weights is formulated, requiring adherence solely to the relaxed interval excitation (IE) conditions. In addition, for the output error, instead of knowing the initial conditions, this article integrates the error transfer function and asymmetric barrier function to achieve the specific performance for position error in both steady and transient states. Furthermore, the fixed-time control methodology and Lyapunov stability criterion are synergistically employed in order to guarantee the convergence of all signals in the manipulators to a compact neighborhood around the origin within a fixed-time. Finally, numerical simulation and experiments with the Baxter robot results both determine the capability of the NN composite learning technique and fixed-time control strategy.

Precise trajectory control is imperative to ensure the safety and efficacy of in vivo therapy employing the magnetic helical millirobots. However, achieving accurate 3-D path following of helical millirobots under fluid flow conditions remains challenging due to the presence of the lumped disturbances, encompassing complex fluid dynamics and input frequency saturation. This study proposes a robust 3-D path following control framework that combines a disturbance observer for perturbation estimation with an adaptive finite-time sliding mode controller for autonomous navigation along the reference trajectories. First, a magnetic helical millirobot's kinematic model based on the 3-D hand position approach is established. Subsequently, a robust smooth differentiator is implemented as an observer to estimate disturbances within a finite time. We then investigate an adaptive finite-time sliding mode controller incorporating an auxiliary system to mitigate the estimated disturbance and achieve precise 3-D path tracking while respecting the input constraints. The adaptive mechanism of this controller ensures fast convergence of the system while alleviating the chattering effects. Finally, we provide a rigorous theoretical analysis of the finite-time stability of the closed-loop system based on the Lyapunov functions. Utilizing a robotically-actuated magnetic manipulation system, experimental results demonstrate the efficacy of the proposed approach in terms of the control accuracy and convergence time.

In this paper, an adaptive fuzzy logic system (FLS) predefined time tracking control technique is investigated for robot systems with constrained position errors. To this end, a practical predefined time control approach that integrates FLSs learning technique, predefined time stability criterion, tan-type constraint function and the backstepping recursive design is constructed for the first time. Compared with finite/fixed-time control schemes, the upper bound of the convergence time no longer depends on initial conditions or complex design parameters. Instead, it can be predefined by making adjustments to a single relevant control parameter in advance. In order to avoid the violation of constraint boundaries by position tracking errors, a suitable tan-type barrier Lyapunov function is constructed, and rigorous stability is proved derived from constrained predefined time Lyapunov theory. The results demonstrate that the error signals converge to a small range near origin within the user-defined time. Additionally, to compensate for unknown nonlinearity, a new adaptive update law associated with the convergence time control parameter is established. Simulations and experiments on the Baxter robot platform are carried out to confirm the validity and functionality of the devised controller.

Visual servoing technology has widely been employed in manufacturing because it is a flexible, realizability, and low-cost way to improve the intelligence of the industry robot. Nevertheless, a worrisome and overlooked issue is that the loss of visual features in the camera's field of view may lead to the failures of the visual servoing tasks. This article addresses the visual features escaping problem, by implementing an asymmetric barrier Lyapunov function with a field-of-view constraint controller. The asymmetric barrier Lyapunov function defines a tightly specified range for the feature coordinate errors and ensures the transient response of the tracking error as well as enables arbitrary tracking accuracy. It is worth noting that the asymmetric barrier Lyapunov function directly handles the visual-robot-coupled dynamics while guaranteeing system stabilities. Besides, to accommodate the uncertain dynamics derived from a high-dimensional coupled system, an adaptive controller is proposed utilizing fuzzy neural networks with computational efficiency and few training parameters to enhance the control performance. Finally, the effectiveness of the proposed control strategy has been demonstrated through both theoretical analysis and experimental verification.

One promising function of interactive robots is to provide a specific interaction force to human users. For example, rehabilitation robots are expected to promote patients' recovery by interacting with them with a prescribed force. However, motion uncertainties of different individuals, which are hard to predict due to the varying motion speed and noises during motion, degrade the performance of existing control methods. This article proposes a method to learn a desired reference trajectory for a robot based on dynamic motion primitives (DMPs) and iterative learning (IL). By controlling the robot to follow the generated desired reference trajectory, the interaction force can achieve a desired value. In our proposed approach, DMPs are first employed to parameterize the demonstration trajectories of the human user. Then, a recursive least square (RLS)-based estimator is developed and combined with the Adam optimization method to update the trajectory parameters so that the desired reference trajectory of the robot is iteratively obtained by resolving the DMPs. Since the proposed method parameterizes the trajectories depending on the phase variable, it removes the essential assumption of traditional IL methods that the iteration period should be invariant, and thus, has improved robustness compared with the existing methods. Experiments are performed using an interactive robot to validate the effectiveness of our proposed scheme.

In consideration of the limitation of the communication and the possibility that redundant robots might deliver information at different power levels, cases under weight-unbalanced directed graphs from the network topology perspective are in larger accordance with those in multiple redundant robot systems. By moving forward along this direction, a distributed controller is proposed in this article to handle circumstances of collaborative control of multiple redundant robots under weight-unbalanced directed graphs. This kind of control problem is modeled into generalized quadratic programming (QP) problems with equality and inequality constraints. Then, the above QP problems are solved by a proposed neural-dynamics-based method, whose stability and convergence are theoretically proved subsequently. Besides, several experimental examples are conducted, and related comparisons are provided to demonstrate the feasibility of the proposed controller.

In this article, an adaptive neural network (NN) predefined-time tracking control strategy is investigated for robot systems with external disturbance. First, under the predefined-time stability criterion, a new time-controlled torque controller is constructed, which allows for the system convergence time to be set beforehand. This is conducive to manipulators performing trajectory tracking tasks that require specific convergence times. In addition, the continuous terms are constructed by smoothly switching between the fractional and cubic terms of state-dependence. This solution successfully resolves the issues of singularity. Moreover, in order to compensate for unknown nonlinearity and torque disturbance, two different adaptive update laws are established, respectively. Furthermore, rigorous stability is proved based on the predefined-time Lyapunov theory. Finally, the accuracy and efficiency of the NN-based predefined-time control algorithm is confirmed and validated through both numerical simulations and practical experiments conducted with the Baxter robot.

Humans are able to compliantly interact with the environment by adapting its motion trajectory and contact force. Robots with the human versatility can perform contact tasks more efficiently with high motion precision. Motivated by multiple capabilities, we develop an approximation-based admittance control strategy that adapts and tracks the trajectory with guaranteed performance for the robots interacting with unknown environments. In this strategy, the robot can adapt and compensate its feedforward force and stiffness to interact with the unknown environment. In particular, a reference trajectory is generated through the admittance control to achieve a desired interaction level. To improve the interaction performance, a tracking error bound for both the transient and steady states is prespecified, and a controller is designed to ensure the tracking control performance. In the presence of unknown robot dynamics, neural networks are integrated into tracking controller to compensate uncertainties. The stability and convergence conditions of the closed-loop system are analysed by the Lyapunov theory. The effectiveness of the proposed control method is demonstrated on the Baxter robot.

This article proposes an adaptive inverse neural network (NN) control of an uncertain flexible single-link manipulator with input backlash and output constraint. First, an adaptive inverse function is applied to eliminate the input backlash of the actuator. Second, an NN is applied to approximate the system uncertainty. Third, a barrier Lyapunov function is used to guarantee that the system is maintained within the constraints. Subsequently, the system’s semi-globally uniformly ultimately bounded stability is proved by the Lyapunov direct method. Finally, the simulation and experimental results manifest the feasibility of the proposed controller.

Meta-reinforcement learning enables artificial agents to learn from related training tasks and adapt to new tasks efficiently with minimal interaction data. However, most existing research is still limited to narrow task distributions that are parametric and stationary, and does not consider out-of-distribution tasks during the evaluation, thus, restricting its application. In this paper, we propose MoSS, a context-based Meta-reinforcement learning algorithm based on Self-Supervised task representation learning to address this challenge. We extend meta-RL to broad non-parametric task distributions which have never been explored before, and also achieve state-of-the-art results in non-stationary and out-of-distribution tasks. Specifically, MoSS consists of a task inference module and a policy module. We utilize the Gaussian mixture model for task representation to imitate the parametric and non-parametric task variations. Additionally, our online adaptation strategy enables the agent to react at the first sight of a task change, thus being applicable in non-stationary tasks. MoSS also exhibits strong generalization robustness in out-of-distributions tasks which benefits from the reliable and robust task representation. The policy is built on top of an off-policy RL algorithm and the entire network is trained completely off-policy to ensure high sample efficiency. On MuJoCo and Meta-World benchmarks, MoSS outperforms prior works in terms of asymptotic performance, sample efficiency (3-50x faster), adaptation efficiency, and generalization robustness on broad and diverse task distributions.

Tactile sensors can be used for motion detection and object perception in robot manipulation. The contact detection within the camera's visual inspection area has been well-developed, but perception outside the field of view of the camera is overlooked. In this paper, we present a new tendon-linked tactile sensor, MechTac, to achieve perceptions inside and outside the field of view. The MechTac is an evolution of the typical TacTip sensor with the following two advantages. 1) The ability to provide perception outside the field of view. This is achieved by using a network of braided tendons to transfer deformation from the blind perception regions (TacSide) to the visual areas (TacTip). 2) The tactility of the TacSide and TacTip is reflected by the movements of multiple papillae pins and visible markers on the pin tips on the inner surface of the TacTip. The pins and markers are differentially sensitive to various touch features, which is similar to the differentiated perceptual ability of humans. TacTip is more sensitive to small touches, corresponding to the fingertip, while the TacSide is less sensitive but has a larger perceptual area, corresponding to the middle part of the finger. Moreover, we propose a new deep learning method to decompose the mixed information affected by the TacSide and the TacTip. A modified DenseNet121 was specifically designed for object perception at the TacTip and localization at the TacSide. The experimental results show that prediction accuracy reaches about 98% for object perception or localization and over 99% for the case requiring two functions.

This paper proposes a whole-body learning from demonstration (LfD) framework that enables differential drive mobile manipulators to learn coordination working and disturbance rejection. First, an efficient kinesthetic teaching method is devised based on the weighted least-norm (WLN) inverse kinematics solution and an admittance controller, which facilitates human users to guide the mobile manipulator to perform tasks. Second, we propose a whole-body LfD framework through Gaussian Process, which endows the mobile manipulator's skill learning process with features of large-scale convergence, coordination working and disturbance rejection, after just a few human demonstrations. The proposed learning framework also allows for human-in-the-loop correction when the whole-body is conducting a task. Finally, the effectiveness of the proposed framework is verified via two simulations and a pick-and-place experiment. Supplementary video for this paper is available in github††https://github.com/yuqiang-yang/SMC2023-Video.

Due to changes in the environment and errors that occurred during skill initialization, the robot's operational skills should be modified to adapt to new tasks. As such, skills learned by the methods with fixed features, such as the classical Dynamical Movement Primitive (DMP), are difficult to use when the using cases are significantly different from the demonstrations. In this work, we propose an incremental robot skill learning and generalization framework including an incremental DMP (IDMP) for robot trajectory learning and an adaptive neural network (NN) control method, which are incrementally updated to enable robots to adapt to new cases. IDMP uses multi-mapping feature vectors to rebuild the forcing function of DMP, which are extended based on the original feature vector. In order to maintain the original skills and represent skill changes in a new task, the new feature vector consists of three parts with different usages. Therefore, the trajectories are gradually changed by expanding the feature and weight vectors, and all transition states are also easily recovered. Then, an adaptive NN controller with performance constraints is proposed to compensate dynamics errors and changed trajectories after using the IDMP. The new controller is also incrementally updated and can accumulate and reuse the learned knowledge to improve the learning efficiency. Compared with other methods, the proposed framework achieves higher tracking accuracy, realizes incremental skill learning and modification, achieves multiple stylistic skills, and is used for obstacle avoidance with different heights, which are verified in three comparative experiments.

Abstract In robot-assisted rehabilitation, the performance of robotic assistance is dependent on the human user’s dynamics, which are subject to uncertainties. In order to enhance the rehabilitation performance and in particular to provide a constant level of assistance, we separate the task space into two subspaces where a combined scheme of adaptive impedance control and trajectory learning is developed. Human movement speed can vary from person to person and it cannot be predefined for the robot. Therefore, in the direction of human movement, an iterative trajectory learning approach is developed to update the robot reference according to human movement and to achieve the desired interaction force between the robot and the human user. In the direction normal to the task trajectory, human’s unintentional force may deteriorate the trajectory tracking performance. Therefore, an impedance adaptation method is utilized to compensate for unknown human force and prevent the human user drifting away from the updated robot reference trajectory. The proposed scheme was tested in experiments that emulated three upper-limb rehabilitation modes: zero interaction force, assistive and resistive. Experimental results showed that the desired assistance level could be achieved, despite uncertain human dynamics.

Abstract In this paper, composite dynamic movement primitives (DMPs) based on radial basis function neural networks (RBFNNs) are investigated for robots’ skill learning from human demonstrations. The composite DMPs could encode the position and orientation manipulation skills simultaneously for human-to-robot skills transfer. As the robot manipulator is expected to perform tasks in unstructured and uncertain environments, it requires the manipulator to own the adaptive ability to adjust its behaviours to new situations and environments. Since the DMPs can adapt to uncertainties and perturbation, and spatial and temporal scaling, it has been successfully employed for various tasks, such as trajectory planning and obstacle avoidance. However, the existing skill model mainly focuses on position or orientation modelling separately; it is a common constraint in terms of position and orientation simultaneously in practice. Besides, the generalisation of the skill learning model based on DMPs is still hard to deal with dynamic tasks, e.g., reaching a moving target and obstacle avoidance. In this paper, we proposed a composite DMPs-based framework representing position and orientation simultaneously for robot skill acquisition and the neural networks technique is used to train the skill model. The effectiveness of the proposed approach is validated by simulation and experiments.

This paper focuses on addressing the visual tracking problem using learning-based methods for object tracking tasks. This problem contains a major difficulty, i.e., how to acquire a satisfactory generalization ability of the developed system? In this paper, firstly, the object state tracking system, including a camera-in-hand, a 3D camera and a Rethink Baxter robot, is introduced. The problem formulation is also presented. Secondly, we propose a Kalman-based estimation strategy to acquire the object’s state. In addition, a learning-based tracking controller is developed using the Gaussian mixture models (GMM) method to steer the robot end-effector to track the mobile object. Thirdly, to guarantee system stability (i.e., the position and velocity errors between the object and end-effector will always converge to zeros), the controller parameter constraints are derived, which is a theoretical contribution of this paper. The controller parameter adjustment is avoided by the proposed training process. Thus, the proposed method becomes easy to implement, which is a practical contribution. Finally, the effectiveness of the proposed method is demonstrated by simulation and experimental examples, and the proposed method has satisfactory generalization ability. Note to Practitioners—This paper studies object tracking problems for different practical applications, such as industrial cutting, grasping and dynamic monitoring. Different trajectory tracking methods have been widely applied in the industrial area. However, users always complain that when the object or trajectory is changed, the tracking controller more or less needs to be re-adjusted. This re-adjust process always requires professional knowledge and programming experience, and thus a factory must employ some professional engineers. In addition, since the objects may be diverse in shape, color and size, to acquire the accurate object position and velocity, an appropriate solution is necessary. Motivated by the above introductions, this paper aims to develop a learning-based controller to track different complex trajectories without frequent and specific parameter adjustment processes. Firstly, a visual measurement system is developed to quickly find and estimate the position and velocity of an object. Secondly, with the object’s information, the learning from demonstration method (GMM method) is applied for the control policy design. Thirdly, the detailed system stability analysis is presented, and the corresponding controller parameter constraints are derived and considered in the proposed control policy. Subsequently, with the demonstration data, the packaged learning algorithm will automatically compute the controller parameters, and users can change the controller performance only by providing the desired demonstrations. In summary, this paper proposes a systematic object tracking solution, and it may bring a new idea to develop a practical object tracking system, using both the learning-based methods to improve the ability of generalization for tracking different objects.

Robot manipulation, for example, pick-and-place manipulation, is broadly used for intelligent manufacturing with industrial robots, ocean engineering with underwater robots, service robots, or even healthcare with medical robots. Most traditional robot manipulations adopt 2-D vision systems with plane hypotheses and can only generate 3-DOF (degrees of freedom) pose accordingly. To mimic human intelligence and endow the robot with more flexible working capabilities, 3-D vision-based robot manipulation has been studied. However, this task is still challenging in the open world especially for general object recognition and pose estimation with occlusion in cluttered backgrounds and human-like flexible manipulation. In this article, we propose a comprehensive analysis of recent progress about the 3-D vision for robot manipulation, including 3-D data acquisition and representation, robot-vision calibration, 3-D object detection/recognition, 6-DOF pose estimation, grasping estimation, and motion planning. We then present some public datasets, evaluation criteria, comparisons, and challenges. Finally, the related application domains of robot manipulation are given, and some future directions and open problems are studied as well.

The widespread application of learning-based methods in robotics has allowed significant simplifications to controller design and parameter adjustment. In this article, robot motion is controlled with learning-based methods. A control policy using a broad learning system (BLS) for robot point-reaching motion is developed. A sample application based on a magnetic small-scale robotic system is designed without detailed mathematical modeling of the dynamic systems. The parameter constraints of the nodes in the BLS-based controller are derived based on Lyapunov theory. The design and control training processes for a small-scale magnetic fish motion are presented. Finally, the effectiveness of the proposed method is demonstrated by convergence of the artificial magnetic fish motion to the targeted area with the BLS trajectory, successfully avoiding obstacles.

An adaptive fuzzy prescribed-time connectivity-preserving consensus protocol is designed for a class of stochastic nonstrict-feedback multiagent systems, in which periodic disturbances, switched nonlinearities, input saturation, and limited communication ranges are taken into consideration simultaneously. The connectivity, determined by the limited communication ranges and initial positions of agents, is preserved by incorporating an error transformation. Further, a common Lyapunov function is considered to deal with the switching modes. By combining a reduced fuzzy logic system with Fourier series expansion, a novel approximator is constructed to deal with periodically disturbed nonlinearities and to surmount the difficulty brought by the nonstrict-feedback structure. More importantly, distinctly from the existing finite/fixed-time control strategies where the settling time is heavily dependent on the accurate value of the initial states and control parameters, the settling time of the proposed prescribed-time consensus is completely independent of the initialization and control parameters and can be given a priori only according to actual demands. Based on the Lyapunov stability theory, the designed controller ensures that the connectivity-preserving consensus is achieved in prescribed time and all the signals remain bounded in probability. To the end, the feasibility of the proposed consensus protocol is demonstrated by simulation.

In this article, an adaptive fixed-time fuzzy control scheme is proposed for an uncertain robot manipulator with user-defined performance. A novel symmetrical barrier Lyapunov function is designed based on the error conversion mechanism and the performance function such that the tracking errors will not violate the prescribed output constraints. A novel adaptive law is constructed and incorporated into the fixed-time controller design such that all the closed-loop signals can be bounded and achieve practical fixed-time convergence regardless of the initial conditions. Finally, the feasibility and superiority of the proposed scheme are demonstrated based on simulation and experimental studies using a Baxter robot.

This article provides solutions for a single-leader–dual-follower teleoperation system to collaboratively transport an object. First, to regulate the direct-teleoperated follower robot (DFR), we employ a relative pose transformation algorithm, based on “refixing” the leader and DFR together, to enable that the operator can ergonomically guide DFR without requiring any specific initial position, ensuring a higher teleoperation precision at the same time. Second, to regulate the assistive follower robot (AFR), we provide an efficient technique to acquire the correct orientation to achieve holding. In addition, we devise an adjustable artificial potential field method to autonomously regulate AFR's position to a ready-to-hold position, where the operator's motion is involved. At last, based on the combination of the autoregressive model and the impedance model, we generate a reference trajectory for AFR to follow, which enables the followers to hold a rigid or a deformable object with a desired contact force. Simulations and experimental results verify the feasibility and effectiveness of the proposed method.

In this article, a robust control scheme is proposed for robots to achieve an optimal performance in the process of interacting with external forces from environments. The environmental dynamics are defined as a linear model, and the interaction performance is evaluated by a defined cost function, which is composed of trajectory errors and force regulation. Based on admittance control, the reference adaptation method is used to minimize the cost function and achieve the optimal interaction performance. To make the trajectory tracking controller robust to the unknown disturbance of internal system dynamics, an auxiliary system is defined and the approximation optimal controller is designed. Experiments on the Baxter robot are conducted to verify the effectiveness of the proposed method.

Multifingered hand dexterous manipulation is quite challenging in the domain of robotics. One remaining issue is how to achieve compliant behaviors. In this work, we propose a human-in-the-loop learning-control approach for acquiring compliant grasping and manipulation skills of a multifinger robot hand. This approach takes the depth image of the human hand as input and generates the desired force commands for the robot. The markerless vision-based teleoperation system is used for the task demonstration, and an end-to-end neural network model (i.e., TeachNet) is trained to map the pose of the human hand to the joint angles of the robot hand in real-time. To endow the robot hand with compliant human-like behaviors, an adaptive force control strategy is designed to predict the desired force control commands based on the pose difference between the robot hand and the human hand during the demonstration. The force controller is derived from a computational model of the biomimetic control strategy in human motor learning, which allows adapting the control variables (impedance and feedforward force) online during the execution of the reference joint angles. The simultaneous adaptation of the impedance and feedforward profiles enables the robot to interact with the environment compliantly. Our approach has been verified in both simulation and real-world task scenarios based on a multifingered robot hand, that is, the Shadow Hand, and has shown more reliable performances than the current widely used position control mode for obtaining compliant grasping and manipulation behaviors.

Learning a stable autonomous dynamic system (ADS) encoding human motion rules has been shown as an effective way for demonstration learning. However, the stability guarantee may sacrifice the demonstration learning accuracy. This article solves the issue by learning a stability certificate, represented by a neural energy function, on the demonstration set. We propose a polarlike space analysis approach to derive parameter constraints to guarantee the unique-minimum property of the neural energy function, which is essential for it to be a cogent stability certificate. Then, the neural energy function is learned to capture the demonstration preferences via constrained optimization algorithms. With the learned neural energy function, a globally asymptotically stable ADS with predefined position constraint is further formulated. We also quantitatively analyze the generalization ability of the learned ADS by utilizing the substantial flexibility of the neural energy function. The effectiveness of the proposed approach is validated on the LASA dataset and two representative robotic experiments.

Previous works have developed impedance control to increase safety and improve performance in contact tasks, where the robot is in physical interaction with either an environment or a human user. This article investigates impedance learning for a robot guided by a human user while interacting with an unknown environment. We develop automatic adaptation of robot impedance parameters to reduce the effort required to guide the robot through the environment, while guaranteeing interaction stability. For nonrepetitive tasks, this novel adaptive controller can attenuate disturbances by learning appropriate robot impedance. Implemented as an iterative learning controller, it can compensate for position dependent disturbances in repeated movements. Experiments demonstrate that the robot controller can, in both repetitive and nonrepetitive tasks: first, identify and compensate for the interaction, second, ensure both contact stability (with reduced tracking error) and maneuverability (with less driving effort of the human user) in contact with real environments, and third, is superior to previous velocity-based impedance adaptation control methods.

Grasping of objects using multi-fingered robotic hands often fails due to small uncertainties in the hand motion control and the object's pose estimation. To tackle this problem, we propose a grasping adjustment strategy based on tactile seroving. Our technique employs feedback from a sensorized multi-fingered robotic hand to collaboratively servo the fingers and palm to achieve the desired grasp. We demonstrate the performance of our method through simulation and physical experiments by having a robot grasp different objects under conditions of variable uncertainty. The results show that our approach achieved a higher success rate and tolerated greater uncertainty than an open-looped grasp.

At present, there are about 70 million deaf people using sign language in the world, but for most normal people, it is difficult to understand the meaning of the sign language expression. Therefore, it is of great importance to explore the ways of recognising the sign language. In this paper, we propose a dynamic sign language recognition method based on the modified long short-term memory (LSTM) model. Firstly, we use Leap Motion to collect the features of Chinese Sign Language (CSL). LSTM has a good effect in processing time series data, but the parameters of its hidden layer are shared, making it important information lost when dealing with long time series. The attention mechanism can give different attention weights to different features according to the correlation between the input data and output data, so as to enhance the model’s attention to key information. Therefore, we combine LSTM with attention mechanism for dynamic sign language recognition. Experimental results show that the recognition accuracy of the modified LSTM model is 99.55%, which is higher than that of LSTM model. Finally, we developed a sign language human-computer interaction system, which verifies the real-time performance and effectiveness of the method proposed in this paper.

In this work, a human motion intention prediction method based on an autoregressive (AR) model for teleoperation is developed. Based on this method, the robot’s motion trajectory can be updated in real time through updating the parameters of the AR model. In the teleoperated robot’s control loop, a virtual force model is defined to describe the interaction profile and to correct the robot’s motion trajectory in real time. The proposed human motion prediction algorithm acts as a feedforward model to update the robot’s motion and to revise this motion in the process of human–robot interaction (HRI). The convergence of this method is analyzed theoretically. Comparative studies demonstrate the enhanced performance of the proposed approach. Note to Practitioners—In general, the robot trajectory is predetermined and it does not consider the influence of the interaction profiles in terms of position and interaction force between the human and the robot. In addition, it is hard to quantify the influence of interaction profile for the robot trajectory. For teleoperation, an AR-based model is proposed to predict the trajectory of the human and then to update the trajectory of the robot. The developed method includes the following aspects: 1) the robot trajectory can be regulated based on the interaction profiles; 2) the feedforward model can estimate the trajectory of the human to achieve the purpose of human intention recognition in advance for the robot; and 3) the proposed method can be potentially utilized for telerehabilitation, microsurgery, and so on.

In this article, an iterative-learning-based robotic controller is developed, which aims at providing a prescribed assistance or resistance force to the human user. In the proposed controller, the characteristic parameter of the human upper limb movement is first learned by the robot using the measurable interaction force, a recursive least square (RLS)-based estimator, and the Adam optimization method. Then, the desired trajectory of the robot can be obtained, tracking which the robot can supply the human’s upper limb with a prescribed interaction force. Using this controller, the robot automatically adjusts its reference trajectory to embrace the differences between different human users with diverse degrees of upper limb movement characteristics. By designing a performance index in the form of interaction force integral, potential adverse effects caused by the time-related uncertainty during the learning process can be addressed. The experimental results demonstrate the effectiveness of the proposed method in supplying the prescribed interaction force to the human user. Note to Practitioners—This article concentrates on developing a novel control technique to make the robot supply a prescribed interaction force to the human user in the presence of time-related uncertainties. The proposed control method is applicable to various scenarios of the human–robot interaction, e.g., it can be used for rehabilitation robots to provide assistive or resistive force to stroke patients or for exoskeleton robots to provide assistive force to human users for completing heavy-load tasks. Moreover, the desired interaction force can be tailored for different human users according to their needs and different task objectives. Consequently, the proposed controller can serve diverse users and has a promising perspective in automation.

The actuator failure compensation control problem of robotic systems possessing dynamic uncertainties has been investigated in this paper. Control design against partial loss of effectiveness (PLOE) and total loss of effectiveness (TLOE) of the actuator are considered and described, respectively, and a disturbance observer (DO) using neural networks is constructed to attenuate the influence of the unknown disturbance. Regarding the prescribed error bounds as time-varying constraints, the control design method based on barrier Lyapunov function (BLF) is used to strictly guarantee both the steady-state performance and the transient performance. A simulation study on a two-link planar manipulator verifies the effectiveness of the proposed controllers in dealing with the prescribed performance, the system uncertainties, and the unknown actuator failure simultaneously. Implementation on a Baxter robot gives an experimental verification of our controller.

In this article, a novel disturbance observer-based adaptive neural control (ANC) scheme is proposed for full-state-constrained pure-feedback nonlinear systems using a new system transformation method. A nonlinear transformation function in a uniformed design framework is constructed to convert the original states with constrained bounds into the ones without any constraints. By combining an auxiliary first-order filter, an augmented nonlinear system without any state constraint is derived to circumvent the difficulty of the controller design caused by the nonaffine input signal. Based on the augmented nonlinear system, a nonlinear disturbance observer (NDO) is designed to enhance the disturbance rejection ability. Subsequently, the NDO-based ANC scheme is presented by combining the second-order filters with backstepping. The proposed scheme confines all states within the predefined bounds, eliminates the condition on both the known sign and bounds of control gains, improves the robustness of the closed-loop system, and alleviates the computational burden. Two simulation examples are performed to show the validity of the presented scheme.

As the controller parameter adjustment process is simplified significantly by using learning algorithms, the studies about learning-based control attract a lot of interest in recent years. This article focuses on the intelligent servo control problem using learning from desired demonstrations. Compared with the previous studies about the learning-based servo control, a control policy using the broad learning system (BLS) is developed and first applied to a microrobotic system, since the advantages of the BLS, such as simple structure and no-requirement for retraining when new demos' data is provided. Then, the Lyapunov theory is skillfully combined with the complex learning algorithm to derive the controller parameters' constraints. Thus, the final control policy not only can obtain the movement skills of the desired demonstrations but also have the strong ability of generalization and error convergence. Finally, simulation and experimental examples verify the effectiveness of the proposed strategy using MATLAB and a microswimmer trajectory tracking system.

Different from previous work on single skill learning from human demonstrations, an incremental motor skill learning, generalization and control method based on dynamic movement primitives (DMP) and broad learning system (BLS) is proposed for extracting both ordinary skills and instant reactive skills from demonstrations, the latter of which is usually generated to avoid a sudden danger (e.g., touching a hot cup). The method is completed in three steps. First, the ordinary skills are basically learned from demonstrations in normal cases by using DMP. Then, the incremental learning idea of BLS is combined with DMP to achieve multistylistic reactive skill learning such that the forcing function of the ordinary skills will be reasonably extended into multiple stylistic functions by adding enhancement terms and updating weights of the radial basis function kernels. Finally, electromyography signals are collected from human muscles and processed to achieve stiffness factors. By using fuzzy logic system, the two kinds of skills learned are integrated and generalized in new cases such that not only start, end and scaling factors but also the environmental conditions, robot reactive strategies and impedance control factors will be generalized to lead to various reactions. To verify the effectiveness of the proposed method, an obstacle avoidance experiment that enables robots to approach destinations flexibly in various situations with barriers will be undertaken.

How to enable robotic compliant manipulation has become a critical problem in the robotics field. Inspired by a biomimetic adaptive control strategy, this article presents a novel representation model named human-like compliant movement primitives (Hl-CMPs) which could allow a robot to learn human-like compliant behaviors. The state-of-the-art approaches can hardly learn complete compliant profiles for a specific task. Comparatively, our model can encode task-specific parametric movement trajectories, correspondingly associated with dynamic trajectories including both impedance and feedforward force profiles. The compliant profiles are learned based on a biomimetic control strategy derived from the human motor learning in the muscle space, enabling the robot to simultaneously learn the impedance and the force while executing the movement trajectories obtained from human demonstration. Furthermore, both the kinematic and the dynamic profiles are learned in the parametric space, thus enabling the representation of a skill using corresponding parameters (i.e, task-specific parameters). Hl-CMps can allow the robot to automatically learn compliant behaviors in an online manner after kinematic demonstration. Our approach is validated by an insertion task and a cutting task based on a KUKA LBR iiwa robot.

This article presents an adaptive control method for dual-arm robot systems to perform bimanual tasks under modeling uncertainties. Different from the traditional symmetric bimanual robot control, we study the dual-arm robot control with relative motions between robotic arms and a grasped object. The robot system is first divided into two subsystems: a settled manipulator system and a tool-used manipulator system. Then, a command filtered control technique is developed for trajectory tracking and contact force control. In addition, to deal with the inevitable dynamic uncertainties, a radial basis function neural network (RBFNN) is employed for the robot, with a novel composite learning law to update the NN weights. The composite learning is mainly based on an integration of the historic data of NN regression such that information of the estimate error can be utilized to improve the convergence. Moreover, a partial persistent excitation condition is employed to ensure estimation convergence. The stability analysis is performed by using the Lyapunov theorem. Numerical simulation results demonstrate the validity of the proposed control and learning algorithm.

In this paper, an adaptive admittance control scheme is developed for robots to interact with time-varying environments. Admittance control is adopted to achieve a compliant physical robot-environment interaction, and the uncertain environment with time-varying dynamics is defined as a linear system. A critic learning method is used to obtain the desired admittance parameters based on the cost function composed of interaction force and trajectory tracking without the knowledge of the environmental dynamics. To deal with dynamic uncertainties in the control system, a neural-network (NN)-based adaptive controller with a dynamic learning framework is developed to guarantee the trajectory tracking performance. Experiments are conducted and the results have verified the effectiveness of the proposed method.

Hysteresis is a complex nonlinear effect in smart materials-based actuators, which degrades the positioning performance of the actuator, especially when the hysteresis shows asymmetric characteristics. In order to mitigate the asymmetric hysteresis effect, an adaptive neural digital dynamic surface control (DSC) scheme with the implicit inverse compensator is developed in this article. The implicit inverse compensator for the purpose of compensating for the hysteresis effect is applied to find the compensation signal by searching the optimal control laws from the hysteresis output, which avoids the construction of the inverse hysteresis model. The adaptive neural digital controller is achieved by using a discrete-time neural network controller to realize the discretization of time and quantizing the control signal to realize the discretization of the amplitude. The adaptive neural digital controller ensures the semiglobally uniformly ultimately bounded (SUUB) of all signals in the closed-loop control system. The effectiveness of the proposed approach is validated via the magnetostrictive-actuated system.

This article studies the formation tracking control problem for a group of underactuated surface vehicles with guaranteed transient properties, including connectivity maintenance, collision avoidance, and tracking performance specifications. The formation is within the leader–follower control framework, in which every follower is controlled to track its leader and maintain a desired relative distance and bearing angle with respect to its leader such that the prescribed formation geometry is achieved based on local sensing capability. The onboard sensor systems are of limited range and angle of view, thus defining a cone of detectable region for every follower. Each follower can detect its leader, if and only if the relative distance and bearing angle keep always inside the predefined detectable region such that the connectivity between the follower and its leader is maintained over time. In addition to the consideration of connectivity maintenance, no collision between the follower and its leader is also considered. A transverse function control approach is introduced to overcome the difficulties caused by the off-diagonal system matrix and underactuation. The barrier Lyapunov function and adaptive backstepping procedure are incorporated into the formation control design to achieve the boundedness of the closed-loop systems with guaranteed transient performance. Collision avoidance and connectivity maintenance between every follower and its leader are also proven mathematically. Simulation studies are performed to show the effectiveness of the proposed control design technique.

This article investigates human-driven robots via physical interaction, which is enhanced by integrating the human partner’s motion intention. A human motor control model is employed to estimate the human partner’s motion intention. A system observer is developed to estimate the human’s control input in this model, so that force sensing is not required. A robot controller is developed to incorporate the estimated human’s motion intention, which makes the robot proactively follow the human partner’s movements. Simulations and experiments on a physical robot are carried out to demonstrate the properties of our proposed controller.

The consumption of communication resources is an essential issue when control tasks are implemented in a wireless network environment. In order to lessen the network resources, a novel model-based (MB) adaptive event-triggered (ET) tracking control scheme is put forward in this article for strict-feedback discrete-time nonlinear systems. In this article, an event-based adaptive model is constructed by the combination of an$n$-step-ahead predictor and event-sampled neural networks. Then, the adaptive neural model is used for designing the MB ET controller. Besides, a modified ET condition is constructed without any delay. By combining a decoupled backstepping framework, the reverse Lyapunov stability technology is developed to verify the ultimate boundedness of all closed-loop signals and the convergence of the tracking error. Compared to the zero-order hold method, which keeps transmitted state signals unchanged in the interevent period, the proposed MB ET control scheme can keep the real-time update of state signals transmitted to the controller. It means that the triggering error will be smaller by the MB trigger mechanism, thereby improving the event-based tracking performance and further saving communication resources. Comparisons of simulation results are given to verify the effectiveness of the proposed control scheme.

This article aims at proposing an adaptive neural control strategy for a class of nonlinear time-delay systems with input delays and unknown control directions. Different from previous researches that investigated delays and constraints separately, the novelty of this article lies in that it simultaneously considers delays (state and input delays) and input constraints (magnitude and rate constraints) for a class of uncertain nonlinear systems. In this article, the uncertain states and input delays are handled by integrating a constructed auxiliary system that functions as an observer with neural networks (NNs), with which the adverse effects caused by the uncertain states and input delays can be approximated and compensated. By involving smooth hyperbolic tangent functions in the designed auxiliary system, the problem of magnitude and rate constraints of the control input is fully addressed. Then, the backstepping technique runs through the entire control designing process, which allows the designed adaptive neural control strategy to handle the input constraints and delays at the same time. Furthermore, Nussbaum functions are employed to resolve the problem of unknown control directions. Due to the introduction of an input-driven filter, only the output of the system is required to be measured as the control feedback, which promotes the applicability of the designed controller. Under the proposed control scheme, semiglobal, uniform, and ultimate boundedness of all signals of the closed-loop system is realized with uncertain control directions, input and state delays, and guaranteed magnitude and rate constraints of control inputs. Finally, simulation results are illustrated to verify the effectiveness of the presented control method.

Learning from demonstration (LfD) is a promising method for robots to learn and generalize human-like skills. It has the advantages of high programming efficiency, easy optimization, and non-professionals can also operate. There is a lot of research work that learn motion trajectories and stiffness curves from human demonstrations simutaneously to make the robot compliant, but previous work rarely consider the changes of environment. In this article, we propose an adaptive stiffness method that enables the robot to learn motion and stiffness trajectories from a single demonstration. When the environment changes, it can spontaneously tune the stiffness according to environmental feedback to ensure the smoothness of the task. Thus the robot has the ability to adapt to environmental changes. We first proved the theoretical feasibility of the method, and then we conducted physical experiments on the Baxter robot to verify the effectiveness of the proposed method.

In this work, we focus on improving the robot’s dexterous capability by exploiting visual sensing and adaptive force control. TeachNet, a vision-based teleoperation learning framework, is exploited to map human hand postures to a multi-fingered robot hand. We augment TeachNet, which is originally based on an imprecise kinematic mapping and position-only servoing, with a biomimetic learning-based compliance control algorithm for dexterous manipulation tasks. This compliance controller takes the mapped robotic joint angles from TeachNet as the desired goal, computes the desired joint torques. It is derived from a computational model of the biomimetic control strategy in human motor learning, which allows adapting the control variables (impedance and feedforward force) online during the execution of the reference joint angle trajectories. The simultaneous adaptation of the impedance and feedforward profiles enables the robot to interact with the environment in a compliant manner. Our approach has been verified in multiple tasks in physics simulation, i.e., grasping, opening-a-door, turning-a-cap, and touching-a-mouse, and has shown more reliable performances than the existing position control and the fixed-gain-based force control approaches.

This article proposes a novel control strategy based on a broad fuzzy neural network (BFNN) which is subjected to contact with the unknown environment. Compared with the conventional fuzzy neural network (NN), a prominent feature can be achieved by taking the advantage of the broad learning system (BLS) to explicitly tackle the problem of how to choose a sufficient number of NN units to approximate the unknown dynamic model. Aiming at providing a soft compliant contact scheme without the requirement of the environment model, an adaptive impedance learning is developed to establish the optimal interaction between the robot and the environment. Meanwhile, the problems related to the state constraints are addressed by incorporating a barrier Lyapunov function (BLF) into the design of a trajectory tracking controller. The proposed method can achieve desired tracking and interaction performance while guaranteeing the stability of the closed-loop system. In addition, simulation and experimental studies are performed to verify the effectiveness of BFNN under optimal impedance control with a two degree-of-freedom (DOF) manipulator and a Baxter robot, respectively.

This article presents a state-of-the-art survey on the robotic systems, sensors, actuators, and collaborative strategies for physical human-robot collaboration (pHRC). This article starts with an overview of some robotic systems with cutting-edge technologies (sensors and actuators) suitable for pHRC operations and the intelligent assist devices employed in pHRC. Sensors being among the essential components to establish communication between a human and a robotic system are surveyed. The sensor supplies the signal needed to drive the robotic actuators. The survey reveals that the design of new generation collaborative robots and other intelligent robotic systems has paved the way for sophisticated learning techniques and control algorithms to be deployed in pHRC. Furthermore, it revealed the relevant components needed to be considered for effective pHRC to be accomplished. Finally, a discussion of the major advances is made, some research directions, and future challenges are presented.

This article proposes a Bayesian method to acquire the estimation of human impedance and motion intention in a human-robot collaborative task. Combining with the prior knowledge of human stiffness, estimated stiffness obeying Gaussian distribution is obtained by Bayesian estimation, and human motion intention can be also estimated. An adaptive impedance control strategy is employed to track a target impedance model and neural networks are used to compensate for uncertainties in robotic dynamics. Comparative simulation results are carried out to verify the effectiveness of estimation method and emphasize the advantages of the proposed control strategy. The experiment, performed on Baxter robot platform, illustrates a good system performance.

Lots of evidence has indicated that many kinds of animals can achieve goal-oriented navigation by spatial cognition and dead reckoning. The geomagnetic field (GF) is a ubiquitous cue for navigation by these animals. Inspired by the goal-oriented navigation of animals, a novel long-distance underwater geomagnetic navigation (LDUGN) method is presented in this article, which only utilizes the declination component ( D ) and inclination component ( I ) of GF for underwater navigation without any prior knowledge of the geographical location or geomagnetic map. The D and I measured by high-precision geomagnetic sensors are compared periodically with that of the destination to determine the velocity and direction in the next step. A model predictive control (MPC) algorithm with control and state constraints is proposed to achieve the control and optimization of navigation trajectory. Because the optimal control is recalculated at each sampling instant, the MPC algorithm can overcome interferences of geomagnetic daily fluctuation, geomagnetic storms, ocean current, and geomagnetic local anomaly. The simulation results validate the feasibility and accuracy of the proposed algorithm.

This article presents a control scheme for robot manipulators with the consideration of output error constraints, unknown dynamics, and bounded disturbances. A modified virtual input variable in the second stage design of the dynamic surface control scheme is proposed, which can enhance the robustness of the controller. Bounded disturbances due to the situations that the base is not well fixed if the robot manipulator is mounted at a mobile platform are considered and suppressed. Besides, the detailed implementation process of the composite learning laws adopted for enhancing the radial basis function neural network is presented. Lyapunov stability analysis verifies that the proposed control scheme ensures the trajectory tracking errors stay within predefined boundaries and parameter estimate errors converge without a stringent condition termed persistent excitation. Experimental results show the superiority of the proposed controller regarding parameter estimation and tracking capabilities.

Transferring human stiffness regulation strategies to robots enables them to effectively and efficiently acquire adaptive impedance control policies to deal with uncertainties during the accomplishment of physical contact tasks in an unstructured environment. In this article, we develop such a physical human-robot interaction system which allows robots to learn variable impedance skills from human demonstrations. Specifically, the biological signals, i.e., surface electromyography are utilized for the extraction of human arm stiffness features during the task demonstration. The estimated human arm stiffness is then mapped into a robot impedance controller. The dynamics of both movement and stiffness are simultaneously modeled by using a model combining the hidden semi-Markov model and the Gaussian mixture regression. More importantly, the correlation between the movement information and the stiffness information is encoded in a systematic manner. This approach enables capturing uncertainties over time and space and allows the robot to satisfy both position and stiffness requirements in a task with modulation of the impedance controller. The experimental study validated the proposed approach.

We aim at the optimization of the tracking control of a robot to improve the robustness, under the effect of unknown nonlinear perturbations. First, an auxiliary system is introduced, and optimal control of the auxiliary system can be seen as an approximate optimal control of the robot. Then, neural networks (NNs) are employed to approximate the solution of the Hamilton-Jacobi-Isaacs equation under the frame of adaptive dynamic programming. Next, based on the standard gradient attenuation algorithm and adaptive critic design, NNs are trained depending on the designed updating law with relaxing the requirement of initial stabilizing control. In light of the Lyapunov stability theory, all the error signals can be proved to be uniformly ultimately bounded. A series of simulation studies are carried out to show the effectiveness of the proposed control.

This article presents a control scheme for the robot manipulator's trajectory tracking task considering output error constraints and control input saturation. We provide an alternative way to remove the feasibility condition that most BLF-based controllers should meet and design a control scheme on the premise that constraint violation possibly happens due to the control input saturation. A bounded barrier Lyapunov function is proposed and adopted to handle the output error constraints. Besides, to suppress the input saturation effect, an auxiliary system is designed and emerged into the control scheme. Moreover, a simplified RBFNN structure is adopted to approximate the lumped uncertainties. Simulation and experimental results demonstrate the effectiveness of the proposed control scheme.

This article discusses the control design and experiment validation of a flexible two-link manipulator (FTLM) system represented by ordinary differential equations (ODEs). A reinforcement learning (RL) control strategy is developed that is based on actor–critic structure to enable vibration suppression while retaining trajectory tracking. Subsequently, the closed-loop system with the proposed RL control algorithm is proved to be semi-global uniform ultimate bounded (SGUUB) by Lyapunov’s direct method. In the simulations, the control approach presented has been tested on the discretized ODE dynamic model and the analytical claims have been justified under the existence of uncertainty. Eventually, a series of experiments in a Quanser laboratory platform are investigated to demonstrate the effectiveness of the presented control and its application effect is compared with PD control.

In this paper, an adaptive neural bounded control scheme is proposed for an ${n}$ -link rigid robotic manipulator with unknown dynamics. With the combination of the neural approximation and backstepping technique, an adaptive neural network control policy is developed to guarantee the tracking performance of the robot. Different from the existing results, the bounds of the designed controller are known a priori, and they are determined by controller gains, making them applicable within actuator limitations. Furthermore, the designed controller is also able to compensate the effect of unknown robotic dynamics. Via the Lyapunov stability theory, it can be proved that all the signals are uniformly ultimately bounded. Simulations are carried out to verify the effectiveness of the proposed scheme.

Most existing adaptive control designs for nonlinear pure-feedback systems have been derived based on backstepping or dynamic surface control (DSC) methods, requiring full system states to be measurable. The neural networks (NNs) or fuzzy logic systems (FLSs) used to accommodate uncertainties also impose demanding computational cost and sluggish convergence. To address these issues, this paper proposes a new output-feedback control for uncertain pure-feedback systems without using backstepping and function approximator. A coordinate transform is first used to represent the pure-feedback system in a canonical form to evade using the backstepping or DSC scheme. Then the Levant's differentiator is used to reconstruct the unknown states of the derived canonical system. Finally, a new unknown system dynamics estimator with only one tuning parameter is developed to compensate for the lumped unknown dynamics in the feedback control. This leads to an alternative, simple approximation-free control method for pure-feedback systems, where only the system output needs to be measured. The stability of the closed-loop control system, including the unknown dynamics estimator and the feedback control is proved. Comparative simulations and experiments based on a PMSM test-rig are carried out to test and validate the effectiveness of the proposed method.

In this article, a novel neural network (NN)-based adaptive event-triggered control scheme is developed for a class of uncertain discrete-time strict-feedback nonlinear systems with asymmetric actuator saturation. To deal with the asymmetric input saturation, a unified smooth nonlinear function is constructed to effectively characterize the limitations between the control signal and the actuator. Subsequently, the novel backstepping design process, instead of the traditional$n$-step-ahead predictor, is developed to design the stable event-triggered adaptive tracking controller by combining one neural approximator. Especially, a modified event-triggering condition is equipped into the designed controller to increase the number of triggering events at the transient-state stage. The proposed control scheme can not only achieve the good tracking performance with the avoidance of the$n$-step time delays and the improvement of the transient-state performance but also alleviate the transmission burden of the network resource, and eliminate the effect of the asymmetric actuator saturation. Numerical simulation results demonstrate the effectiveness of the control scheme proposed in this article.

Nonlinear optimization problems with dynamical parameters are widely arising in many practical scientific and engineering applications, and various computational models are presented for solving them under the hypothesis of short-time invariance. To eliminate the large lagging error in the solution of the inherently dynamic nonlinear optimization problem, the only way is to estimate the future unknown information by using the present and previous data during the solving process, which is termed the future dynamic nonlinear optimization (FDNO) problem. In this paper, to suppress noises and improve the accuracy in solving FDNO problems, a novel noise-tolerant neural (NTN) algorithm based on zeroing neural dynamics is proposed and investigated. In addition, for reducing algorithm complexity, the quasi-Newton Broyden-Fletcher-Goldfarb-Shanno (BFGS) method is employed to eliminate the intensively computational burden for matrix inversion, termed NTN-BFGS algorithm. Moreover, theoretical analyses are conducted, which show that the proposed algorithms are able to globally converge to a tiny error bound with or without the pollution of noises. Finally, numerical experiments are conducted to validate the superiority of the proposed NTN and NTN-BFGS algorithms for the online solution of FDNO problems.

In this paper, a force sensorless control scheme based on neural networks (NNs) is developed for interaction between robot manipulators and human arms in physical collision. In this scheme, the trajectory is generated by using geometry vector method with Kinect sensor. To comply with the external torque from the environment, this paper presents a sensorless admittance control approach in joint space based on an observer approach, which is used to estimate external torques applied by the operator. To deal with the tracking problem of the uncertain manipulator, an adaptive controller combined with the radial basis function NN (RBFNN) is designed. The RBFNN is used to compensate for uncertainties in the system. In order to achieve the prescribed tracking precision, an error transformation algorithm is integrated into the controller. The Lyapunov functions are used to analyze the stability of the control system. The experiments on the Baxter robot are carried out to demonstrate the effectiveness and correctness of the proposed control scheme.

Picking up the clustered objects is always a challenging task in robot research field. And reinforcement learning enables robot to adapt to different tasks through plenty of attempts. To reduce the complexity of strategy learning, we propose a framework for robots to pick up the objects in clutter on table based on deep reinforcement learning and rule-based method. To manipulate the objects on table, we mainly divide the robot actions into two categories: one is pushing that uses the reinforcement learning method, while the other one is grasping that is inferred by image morphological processing. The pushing action can separate the stacking objects, create a robust grasp point for the following grasp. The grasp detect algorithm determines if there is a suitable grasp point. Judging on the result of pushing, the grasp detect algorithm will return a reward for pushing learning. Taking images as input, our framework can keep a high grasp rate with low computational complexity, which makes it achieve clutter clearing quickly.

It is necessary to implement real-time grasp detection in robotic grasping tasks. To this end, in this paper we propose a method for effective graspable feature selection and precise classification. In a robotic grasping scene, our method can effectively select graspable rectangles and further extract useful features from them to generate a feature set. A convolutional neural network (CNN) is then developed to score and classify the elements in the feature set. Finally, we compute the desired robotic grasp pose based on the graspable feature that gets the highest score. In the test phase the proposed CNN network achieves an accuracy of 96.5% on the Cornell Grasping Dataset. In real-world grasping experiments 105 frames per second (fps) for the object's grasp detection and a grasp success rate of 89.9% have been achieved with our method.

In human-robot collaborative transportation and sawing tasks, the human operator physically interacts with the robot and directs the robot's movement by applying an interaction force. The robot needs to update its control strategy to adapt to the interaction with the human and to minimize the interaction force. To this end, we propose an integrated algorithm of robot's trajectory adaptation and adaptive impedance control to minimize the interaction force in physical humanrobot interaction (pHRI) and to guarantee the performance of the collaboration tasks. We firstly utilize the information of the interaction force to regulate the robot's reference trajectory. Then, an adaptive impedance controller is developed to ensure automatic adaptation of the robot's impedance parameters. While one can reduce the interaction force by using either trajectory adaptation or adaptive impedance control, we investigate the task performance when combining both. Experimental results on a planar robotic platform verify the effectiveness of the proposed method.

Abstract Real-time grasp detection plays a key role in manipulation, and it is also a complex task, especially for detecting how to grasp novel objects. This paper proposes a very quick and accurate approach to detect robotic grasps. The main idea is to perform grasping of novel objects in a typical RGB-D scene view. Our goal is not to find the best grasp for every object but to obtain the local optimal grasps in candidate grasp rectangles. There are three main contributions to our detection work. Firstly, an improved graph segmentation approach is used to do objects detection and it can separate objects from the background directly and fast. Secondly, we develop a morphological image processing method to generate candidate grasp rectangles set which avoids us to search grasp rectangles globally. Finally, we train a random forest model to predict grasps and achieve an accuracy of 94.26%. The model is mainly used to score every element in our candidate grasps set and the one gets the highest score will be converted to the final grasp configuration for robots. For real-world experiments, we set up our system on a tabletop scene with multiple objects and when implementing robotic grasps, we control Baxter robot with a different inverse kinematics strategy rather than the built-in one.

In this article, an admittance-based controller for physical human-robot interaction (pHRI) is presented to perform the coordinated operation in the constrained task space. An admittance model and a soft saturation function are employed to generate a differentiable reference trajectory to ensure that the end-effector motion of the manipulator complies with the human operation and avoids collision with surroundings. Then, an adaptive neural network (NN) controller involving integral barrier Lyapunov function (IBLF) is designed to deal with tracking issues. Meanwhile, the controller can guarantee the end-effector of the manipulator limited in the constrained task space. A learning method based on the radial basis function NN (RBFNN) is involved in controller design to compensate for the dynamic uncertainties and improve tracking performance. The IBLF method is provided to prevent violations of the constrained task space. We prove that all states of the closed-loop system are semiglobally uniformly ultimately bounded (SGUUB) by utilizing the Lyapunov stability principles. At last, the effectiveness of the proposed algorithm is verified on a Baxter robot experiment platform. Note to Practitioners-This work is motivated by the neglect of safety in existing controller design in physical human-robot interaction (pHRI), which exists in industry and services, such as assembly and medical care. It is considerably required in the controller design for rigorously handling constraints. Therefore, in this article, we propose a novel admittance-based human-robot interaction controller. The developed controller has the following functionalities: 1) ensuring reference trajectory remaining in the constrained task space: a differentiable reference trajectory is shaped by the desired admittance model and a soft saturation function; 2) solving uncertainties of robotic dynamics: a learning approach based on radial basis function neural network (RBFNN) is involved in controller design; and 3) ensuring the end-effector of the manipulator remaining in the constrained task space: different from other barrier Lyapunov function (BLF), integral BLF (IBLF) is proposed to constrain system output directly rather than tracking error, which may be more convenient for controller designers. The controller can be potentially applied in many areas. First, it can be used in the rehabilitation robot to avoid injuring the patient by limiting the motion. Second, it can ensure the end-effector of the industrial manipulator in a prescribed task region. In some industrial tasks, dangerous or damageable tools are mounted on the end-effector, and it will hurt humans and bring damage to the robot when the end-effector is out of the prescribed task region. Third, it may bring a new idea to the designed controller for avoiding collisions in pHRI when collisions occur in the prescribed trajectory of end-effector.

In this paper, the complex problems of internal forces and position control are studied simultaneously and a disturbance observer-based radial basis function neural network (RBFNN) control scheme is proposed to: 1) estimate the unknown parameters accurately; 2) approximate the disturbance experienced by the system due to input saturation; and 3) simultaneously improve the robustness of the system. More specifically, the proposed scheme utilizes disturbance observers, neural network (NN) collaborative control with an adaptive law, and full state feedback. Utilizing Lyapunov stability principles, it is shown that semiglobally uniformly bounded stability is guaranteed for all controlled signals of the closed-loop system. The effectiveness of the proposed controller as predicted by the theoretical analysis is verified by comparative experimental studies.

This paper proposed an adaptive neural admittance control strategy for collision avoidance in human-robot collaborative tasks. In order to ensure that the robot end-effector can avoid collisions with surroundings, robot should be operated compliantly by human within a constrained task space. An impedance model and a soft saturation function are employed to generate a differentiable reference trajectory. Then, adaptive neural network control with position constraint, based on integral barrier Lyapunov function (IBLF), is designed to achieve precise tracking while guaranteeing constrained satisfaction. Utilizing Lyapunov stability principles, we prove that semi-globally uniformly bounded stability is guaranteed for all states of the closed-loop system. At last, the effectiveness of the proposed algorithm is verified on a Baxter robot experimental platform. Collisions with surroundings can be avoided in human-robot collaborative tasks.

Electromyogram (EMG) signal has been long used in human-robot interface in literature, especially in the area of rehabilitation. Recent rapid development in artificial intelligence (AI) has provided powerful machine learning tools to better explore the rich information embedded in EMG signals. For our specific application task in this work, i.e. estimate human finger force based on EMG signal, a LightGBM (Gradient Boosting Machine) model has been used. The main contribution of this study is the development of an objective and automatic optimal feature selection algorithm that can minimize the number of features used in the LightGBM model in order to simplify implementation complexity, reduce computation burden and maintain comparable estimation performance to the one with full features. The performance of the LightGBM model with selected optimal features is compared with 4 other popular machine learning models based on a dataset including 45 subjects in order to show the effectiveness of the developed feature selection method.

Due to the lack of transparent and friendly human-robot interaction (HRI) interface, as well as various uncertainties, it is usually a challenge to remotely manipulate a robot to accomplish a complicated task. To improve the teleoperation performance, we propose a new perception mechanism by integrating a novel learning method to operate the robots in the distance. In order to enhance the perception of the teleoperation system, we utilize a surface electromyogram signal to extract the human operator's muscle activation. As a response to the changes in the external environment, as sensed through haptic and visual feedback, a human operator naturally reacts with various muscle activations. By imitating the human behaviors in task execution, not only motion trajectory but also arm stiffness adjusted by muscle activation, it is expected that the robot would be able to carry out the repetitive tasks autonomously or uncertain tasks with improved intelligence. To this end, we develop a robot learning algorithm based on probability statistics under an integrated framework of the hidden semi-Markov model (HSMM) and the Gaussian mixture method. This method is employed to obtain a generative task model based on the robot's trajectory. Then, Gaussian mixture regression based on HSMM is applied to correct the robot trajectory with the reproduced results from the learned task model. The execution procedures consist of a learning phase and a reproduction phase. To guarantee the stability, immersion, and maneuverability of the teleoperation system, a variable gain control method that involves electromyography (EMG) is introduced. Experimental results have demonstrated the effectiveness of the proposed method.

In this paper, we investigate fuzzy neural network (FNN) control using impedance learning for coordinated multiple constrained robots carrying a common object in the presence of the unknown robotic dynamics and the unknown environment with which the robot comes into contact. First, an FNN learning algorithm is developed to identify the unknown plant model. Second, impedance learning is introduced to regulate the control input in order to improve the environment-robot interaction, and the robot can track the desired trajectory generated by impedance learning. Third, in light of the condition requiring the robot to move in a finite space or to move at a limited velocity in a finite space, the algorithm based on the position constraint and the velocity constraint are proposed, respectively. To guarantee the position constraint and the velocity constraint, an integral barrier Lyapunov function is introduced to avoid the violation of the constraint. According to Lyapunov's stability theory, it can be proved that the tracking errors are uniformly bounded ultimately. At last, some simulation examples are carried out to verify the effectiveness of the designed control.

In this paper, an admittance adaptation method has been developed for robots to interact with unknown environments. The environment to be interacted with is modeled as a linear system. In the presence of the unknown dynamics of environments, an observer in robot joint space is employed to estimate the interaction torque, and admittance control is adopted to regulate the robot behavior at interaction points. An adaptive neural controller using the radial basis function is employed to guarantee trajectory tracking. A cost function that defines the interaction performance of torque regulation and trajectory tracking is minimized by admittance adaptation. To verify the proposed method, simulation studies on a robot manipulator are conducted.

Due to strongly coupled nonlinearities of the grasped dual-arm robot and the internal forces generated by grasped objects, the dual-arm robot control with uncertain kinematics and dynamics raises a challenging problem. In this paper, an adaptive fuzzy control scheme is developed for a dual-arm robot, where an approximate Jacobian matrix is applied to address the uncertain kinematic control, while a decentralized fuzzy logic controller is constructed to compensate for uncertain dynamics of the robotic arms and the manipulated object. Also, a novel finite-time convergence parameter adaptation technique is developed for the estimation of kinematic parameters and fuzzy logic weights, such that the estimation can be guaranteed to converge to small neighborhoods around their ideal values in a finite time. Moreover, a partial persistent excitation property of the Gaussian-membership-based fuzzy basis function was established to relax the conventional persistent excitation condition. This enables a designer to reuse these learned weight values in the future without relearning. Extensive simulation studies have been carried out using a dual-arm robot to illustrate the effectiveness of the proposed approach.

Robots are often required to generalize the skills learned from human demonstrations to fulfil new task requirements. However, skill generalization will be difficult to realize when facing with the following situations: the skill for a complex multistep task includes a number of features; some special constraints are imposed on the robots during the process of task reproduction; and a completely new situation quite different with the one in which demonstrations are given to the robot. This work proposes a new framework to facilitate robot skill generalization. The basic idea lies in that the learned skills are first segmented into a sequence of subskills automatically, then each individual subskill is encoded and regulated accordingly. Specifically, we adapt each set of the segmented movement trajectories individually instead of the whole movement profiles, thus, making it more convenient for the realization of skill generalization. In addition, human limb stiffness estimated from surface electromyographic signals is considered in the framework for the realization of human-to-robot variable impedance control skill transfer, as well as the generalization of both movement trajectories and stiffness profiles. Experimental study has been performed to verify the effectiveness of the proposed framework.

This paper proposes a novel adaptive control methodology based on the admittance model for multiple manipulators transporting a rigid object cooperatively along a predefined desired trajectory. First, an admittance model is creatively applied to generate reference trajectory online for each manipulator according to the desired path of the rigid object, which is the reference input of the controller. Then, an innovative integral barrier Lyapunov function is utilized to tackle the constraints due to the physical and environmental limits. Adaptive neural networks (NNs) are also employed to approximate the uncertainties of the manipulator dynamics. Different from the conventional NN approximation method, which is usually semiglobally uniformly ultimately bounded, a switching function is presented to guarantee the global stability of the closed loop. Finally, the simulation studies are conducted on planar two-link robot manipulators to validate the efficacy of the proposed approach.

This paper proposes an enhanced robot skill learning system considering both motion generation and trajectory tracking. During robot learning demonstrations, dynamic movement primitives (DMPs) are used to model robotic motion. Each DMP consists of a set of dynamic systems that enhances the stability of the generated motion toward the goal. A Gaussian mixture model and Gaussian mixture regression are integrated to improve the learning performance of the DMP, such that more features of the skill can be extracted from multiple demonstrations. The motion generated from the learned model can be scaled in space and time. Besides, a neural-network-based controller is designed for the robot to track the trajectories generated from the motion model. In this controller, a radial basis function neural network is used to compensate for the effect caused by the dynamic environments. The experiments have been performed using a Baxter robot and the results have confirmed the validity of the proposed methods.

In this paper, an adaptive fuzzy neural network (FNN) control scheme is developed for a class of multipleinput and multiple-output (MIMO) nonlinear systems subject to unknown dynamics and state constraints. FNNs are used to approximate the unknown dynamics that comprises the effects of uncertain parameters and functions. Also, integral Lyapunov functions are introduced to address state constraints. A neuralnetwork-based observer is designed to estimate the unmeasurable states. With state-feedback and output feedback tracking control, the stability of closed-loop system is guaranteed via Lyapunov's stability theory. Two cases of simulations for MIMO systems with state constraints are conducted to verify the effectiveness of the proposed control.

It has been established that the transfer of human adaptive impedance is of great significance for physical human-robot interaction (pHRI). By processing the electromyography (EMG) signals collected from human muscles, the limb impedance could be extracted and transferred to robots. The existing impedance transfer interfaces rely only on visual feedback and, thus, may be insufficient for skill transfer in a sophisticated environment. In this paper, physical haptic feedback mechanism is introduced to result in muscle activity that would generate EMG signals in a natural manner, in order to achieve intuitive human impedance transfer through a designed coupling interface. Relevant processing methods are integrated into the system, including the spectral collaborative representation-based classifications method used for hand motion recognition; fast smooth envelop and dimensionality reduction algorithm for arm endpoint stiffness estimation. The tutor's arm endpoint motion trajectory is directly transferred to the robot by the designed coupling module without the restriction of hands. Haptic feedback is provided to the human tutor according to skill learning performance to enhance the teaching experience. The interface has been experimentally tested by a plugging-in task and a cutting task. Compared with the existing interfaces, the developed one has shown a better performance.

Sliding mode control (SMC) is attractive for nonlinear systems due to its invariance for both parametric and nonparametric uncertainties. However, the invariance of SMC is not guaranteed in a reaching phase. Integral SMC (ISMC) eliminates the reaching phase such that the invariance is achieved in an entire system response. To reduce chattering in ISMC, it was suggested that the switching element is smoothed by using a low-pass filter and an integral sliding variable is modified. This study discusses several crucial problems regarding the performance, modification, and improvement of ISMC. First, the modification of the integral sliding variable is revealed to be unnecessary as it degrades the performance of a sliding phase; second, ISMC is shown to be a kind of global SMC; third, it is manifested that a high-order ISMC design with super twisting involves a stability condition that may be infeasible in theory; finally, an efficient solution is suggested to attenuate chattering in ISMC without the degradation of tracking accuracy and the solution is extended to the case with uncertain control gain functions. Comprehensive simulation results have verified the arguments of this study.

This paper reports the development of an intelligent shared control system for a robotic manipulator that is commanded by the user's mind. The target objects are detected by a vision system and then displayed to the user in a video that shows them fused with flicking diamonds that are designed to excite electroencephalograph (EEG) signals at different frequency bands. Through the analysis of the invoked EEG signals, a brain-computer interface is developed to infer the exact object that is required by the user. These results are then transferred to the shared control system, which is enabled by visual servoing techniques to achieve accurate object manipulation. The task motion and self-motion (CTS) methods are coordinated to enhance the intelligence of the shared control system by equipping the robot with an autonomous obstacle avoidance function. Extensive experimental studies are performed to verify that the adaptive object tracking algorithm, the CTS method, and the least-squares method are helpful in improving the performance of the intelligent robotic system.

In this paper, two upper limbs of an exoskeleton robot are operated within a constrained region of the operational space with unidentified intention of the human operator's motion as well as uncertain dynamics including physical limits. The new human-cooperative strategies are developed to detect the human subject's movement efforts in order to make the robot behavior flexible and adaptive. The motion intention extracted from the measurement of the subject's muscular effort in terms of the applied forces/torques can be represented to derive the reference trajectory of his/her limb using a viable impedance model. Then, adaptive online estimation for impedance parameters is employed to deal with the nonlinear and variable stiffness property of the limb model. In order for the robot to follow a specific impedance target, we integrate the motion intention estimation into a barrier Lyapunov function based adaptive impedance control. Experiments have been carried out to verify the effectiveness of the proposed dual-arm coordination control scheme, in terms of desired motion and force tracking.

Teleoperated robot systems are able to support humans to accomplish their tasks in many applications. However, the performance of teleoperation largely depends on motor functionality and human operator's skill, especially when a human operator is short of skill training. In order to adapt to various unstructured environments for the robot system and the human operator, in this paper, a teleoperation scheme using integrated tremor attenuation with a variable gain control algorithm involving surface electromyogram is proposed to achieve personalized control performance and to reduce reliance on operator's skill. For attenuating tremor, a filter based on support vector machine is developed to guarantee normal operation. This filter depends on the machine learning scheme and does not rely on a priori filter parameters. Semiphysical experiments have been performed to demonstrate the effectiveness of the proposed methods.

Human motion recognition is a trending topic and could be applied in many areas, the motion estimation of ASD children is more challenging because of the high uncertainty of their activities, we thus introduced a novel method which is designed for estimating the upper joints and recognising their special motions, we verified the proposed method on our recorded ASD children dataset and adult dataset, the experimental results show the proposed method is effective on the dataset.

In this paper, a mind controlled multi-task manipulator based on motor imagery electroencephalogram (EEG) is proposed. Describe the system function first: In the case of only two types of control signal, the implementation of multi-task Manipulator relies on a toggle-confirmation mode of operation: the task is switched when imagining the left-hand movement, and the task is confirmed when the right-hand movement is imagined. In the BCI system, common spatial pattern (CSP) is used for feature extraction, mutual information for feature selection, and linear discriminant analysis (LDA) for pattern classification. The EEG signal is processed and classified into two categories, imagery of left-hand and right-hand movement. In this way, we can achieve the multi-task control of the manipulator under the premise of ensuring the accuracy of EEG recognition.

Obstacle detection plays an important role for robot collision avoidance and motion planning. This paper focuses on the study of the collision prediction of a dual-arm robot based on a 3D point cloud. Firstly, a self-identification method is presented based on the over-segmentation approach and the forward kinematic model of the robot. Secondly, a simplified 3D model of the robot is generated using the segmented point cloud. Finally, a collision prediction algorithm is proposed to estimate the collision parameters in real-time. Experimental studies using the Kinect Ⓡ sensor and the Baxter Ⓡ robot have been performed to demonstrate the performance of the proposed algorithms.

In this paper, a neural networks (NNs) enhanced telerobot control system is designed and tested on a Baxter robot. Guaranteed performance of the telerobot control system is achieved at both kinematic and dynamic levels. At kinematic level, automatic collision avoidance is achieved by the control design at the kinematic level exploiting the joint space redundancy, thus the human operator would be able to only concentrate on motion of robot's end-effector without concern on possible collision. A posture restoration scheme is also integrated based on a simulated parallel system to enable the manipulator restore back to the natural posture in the absence of obstacles. At dynamic level, adaptive control using radial basis function NNs is developed to compensate for the effect caused by the internal and external uncertainties, e.g., unknown payload. Both the steady state and the transient performance are guaranteed to satisfy a prescribed performance requirement. Comparative experiments have been performed to test the effectiveness and to demonstrate the guaranteed performance of the proposed methods.

This paper presents a teleoperation control for an exoskeleton robotic system based on the brain-machine interface and vision feedback. Vision compressive sensing, brain-machine reference commands, and adaptive fuzzy controllers in joint-space have been effectively integrated to enable the robot performing manipulation tasks guided by human operator's mind. First, a visual-feedback link is implemented by a video captured by a camera, allowing him/her to visualize the manipulator's workspace and movements being executed. Then, the compressed images are used as feedback errors in a nonvector space for producing steady-state visual evoked potentials electroencephalography (EEG) signals, and it requires no prior information on features in contrast to the traditional visual servoing. The proposed EEG decoding algorithm generates control signals for the exoskeleton robot using features extracted from neural activity. Considering coupled dynamics and actuator input constraints during the robot manipulation, a local adaptive fuzzy controller has been designed to drive the exoskeleton tracking the intended trajectories in human operator's mind and to provide a convenient way of dynamics compensation with minimal knowledge of the dynamics parameters of the exoskeleton robot. Extensive experiment studies employing three subjects have been performed to verify the validity of the proposed method.

Robots with coordinated dual arms are able to perform more complicated tasks that a single manipulator could hardly achieve. However, more rigorous motion precision is required to guarantee effective cooperation between the dual arms, especially when they grasp a common object. In this case, the internal forces applied on the object must also be considered in addition to the external forces. Therefore, a prescribed tracking performance at both transient and steady states is first specified, and then, a controller is synthesized to rigorously guarantee the specified motion performance. In the presence of unknown dynamics of both the robot arms and the manipulated object, the neural network approximation technique is employed to compensate for uncertainties. In order to extend the semiglobal stability achieved by conventional neural control to global stability, a switching mechanism is integrated into the control design. Effectiveness of the proposed control design has been shown through experiments carried out on the Baxter Robot.

In this paper, we investigate the trajectory tracking problem for a fully actuated autonomous underwater vehicle (AUV) that moves in the horizontal plane. External disturbances, control input nonlinearities and model uncertainties are considered in our control design. Based on the dynamics model derived in the discrete-time domain, two neural networks (NNs), including a critic and an action NN, are integrated into our adaptive control design. The critic NN is introduced to evaluate the long-time performance of the designed control in the current time step, and the action NN is used to compensate for the unknown dynamics. To eliminate the AUV's control input nonlinearities, a compensation item is also designed in the adaptive control. Rigorous theoretical analysis is performed to prove the stability and performance of the proposed control law. Moreover, the robustness and effectiveness of the proposed control method are tested and validated through extensive numerical simulation results.

In this paper, model identification and adaptive control design are performed on Devanit-Hartenberg model of a humanoid robot. We focus on the modeling of the 6 degree-of-freedom upper limb of the robot using recursive Newton-Euler (RNE) formula for the coordinate frame of each joint. To obtain sufficient excitation for modeling of the robot, the particle swarm optimization method has been employed to optimize the trajectory of each joint, such that satisfied parameter estimation can be obtained. In addition, the estimated inertia parameters are taken as the initial values for the RNE-based adaptive control design to achieve improved tracking performance. Simulation studies have been carried out to verify the result of the identification algorithm and to illustrate the effectiveness of the control design.

In this paper, a sensory-motor fusion-based manipulation and grasping control strategy has been developed for a robotic hand-eye system. The proposed hierarchical control architecture has three modules: 1) vision servoing; 2) surface electromyography (sEMG)-based movement recognition; and 3) hybrid force and motion optimization for manipulation and grasping. A stereo camera is used to obtain the 3-D point cloud of a target object and provides the desired operational position. The AdaBoost-based motion recognition is employed to discriminate different movements based on sEMG of human upper limbs. The operational space motion planning for bionic arm and force planning for multifingered robotic hand can be both transformed as a convex optimization problem with various constraints. A neural dynamics optimization solution is proposed and implemented online. The proposed formulation can achieve a substantial reduction of computational load. The actual implementation includes a bionic arm with dextrous hand, high-speed active vision, and an EMG sensors. A series of manipulation tasks consisting of tracking/recogniting/grasping of an object are implemented, and experiment results exhibit the responsiveness and flexibility of the proposed sensory motion fusion approach.

This paper presents an extreme learning machine (ELM)-based control scheme for uncertain robot manipulators to perform haptic identification. ELM is used to compensate for the unknown nonlinearity in the manipulator dynamics. The ELM enhanced controller ensures that the closed-loop controlled manipulator follows a specified reference model, in which the reference point as well as the feedforward force is adjusted after each trial for haptic identification of geometry and stiffness of an unknown object. A neural learning law is designed to ensure finite-time convergence of the neural weight learning, such that exact matching with the reference model can be achieved after the initial iteration. The usefulness of the proposed method is tested and demonstrated by extensive simulation studies.

In this paper, a novel control scheme is developed for a teleoperation system, combining the radial basis function (RBF) neural networks (NNs) and wave variable technique to simultaneously compensate for the effects caused by communication delays and dynamics uncertainties. The teleoperation system is set up with a TouchX joystick as the master device and a simulated Baxter robot arm as the slave robot. The haptic feedback is provided to the human operator to sense the interaction force between the slave robot and the environment when manipulating the stylus of the joystick. To utilize the workspace of the telerobot as much as possible, a matching process is carried out between the master and the slave based on their kinematics models. The closed loop inverse kinematics (CLIK) method and RBF NN approximation technique are seamlessly integrated in the control design. To overcome the potential instability problem in the presence of delayed communication channels, wave variables and their corrections are effectively embedded into the control system, and Lyapunov-based analysis is performed to theoretically establish the closed-loop stability. Comparative experiments have been conducted for a trajectory tracking task, under the different conditions of various communication delays. Experimental results show that in terms of tracking performance and force reflection, the proposed control approach shows superior performance over the conventional methods.

For most teleoperation tasks, it is desired that the telerobot manipulator follows timely and precisely the reference motion set at the master side. However, the conventional control approach may not guarantee the desired performance when there are dynamic uncertainties, especially when there is a notable variation of the telerobot's payload. In this paper, a neural learning based compensation mechanism has been exploited to overcome the effect of the unknown payload as well as uncertainties associated with the telerobot model and the environment. Guaranteed transient performance has been theoretically established. The deterministic learning technique has been employed, such that the neural learned knowledge can be efficiently reused. We performed comparative experiments and demonstrate the effectiveness of the proposed design techniques.

In this paper, the position tracking control with finite-time convergence has been studied for a class of nonliear uncertain robot manipulators. Radial basis function neural network (RBFNN) based adaptive control is designed to compensate for the effect of the unknown dynamics. To achieve the finite-time convergence of both trajectory tracking error and RBFNN learning error, barrier Lyapunov functions (BLFs) and and filtering techniques are employed to design a performance function and a tracking error region to ensure position tracking error converge to a pair of specified bounds in a finite time. The effectiveness and efficiency of the proposed control method is tested and verified by simulation studies.

The tutor-tutee hand-in-hand teaching may be the most effective approach for a tutee to acquire new motor skills. Repetitive nature of such procedures in a group setting usually results in a high labour cost and time inefficiency. Potential solution can be utilizing robotic platforms playing the role of tutors for demonstrating and transferring the required skills. This requires an appropriate guidance scheme to integrate the tutor's motor functionalities into the robot's control architecture. For instance, for hand-in-hand supervision of the writing task, the tutor's corrections can be applied when necessary, while a very compliant motion can be achieved if no errors are detected. Inspired by this behavior, we develop a teaching interface using a dual-arm robotic platform. In our setup, one arm is connected to the tutees arm providing guidance through a variable stiffness control approach, and the other to the tutor to capture the motion and to feedback the tutees performance in a haptic manner. The reference stiffness for the tutors arm stiffness is estimated in real-time and replicated by the tutees robotic arm. Comparative experiments have been carried out on a dual-arm Baxter robot. The results imply that the human tutor is able to intuitively transfer writing skills to the tutee and also show superior learning performance over over some conventional teaching by demonstration techniques.

In this paper, a vision-based adaptive control is designed for a wheeled inverted pendulum (WIP) robot to track a moving human target by integration of multisensor data. A new algorithm is employed in the system to combine an OptiTrack camera and a Kinect camera, such that more robust and efficient performance can be achieved for human target detection and tracking. Robust adaptive control has been developed for the WIP robot to maintain its balance on two wheels and to follow the human target using visual feedback. Leader-follower control, dynamic balance control and visual tracking are efficiently combined together to achieved desired tracking and balancing performance. Extensive experiment studies have been performed to test the effectiveness of the proposed control strategies.

The studies on bio-mechatronics systems and applications have been carried out for more than three decades, to overcome the challenges raised from both theoretical and experimental sides, especially those posed by the application of mechatronics and robotics in healthcare and medical fields. The research on biomedical robotics and bio-mechatronics covers a diverse spectrum of rapid rising interdisciplinary areas including bio-inspired robots for industrial, military, medical, and rehabilitation applications. This special issue aims at showcasing the most exciting and recent advances in the application of robotics and mechatronics in various fields and brings together a broad spectrum of topics covering various definition, development, control, and deployment of bio-mechatronics/robot systems, including social robots, wearable robot systems such as exoskeleton, rehabilitation robot, tele-robot, and a numbers of systems engineering approaches such as modeling, optimization and control. This special issue is to give analysis to the biological systems from a “bio-mechatronic” point of view, and to investigate the engineering and scientific principles behind their remarkable performance. High-quality original papers of innovative ideas and concepts have been included in the special issue of biomedical robotics and bio-mechatronics systems and application. While the design and development of bio-inspired machines and systems with novel and high performance in various applications have been investigated as well. The recent development of multidisciplinary research shall contribute to the promotion of the research on biomedical robotics and bio-mechatronics systems and application, with application to transportation, diagnosis, surgery, assistive technology, prosthetics, personal assistance, rehabilitation, health care, in laboratory, hospital, and the real world.

In this paper, a human robot shared control strategy is developed and tested on a Baxter robot. Using the proposed method, the human operator only needs to consider the motion of the end-effector of the manipulator, while the manipulator will avoid obstacle by itself without sacrificing the end effector motion performance. An improved obstacle avoidance strategy based on the joint space redundancy of the manipulator is designed. A dimension reduction method is presented to solve the over defined problem of avoiding velocity to achieve a more efficient use of the redundancy. By employment of an artificial parallel system of the teleoperate manipulator and the task switching weighting factor, the proposed control method enable the robot restoring back to the commanded pose smoothly when the obstacle is removed. By implementing the dimension reduction method, the trajectory of each joint of the manipulator can be controlled at the same time to achieve the restoring task. Thus, the proposed control method can eliminate the impact of the obstacle on the remaining task. Satisfactory experiment results demonstrate the effectiveness of the proposed methods.

In this paper, a decentralized adaptive fuzzy control has been developed for two cooperating robotic manipulators moving an object with impedance interaction. The contact forces are described using gradients of nonlinear potentials; then, the deformations of the contact surface can be obtained by an impedance approach. The cooperating manipulators are considered as a combination of subsystems, and the decentralized local dynamics coupled with physical interactions among the subsystems are developed. To compensate for the effect of dynamics uncertainties and external disturbances, decentralized fuzzy control combining parameter adaptations and disturbance observers is constructed. It guarantees the motion trajectories and impedance forces of the constrained object converging to the desired manifolds. It is theoretically established that the disturbance observers compensate for unparameterizable uncertainties, while the adaptive fuzzy mechanism compensates for the fast-changing components of the uncertainties that go beyond the disturbance observers. Moreover, unknown nonlinear dynamics such as the inertia matrix, Coriolis/centripetal matrix, and frictions, as well as interconnections with nonlinear bounds, can be accommodated through online learning. The experiments on two real robots have been carried out to verify the effectiveness of the proposed theoretical results.

This paper studies both indirect and direct global neural control of strict-feedback systems in the presence of unknown dynamics, using the dynamic surface control (DSC) technique in a novel manner. A new switching mechanism is designed to combine an adaptive neural controller in the neural approximation domain, together with the robust controller that pulls the transient states back into the neural approximation domain from the outside. In comparison with the conventional control techniques, which could only achieve semiglobally uniformly ultimately bounded stability, the proposed control scheme guarantees all the signals in the closed-loop system are globally uniformly ultimately bounded, such that the conventional constraints on initial conditions of the neural control system can be relaxed. The simulation studies of hypersonic flight vehicle (HFV) are performed to demonstrate the effectiveness of the proposed global neural DSC design.

In this paper, a class of nonholonomic chained systems is first converted into two subsystems, and then an explicit exponential decaying term is introduced into the input of the first subsystem to guarantee its controllability. After a state-scaling transformation, a model predictive control (MPC) scheme is proposed for the nonholonomic chained systems. The proposed MPC scheme employs a general projection neural network (GPN) to iteratively solve a quadratic programming (QP) problem over a finite receding horizon. The GPN employed in this paper is proved to be stable in the sense of Lyapunov, and its global convergence to the optimal solution is guaranteed for the reformulated QP. A simulation study is performed to show stable and convergent control performance under the proposed method, irrespective of whether the control input $\boldsymbol {u_{1}}$ vanishes or not.

This paper describes our preliminary development of a virtual robot teleoperation platform based on hand gesture recognition using visual information. Hand gestures in images captured by a camera are recognised to control a virtual iCub. We employ two methods to realise the classification: Adaptive Neuro-fuzzy Inference Systems (ANFIS) and Support Vector Machines (SVM). We realise the teleoperation of a virtual robot using iCubSimulator. The technique in the paper will enable us to teleoperate a physical robot in the future work. In addition, a video server is set up to monitor the real robot. By using the parallel system we are able to improve the robot's performance. Based on the techniques presented in this paper, the virtual iCub can perform the specified actions remotely in a natural manner.

In this paper, in order to combine intelligence of human operator and automatic function of the robot, we design a control scheme for the bimanual robot manipulation, in which the leading robot arm is directly manipulated by a human operator through a haptic device and the following robot arm will automatically adjust its motion to match the operator's motion. In this paper, we propose a fuzzy-based adaptive feedforward compensation controller and apply it into the robot control. According to the comparison results in the simulated experiment, we conclude that the fuzzy-adaptive controller performs better than the non-fuzzy controller, although they can both complete the specified task by tracking the leading robot arm controlled by the human operator. The techniques developed in this paper could be very useful for our future study on adaptation in human-robot interaction in improving the reliability, safety and intelligence.

This paper studies the composite adaptive tracking control for a class of uncertain nonlinear systems in strict-feedback form. Dynamic surface control technique is incorporated into radial-basis-function neural networks (NNs)-based control framework to eliminate the problem of explosion of complexity. To avoid the analytic computation, the command filter is employed to produce the command signals and their derivatives. Different from directly toward the asymptotic tracking, the accuracy of the identified neural models is taken into consideration. The prediction error between system state and serial-parallel estimation model is combined with compensated tracking error to construct the composite laws for NN weights updating. The uniformly ultimate boundedness stability is established using Lyapunov method. Simulation results are presented to demonstrate that the proposed method achieves smoother parameter adaption, better accuracy, and improved performance.

This paper investigates two surface electromyogram (sEMG)-based control strategies developed for a power-assist exoskeleton arm. Different from most of the existing position control approaches, this paper develops force control methods to make the exoskeleton robot behave like humans in order to provide better assistance. The exoskeleton robot is directly attached to a user's body and activated by the sEMG signals of the user's muscles, which reflect the user's motion intention. In the first proposed control method, the forces of agonist and antagonist muscles pair are estimated, and their difference is used to produce the torque of the corresponding joints. In the second method, linear discriminant analysis-based classifiers are introduced as the indicator of the motion type of the joints. Then, the classifier's outputs together with the estimated force of corresponding active muscle determine the torque control signals. Different from the conventional approaches, one classifier is assigned to each joint, which decreases the training time and largely simplifies the recognition process. Finally, the extensive experiments are conducted to illustrate the effectiveness of the proposed approaches.

In this brief, a novel adaptive-critic-based neural network (NN) controller is investigated for nonlinear pure-feedback systems. The controller design is based on the transformed predictor form, and the actor-critic NN control architecture includes two NNs, whereas the critic NN is used to approximate the strategic utility function, and the action NN is employed to minimize both the strategic utility function and the tracking error. A deterministic learning technique has been employed to guarantee that the partial persistent excitation condition of internal states is satisfied during tracking control to a periodic reference orbit. The uniformly ultimate boundedness of closed-loop signals is shown via Lyapunov stability analysis. Simulation results are presented to demonstrate the effectiveness of the proposed control.

In this paper, automatic motion control is investigated for one of wheeled inverted pendulum (WIP) models, which have been widely applied for modeling of a large range of two wheeled modern vehicles. First, the underactuated WIP model is decomposed into a fully actuated second order subsystem Σa consisting of planar movement of vehicle forward and yaw angular motions, and a nonactuated first order subsystem Σb of pendulum motion. Due to the unknown dynamics of subsystem Σa and the universal approximation ability of neural network (NN), an adaptive NN scheme has been employed for motion control of subsystem Σa . The model reference approach has been used whereas the reference model is optimized by the finite time linear quadratic regulation technique. The pendulum motion in the passive subsystem Σb is indirectly controlled using the dynamic coupling with planar forward motion of subsystem Σa , such that satisfactory tracking of a set pendulum tilt angle can be guaranteed. Rigours theoretic analysis has been established, and simulation studies have been performed to demonstrate the developed method.

This paper develops a learning control algorithm adapting the reference point and force to interact with an object of unknown geometry and elasticity. The controller is inspired by neuroscience studies that investigated the neural mechanisms when human adapt to virtual objects of different properties. The learning control algorithm estimates the shape and stiffness of the given object while maintaining a specified contact force with the environment. Simulations demonstrate the efficiency of the algorithm to identify the geometry and impedance of an unknown object without requiring force sensing. These properties are attractive for robotic haptic exploration with little demand on the sensing.

In this paper, we investigate optimized adaptive control and trajectory generation for a class of wheeled inverted pendulum (WIP) models of vehicle systems. Aiming at shaping the controlled vehicle dynamics to be of minimized motion tracking errors as well as angular accelerations, we employ the linear quadratic regulation optimization technique to obtain an optimal reference model. Adaptive control has then been developed using variable structure method to ensure the reference model to be exactly matched in a finite-time horizon, even in the presence of various internal and external uncertainties. The minimized yaw and tilt angular accelerations help to enhance the vehicle rider's comfort. In addition, due to the underactuated mechanism of WIP, the vehicle forward velocity dynamics cannot be controlled separately from the pendulum tilt angle dynamics. Inspired by the control strategy of human drivers, who usually manipulate the tilt angle to control the forward velocity, we design a neural-network-based adaptive generator of implicit control trajectory (AGICT) of the tilt angle which indirectly "controls" the forward velocity such that it tracks the desired velocity asymptotically. The stability and optimal tracking performance have been rigorously established by theoretic analysis. In addition, simulation studies have been carried out to demonstrate the efficiency of the developed AGICT and optimized adaptive controller.

The high conventional accuracy of pattern recognition-based surface myoelectric classification in laboratory experiments does not necessarily result in high accessibility to practical protheses. An obvious reason is the effect of signals of untrained classes caused by the relatively small training dataset. In order to make the classifier robust to untrained classes, a classification scheme is developed based on boosting and random forest classifiers in this paper. Meanwhile, a threshold, the post probability of the prediction, is introduced as a balance (i.e., adjust) between the accurate classification and the rejection of the samples belonging to some untrained classes. The experiments are conducted to compare with other two schemes using linear discriminant analysis and support vector machines. Surface electromyogram signals, labeled with seven isometric movements, are collected from six healthy subjects' forearm. It is shown that the proposed scheme can reach up to about 92% accuracy in recognizing trained classes and 20% for untrained classes. Through adjusting the threshold, the accuracy of rejecting untrained classes reaches up to around 80%, with small decrease in recognizing trained classes (down to 80%). In the analysis of experiments' results, we also find that the proposed scheme has better error distribution among the classes.

In this paper, model-free impedance control is designed for the safe human-robot interaction. A passive impedance model is imposed on the robot and a control method is proposed to guarantee the robot dynamics governed by the target model. The proposed method does not require any model information except for upper bounds of system matrix. It is thus easy to apply to practical implementation. The rigorous analysis of the control performance and robustness is presented. The validity of the proposed method is verified on the six degrees-of-freedom (DOF) PUMA 560 robot arm through simulation.

This paper introduces and analyzes an algorithm for adaptation of the reference trajectory of a human or robot arm interacting with a novel environment. The algorithm, based on the minimization of interaction force and performance error by satisfying a desired impedance, yields a mathematically rigorous model of the underlying mechanism of motion planning adaptation in humans. Simulations demonstrate a decrease of the interaction force to a limited amount as well as identification of the unknown interaction surface shape. These properties are attractive for adaptive motion of robots interacting with unknown surfaces, providing a robust behavior with little demand on the sensing.

This paper presents a novel human-like learning controller to interact with unknown environments. Strictly derived from the minimization of instability, motion error, and effort, the controller compensates for the disturbance in the environment in interaction tasks by adapting feedforward force and impedance. In contrast with conventional learning controllers, the new controller can deal with unstable situations that are typical of tool use and gradually acquire a desired stability margin. Simulations show that this controller is a good model of human motor adaptation. Robotic implementations further demonstrate its capabilities to optimally adapt interaction with dynamic environments and humans in joint torque controlled robots and variable impedance actuators, without requiring interaction force sensing.

In this paper, adaptive neural network (NN) control is investigated for a class of block triangular multiinput-multioutput nonlinear discrete-time systems with each subsystem in pure-feedback form with unknown control directions. These systems are of couplings in every equation of each subsystem, and different subsystems may have different orders. To avoid the noncausal problem in the control design, the system is transformed into a predictor form by rigorous derivation. By exploring the properties of the block triangular form, implicit controls are developed for each subsystem such that the couplings of inputs and states among subsystems have been completely decoupled. The radial basis function NN is employed to approximate the unknown control. Each subsystem achieves a semiglobal uniformly ultimately bounded stability with the proposed control, and simulation results are presented to demonstrate its efficiency.

Wheeled inverted pendulum (WIP) models have been widely used in the field of autonomous robotics and intelligent vehicles. A novel transportation system, WIP-car is proposed in this paper, which is composed of a mobile wheeled inverted pendulum system, a driven chair, an acceleration pedal and a deceleration pedal, which are used to drive the chair forward or backward such that the car can be accelerated or decelerated. The neural-adaptive implicit control is designed for dynamic balance and stable tracking of desired trajectories of WIP-car. Neither the dynamics nor the dimension of the regulated system is required to be known, while the relative degree of the regulated output is assumed to be known. Under the assumption that WIP-car is feedback linearizable, adaptive neural network is introduced to cancel the inversion dynamics error. Simulation results demonstrate that the system is able to track reference signals satisfactorily with all closed loop signals uniformly bounded.

In this paper, adaptive control is investigated for a class of discrete-time nonlinear multi-agent systems (MAS). Each agent is of uncertain dynamics and is affected by other agents in its neighborhood. An agent is able to sense the outputs of the agents inside its neighborhood but is unable to sense those outside its neighborhood. Among all the agents, there is a hidden leader, which knows the desired tracking trajectory, but it is affected by and can only affect those agents inside its neighborhood while all other agents are not aware of its leadership. The decentralized adaptive control is designed for each agent by using the information of its neighbors. Under the proposed decentralized adaptive controls, both rigid mathematical proof and simulation studies are provided to show that all the agents are guaranteed to reach their common goal, i.e., following the desired reference.

In this paper, adaptive neural network (NN) control is investigated for a class of nonlinear pure-feedback discrete-time systems. By using prediction functions of future states, the pure-feedback system is transformed into an n-step-ahead predictor, based on which state feedback NN control is synthesized. Next, by investigating the relationship between outputs and states, the system is transformed into an input-output predictor model, and then, output feedback control is constructed. To overcome the difficulty of nonaffine appearance of the control input, implicit function theorem is exploited in the control design and NN is employed to approximate the unknown function in the control. In both state feedback and output feedback control, only a single NN is used and the controller singularity is completely avoided. The closed-loop system achieves semiglobal uniform ultimate boundedness (SGUUB) stability and the output tracking error is made within a neighborhood around zero. Simulation results are presented to show the effectiveness of the proposed control approach.

In this paper, output feedback adaptive neural network (NN) controls are investigated for two classes of nonlinear discrete-time systems with unknown control directions: 1) nonlinear pure-feedback systems and 2) nonlinear autoregressive moving average with exogenous inputs (NARMAX) systems. To overcome the noncausal problem, which has been known to be a major obstacle in the discrete-time control design, both systems are transformed to a predictor for output feedback control design. Implicit function theorem is used to overcome the difficulty of the nonaffine appearance of the control input. The problem of lacking a priori knowledge on the control directions is solved by using discrete Nussbaum gain. The high-order neural network (HONN) is employed to approximate the unknown control. The closed-loop system achieves semiglobal uniformly-ultimately-bounded (SGUUB) stability and the output tracking error is made within a neighborhood around zero. Simulation results are presented to demonstrate the effectiveness of the proposed control.