Few-shot learning seeks to recognize novel classes from limited examples. Model-agnostic meta-learning (MAML), known for its simplicity and flexibility, learns an effective initialization for fast adaptation in data-scarce settings. However, MAML-based methods face challenges when there is a significant distributional shift between training and testing tasks, leading to inefficient learning and poor generalization across domains. In this work, we identify the core issues: inflexible weight update rules and limited adaptive learning capabilities. Instead of focusing solely on better initialization, we aim to enhance the adaptation process. Consequently, we propose a novel Layer-Adaptive Proportional-Integral-Derivative (LA-PID) optimizer integrated into a meta-learning framework. This design incorporates classical control theory, utilizing PID control to dynamically adjust task-specific gains at each network layer. Additionally, the theoretical conditions for optimal hyperparameter initialization and global model convergence are addressed from both control and optimization perspectives. Experiments on benchmark datasets show that LA-PID achieves state-of-the-art performance in few-shot classification, cross-domain, and regression tasks, while requiring fewer training steps.

Humanoid locomotion remains a fundamental challenge in robotics due to the high degrees of freedom, strong coupling between centroidal and joint-level dynamics, underactuation during walking phases, as well as pervasive model uncertainties and external disturbances. To address these challenges, this paper proposes an advanced whole-body planning and control framework that enables compliant posture regulation and stable locomotion through optimized centroidal motion and force planning. In the proposed framework, the full-body humanoid dynamics are first decoupled into centroidal momentum dynamics and joint-level dynamics. Based on this decomposition, a whole-body planning module formulates an optimization problem to generate dynamically consistent center-of-mass (CoM) trajectories and desired ground reaction forces (GRFs), which serve as reference inputs for posture adjustment and balance stabilization during walking. These planned centroidal quantities are then tracked by a whole-body control layer, where the desired GRFs are mapped into joint torques through a torque distribution scheme that preserves compliance and accommodates contact constraints. To cope with model uncertainties and unmodeled dynamics, an adaptive robust decentralized repetitive learning control strategy is integrated into the tracking control framework. By exploiting task repetitiveness, the proposed learning scheme iteratively refines control inputs, effectively compensating for system uncertainties without highly accurate models. A Lyapunov-based stability analysis demonstrates that the closed-loop tracking errors are uniformly ultimately bounded under bounded uncertainties. Extensive experimental results validate the effectiveness of the proposed approach, demonstrating stable posture regulation, robust walking performance, and strong disturbance rejection capabilities.

The task performance of mobile manipulators can be significantly enhanced by whole-body control and optimization in complex scenarios. Due to the nonlinear properties of whole-body dynamics and parameter uncertainty, modeling accurate system dynamics is essential in addition to designing an effective control strategy. However, traditional control methods have high computational costs and fail to deal with the parameter errors caused by model linearization. To address these issues, we propose a model predictive control (MPC)-Net, a learning-based approach that facilitates rapid online optimization by combining deep learning with multiple MPCs. Firstly, we develop a parameter identification algorithm based on a deep neural model to estimate the unknown dynamics parameters. Although the control performance of the MPC approach positively correlated with the prediction horizon, a long horizon would result in additional computational costs. Thus, MPC-Net is constructed by combining multiple sub-MPC issues, and the nonlinear coefficients are obtained by using a deep neural network-based optimization framework. Furthermore, MPC-Net generates the solution by combining the outputs of multiple sub-MPC problems using the nonlinear transformation of learned coefficients. Experiments are conducted on a mobile manipulator, which demonstrates the proposed MPC-Net-based optimization control offers fast efficient computation and low tracking error performance.

LiDAR point cloud registration is a fundamental problem in 3D computer vision. Recent learning-based methods have significantly improved the robustness and accuracy of LiDAR point cloud registration, while most of these methods are designed for global registration. In this paper, we focus on the post-refinement problem of LiDAR point cloud registration, which has been largely overlooked in previous learning-based approaches. To address the correspondence error between LiDAR point clouds, we formulate the transformation refinement problem as a correspondence optimization problem and propose DCR, a diffusion-based correspondence refinement model. DCR is built upon denoising diffusion models and recovers accurate correspondences from noisy initial correspondences estimated by arbitrary global registration methods. To achieve precise correspondence refinement, we employ a residual-based modeling scheme to constrain the output distribution and design a conditional generation paradigm to control the randomness of diffusion models. We combine DCR with both learning-based and traditional global registration methods and perform experiments on three large-scale outdoor LiDAR point cloud datasets to verify the performance. Extensive experimental results demonstrate DCR’s versatility and effectiveness in improving the accuracy of LiDAR point cloud registration.

This paper addresses the critical task of grounding clamp removal during live substation maintenance by developing an integrated perception–navigation–control framework for autonomous power station robots. The proposed system comprises three key components: (i) a vision–language navigation model that interprets natural-language commands and visual inputs for task-oriented motion planning; (ii) a vision–language perception module that localizes grounding clamps through multimodal reasoning; and (iii) a Lyapunov-regularized meta-learning adaptive control framework. While an integral Lyapunov function and a disturbance observer are utilized to construct the baseline robust controller, our primary control contribution lies in overcoming the persistent generalization bottlenecks of heuristic tuning. To achieve rapid adaptation under unmodeled dynamics and limited demonstration data, an offline neural dynamics model is first established to capture the robot’s highly nonlinear behaviors. Building upon this, we introduce a novel meta-optimization objective explicitly constrained by a Lyapunov stability penalty. This physics-informed meta-learning mechanism empowers the controller to autonomously extract generalized, stability-aware parameter initializations across diverse historical trajectories, thereby guaranteeing both data-driven adaptability and strict physical safety during complex substation tasks. The complete framework is validated on a wheeled humanoid robot performing live grounding operations in substation environments. Experimental results demonstrate that the proposed method achieves reliable autonomous navigation, accurate target manipulation, and significantly improved adaptability and stability compared to conventional and non–meta-learned controllers.

Ensuring the safety of interaction targets is paramount in robot interaction tasks, yet achieving this is particularly challenging with continuum manipulators due to their complex and highly nonlinear dynamics. To address this challenge, we propose a safety and energy-aware impedance control framework for robot interaction, which integrates a Vision Transformer (ViT) for target object recognition and is specifically designed for continuum manipulators with variable stiffness characteristics in both tilt and pan motions. This framework employs a dual-event triggering mechanism based on impedance control to enhance the safety and energy awareness of robot interaction systems. This mechanism adaptively adjusts the stiffness and damping coefficients when energy exceeds the object-specific thresholds determined through ViT classification, ensuring safe interaction levels and significantly enhancing the safety of interaction targets. Additionally, the incorporation of passivity theories and energy-tank methods further assures stability, preventing unintended energy generation and maintaining asymptotic stability throughout interactions. Finally, the effectiveness of the proposed approach is evaluated through collaborative operation experiments on a continuum manipulator, demonstrating its capability to ensure that interaction targets remain within safe energy levels in real-world applications. The proposed framework integrates ViT and dual-event triggered impedance control to ensure safe and energy-aware interactions with targets on continuum manipulators.

Developing non-visual sensing and intelligent recognition technologies is crucial for enhancing the manipulation performance of robots in dim or obstructed environments. Although precise object recognition has been extensively researched for rigid manipulators, the adoption of these techniques in soft robotic systems has been limited by the high modulus or low sensitivity of existing sensors (gauge factor/Young’s modulus$<$10 kPa$^{-1}$). Meanwhile, these systems necessitate advanced artificial intelligence(AI) algorithms to effectively process multiple sensing data. In this study, we utilize newly developed soft sensors and AI algorithm to establish a biomimetic perceptual soft gripper system capable of sensing and generating category object information during the grasping task. A strain/pressure bimodal sensor, mimicking the exceptional softness (Young’s modulus$<$10 kPa) and high sensitivity (gauge factor$>$2000) of human skin, has been developed and seamlessly integrated into a three-finger soft robotic gripper. A Swin Transformer network was developed to learn rules from bimodal data acquired from sensors and generate category information of the grasping objects. The perceptual gripper system exhibited superior recognition accuracy compared to previously reported systems, achieving an impressive 94.9% accuracy in categorizing 18 objects of varying shapes and sizes. We believe this advancement unlocks soft machines’ potential for automated applications.Note to Practitioners—Non-visual sensing and recognition techniques for soft robotic grippers are valuable for operation in dimly lit or obstructed environments. Traditional rigid sensor have limitations when applied to soft robotic systems due to their high modulus. This research developed novel ultra-soft and high sensitive strain/pressure bimodal sensors integrated into a three-fingered soft robotic gripper. Meanwhile, an AI algorithm applicable to the soft gripper system was developed to process the sensing data and generate the corresponding object category information when grasping the object. With the aid of the developed soft sensor and AI algorithm, the perceptual soft gripper can achieve high-precision object recognition, giving it great potential for automating tasks such as item picking, assembly, and handling in energy-efficient warehouses and logistics centers. The developed soft sensor and AI algorithm co-pilot object recognition strategy can contribute to the advancement of soft machine automation capability technology.

Humans can naturally learn and adapt to walking patterns in a variety of terrains. To simulate this learning characteristic, this article introduces a neural dynamics-based impedance optimization and trajectory adaptation approach for our designed soft exosuit, with a dual-driven configuration to assist both ankles of individuals. This method adaptively learns the impedance of the human ankle joint using measured interaction forces and dynamically adjusts trajectories to align with real-time human-robot interaction. Additionally, an adaptive control framework integrating neural dynamics-based optimization with several adaptive laws is developed to achieve stable tracking of updated reference trajectories, with Lyapunov stability analysis confirming uniform ultimate boundedness (UUB) of the closed-loop system. The designed controller offers the benefit of concurrently addressing trajectory adaptation, force control, and impedance tuning for soft exosuits. Experimental validation on human subjects across various terrains demonstrates that the proposed method reduces maximum trajectory tracking error to 0.016 rad (lower than PID and ADRC controllers) and enables impedance parameters to converge within 3 gait cycles. The controller concurrently addresses trajectory adaptation, force control, and impedance tuning, offering a lightweight (8 kg) and wearability-optimized solution for walking assistance.

In human-machine (autonomous vehicles and robots) interaction scenarios, pedestrians often appear in groups. Pedestrian groups provide richer information compared to individuals, which helps address occlusion problems in pedestrian-machine interactions. However, the randomness and spatiotemporal complexity of pedestrian activity make pedestrian group activity recognition (PGAR) a highly challenging task. This article provides a detailed description of the PGAR task. For the first time, a definition of pedestrian group and activity for autonomous vehicles and robots is provided. Existing datasets and methods are systematically summarized. Furthermore, the unique challenges and trends in PGAR for autonomous vehicles and robots are outlined. Although some related surveys have been published, there has not yet been a survey specifically focused on PGAR in autonomous driving and robotics scenarios. Therefore, the goal of this article is to narrow the gap in this topic and provide a comprehensive reference for researchers in this field.

Human-gaze-target prediction aims to predict the target point or object that humans are looking at in images. However, existing methods predominantly rely on vision-only features, which often struggle to capture the semantic context of small or occluded objects and lack explicit priors for precise head direction regression, leading to slow convergence and suboptimal performance. Therefore, we introduce VL-HTR, a novel vision-language learning method for human-target representation, which integrates multimodal knowledge from vision-language models (VLMs) to construct robust human-target relationships. Unlike traditional approaches, extracting multimodal features via pretrained VLMs enhances the model's grasp of human-target knowledge through the learnable target class and direction context. Then, a language-guided query alignment (LQA) module is introduced to improve the semantic-aware object representation capability through vision-language query alignment. Finally, to accelerate the gaze point regression learning process, we design a language-guided direction prediction (LDP) module to introduce multimodal human gaze direction priors, thereby facilitating the human-target relationship construction. Extensive validations across two distinct tasks, i.e., gaze object prediction (GOP) and gaze target estimation, involving five challenging benchmarks, demonstrating that VL-HTR achieves superior performance and much faster training convergence.

Wearable flexible lower-limb exoskeletons, commonly referred to as soft exosuits, have emerged as a promising technology for enhancing mobility and assisting gait rehabilitation. However, most conventional designs provide assistance failing to adapt to variations in walking cadence or gait phase subdivision. This study presents a planning and control frame work for a cable-driven exosuit that delivers personalised ankle assistance in the triadic interaction encompassing the human, the robot, and the environment. The framework enables automatic gait phase detection and adaptive assistance, dynamically responding to changes in gait, environmental conditions, and metabolic demands. Inspired by the periodic nature of human walking, a gait cycle is divided into eight phases using a designed transformer classification model that processes foot force and inertial measurement unit (IMU) data. Heart rate is incorporated to provide feedback on metabolic changes. A human-in-the-loop control strategy based on an event-triggered mechanism utilizing fine gait classification and heart rate is proposed to achieve ankle joint assistance adaptation. When the deviations in gait phases or metabolic conditions exceed thresholds, the ankle joint assistive trajectory is replanned online via optimization with safety constraints. An adaptive fuzzy controller ensures stable tracking under uncertain dynamics and external disturbances. The stability of the control system is analytically verified using Lyapunov theory, and experimental results demonstrate the effectiveness of the proposed approach.

In this paper, for manipulating flexible objects, e.g., connecting a grounding wire with the power line, in live-maintaining of power substations, we propose an action-level vision-language model (VLM) for flexible object positioning, including a fuzzy-based dynamic motion primitive (DMP), an adaptive admittance model for reshaping the task manipulation trajectory, and a fuzzy logic system-based (FLS) adaptive controller with event-triggered mechanism (ETM) for dynamic uncertainties. First, a pretrained VLM is co-finetuned with vision data to construct an action-level VLM called vision-language-action model (VLAM). Such an end-to-end model can directly generate proper target poses on deformable linear objects (DLOs), without the need for complex segmentation modules. Second, an admittance control with stiffness and damping adaptation is proposed to enable the robot to effectively interact with unmodeled DLOs. Then, a learning from demonstration (LfD) strategy based on DMP with fuzzy clustering is designed to transfer the live-maintaining skill. Additionally, an FLS-based adaptive controller with ETM is proposed to compensate for complex nonlinear dynamics and reduce communication frequency. Finally, a mobile manipulator for power substation live-maintaining is exploited to validate the proposed approaches through extensive experiments.

Self-supervised learning of point cloud aims to leverage unlabeled 3D data to learn meaningful representations without reliance on manual annotations. However, current approaches face challenges such as limited data diversity and inadequate augmentation for effective feature learning. To address these challenges, we propose GS-PT, which integrates 3D Gaussian Splatting (3DGS) into point cloud self-supervised learning for the first time. Our pipeline utilizes transformers as the backbone for self-supervised pre-training and introduces novel contrastive learning tasks through 3DGS. Specifically, the transformers aim to reconstruct the masked point cloud. 3DGS utilizes multi-view rendered images as input to generate enhanced point cloud distributions and novel view images, facilitating data augmentation and cross-modal contrastive learning. Additionally, we incorporate features from depth maps. By optimizing these tasks collectively, our method enriches the tri-modal self-supervised learning process, enabling the model to leverage the correlation across 3D point clouds and 2D images from various modalities. We freeze the encoder after pre-training and test the model’s performance on multiple downstream tasks. Experimental results indicate that GS-PT outperforms the off-the-shelf self-supervised learning methods on various downstream tasks including 3D object classification, real-world classifications, and few-shot learning and segmentation. Project page: https://github.com/Luoyeqi1/GS-PT.git

This paper proposes a hierarchical task-oriented whole-body locomotion framework for the exoskeleton walking-cart pushing (EWCP) task, which includes a straight gait and a bypassing gait, allowing the exoskeleton robot to avoid obstacles during walking. In this framework, the core components are gait planning and phase estimation for locomotion in unstructured environments. Notably, our mobile redundancy exoskeleton system can provide more flexibility and versatility in manipulation when performing complex tasks. For the hierarchical task-oriented whole-body locomotion, the detour gait consists of straight lines and turning shapes so that the EWCP system can avoid obstacles on the ground. For gait planning, we use the dynamic motion primitives to learn the joint motion trajectory of whole-body locomotion, which has good generalization ability and adaptability with respect to the gait. For phase estimation, the current gait phase can be estimated from the joint angles. Additionally, we design an task switching mechanism, where the exoskeleton system can switch different configurations flexibly for different scenarios, such as track switching only with both feet supported. And the phase estimation and gait switching strategy ensure the stability of task switching. The experimental results show that the exoskeleton can effectively accomplish EWCP tasks in an environment with obstacles. Our work has also shown that even some challenging motion tasks can be implemented with relatively simple controllers, which greatly simplifies the design of control systems.Note to Practitioners—This paper is motivated by issues of hierarchical tasks of the lower limb exoskeleton. Traditional exoskeletons cannot achieve obstacle avoidance in complex scenes because they do not have enough degrees of freedom or they do not use hierarchical locomotion. In this paper, a hierarchical task-oriented whole-body locomotion is proposed, which includes a straight gait and a bypassing gait, allowing the exoskeleton robot to avoid obstacles during walking. For gait planning, we use DMP to generate the gait trajectory, which can ensure the smoothness of trajectory. For phase estimation, the current gait phase can be estimated from the joint angles. Additionally, we design an task switching mechanism, where the exoskeleton system can switch different configurations flexibly for different scenarios, such as track switching only with both feet supported. And the phase estimation and gait switching strategy ensure the stability of task switching. The proposed hierarchical task-oriented whole-body locomotion framework is expected to be applied to exoskeletons to assist patients in rehabilitation training and users in mobility in daily life. Additionally, the proposed framework uses a series of motion primitives to learn and reproduce the trajectory and update it online, which requires a heavy computation load calculation to ensure the update speed of the trajectory. Therefore, we expect to reduce computation complexity for a walking exoskeleton in the future.

In this paper, an upper limb prosthesis has been furnished with a novel vision-based manipulation and grasping strategy. The proposed whole-body safety-critical control design includes vision servoing, multiple tasks planning with strict priorities, which can be formulated as an hierarchical multi-task optimization (HMO) problem with safety conditions—expressed as control barrier functions (CBF). Firstly, a modified YOLOv7 algorithm with key points detection is developed to determine the grasping pattern of the object and extract its edge contour information using a depth camera. An HMO-based strategy with a notion of CBF, providing inequality constraints in the control input, is proposed to handle multiple prioritized tasks with various constraints to offer guarantees of safety with the whole-body motion in consideration. Then the HMO problem is solved by a neuro-dynamics optimization solution online. Finally, experiments are implemented by using a self-developed upper limb prosthesis. Experimental results validate the performance of the proposed whole-body control strategy. Note to Practitioners—The intuitive, convenient and autonomous control of prosthetics has always been the object of researchers’ efforts. This paper proposes a whole-body safety-critical control design, including artificial perception system, autonomous control system under visual guidance, and user volition control system. The HMO strategy with safety conditions-expressed as control barrier functions in the context of real-time optimization-based method can fully consider the task requirements of different priorities and realize the orderly execution of prosthetic tasks. The vision feedback can detect surrounding environments in real time to obtain fine position information and optimal grasping pattern, which are conducive to reduce the amputees’ cognitive burden and provide new ideas for the whole body control of prostheses.

Enhancing energy efficiency is pivotal for advancing robotic lower limb prostheses. However, the conventional control strategy in powered prostheses displays notable variance in energy economy across terrains during walking. This challenge could be managed by developing control optimization methods to effectively reduce the subject’s metabolic cost and the device’s energy expenditure, thereby improving the walking economy in diverse environments. In this study, we investigate a multi-objective optimization problem (MOP) for controlling above-knee prostheses, intending to reduce prosthetic power consumption and the subject’s metabolic costs during walking activities. To bolster real-time performance in the optimization loop, we introduce a knee-guided evolutionary algorithm (KGEA) for MOP, which efficiently reduces exploration space and time complexity, thus enabling the practical implementation of a limited number of solutions in each iteration. The effectiveness of the proposed optimization-based strategy was evaluated through experiments involving two lower limb amputees engaging in slope, stair, and flat walking. Our results demonstrated that the optimization-based strategy significantly reduced the subjects’ metabolic consumption, with an average reduction of 16.53% for subject 1 and 10.23% for subject 2 across different terrains, compared to using the prosthesis without the optimization strategy. This promising outcome marks a substantial advancement in the development of energy-efficient, multi-degree-of-freedom power prostheses.Note to Practitioners—Lower Limb amputees using powered prostheses experience energy efficiency challenges due to limited battery life, particularly with multi-degree-of-freedom prostheses over varied terrains. Additionally, compared to passive alternatives, the increased weight of powered prosthetics poses a challenge in managing users’ metabolic costs, requiring suitable optimization strategies to offset the additional burden and optimize body economy during locomotion. The proposed KGEA-based multi-objective optimization method, designed explicitly for above-knee prostheses, targets the reduction of prosthetic power consumption and subject’s metabolic costs during walking activities. This innovation will likely benefit individuals with lower limb amputations, offering them improved mobility and comfort across various terrains. By addressing variance in varied terrains and inefficiencies in energy consumption caused by traditional control strategies, this work can assist practitioners in creating more effective and user-friendly prosthetic devices. Furthermore, we explore continuous refinement of the multi-objective optimization model and KGEA to accommodate a broader range of user-profiles and walking scenarios, ensuring its applicability in real-world situations.

Visual impairments significantly affect individuals’ ability to perform essential tasks such as communication, object search, and navigation. Traditional wearable augmented vision devices rely on modular designs that separate functions like perception and path planning, leading to cumulative errors and inefficiencies in real-world applications. To address these challenges, we propose an end-to-end multimodal large model framework, UniANS, specifically designed for wearable augmented vision devices. UniANS integrates visual perception, speech interaction, and path planning into a unified framework. Such integration improves task coordination, reduces error propagation, and enhances overall performance. We also propose a prompt design strategy with a mixture of cluster-conditional low-rank adaptation experts architectures and dual-branch encoders, combined with advanced preprocessing techniques for visual and speech modules. The framework has been validated through ablation studies, showing superior performance in accuracy and task effectiveness compared to existing methods. We further showcase its capabilities in addressing challenges related to communication, object search, and indoor navigation tasks. The design of UniANS enhances mobility and quality of life for visually impaired individuals.

Visually impaired individuals face limited mobility and restricted independent navigation in complex environments. Improving mobility and independent navigation for people with visual impairments is essential. In this paper, we introduce wearable electronic glasses (E-Glasses) that utilize a target detection network to fuse visual and auditory information for searching desired targets. Integrated with the electronic glasses, we present a neural path planning that combines spiking neural and convolutional neural networks. Several participants took part in experiments that showcased the remarkable capabilities of the developed system in target detection and navigation. The experimental results revealed an impressive success rate of 95.46% for the target detection network, providing participants with more accurate target information. Additionally, the neural path planning network achieved a success rate of 92.60%, demonstrating a significant speed advantage compared to the enhanced$A^{\ast}$algorithm.Note to Practitioners—This article aims to address the mobility and navigation challenges faced by visually impaired individuals in complex environments. Many solutions have been proposed to assist with wearable navigation by utilizing sensors to perceive the environment and provide path information to users. In this article, we introduce wearable electronic glasses (E-Glasses) with advanced target detection and path planning capabilities to help visually impaired individuals navigate more effectively in complex environments. We have developed a target detection network that integrates visual and auditory information to effectively search for desired targets. Additionally, we have developed a neural path planning algorithm that combines spiking neural networks and convolutional neural networks. Furthermore, experiments conducted in indoor navigation challenges demonstrate the feasibility of this approach. In future research, we will focus on further exploring the target detection network and refining the neural path planning algorithms to improve the overall performance of the system.

This paper proposes a novel navigation and optimization approach for mobile robots operating in intelligent manufacturing (IM) scenarios with unknown obstacles. First, we introduce a set of evaluation functions that simultaneously consider multiple metrics, including speed, security, sampling step, and obstacle avoidance ability, for scenarios with unknown dynamic obstacles. We then adopt a knee-guided multi-objective evolutionary algorithm capable of balancing parameter size and performance to optimize these conflicting objectives. Finally, we present a local navigation framework that integrates navigation and multi-objective optimization to enable obstacle avoidance in unstructured environments. The proposed algorithm is validated through simulations and practical scenarios, demonstrating its effectiveness in both static and dynamic scenarios.Note to Practitioners—The Human-cyber-physical System (HCPS) is the basic principle of the new generation of intelligent manufacturing (IM). It presents new requirements for the production method, and robot operate completely autonomously is expected to be the future solution for some complex tasks. However, traditional navigation methods often fail in these scenarios due to environmental changes and restricted workspaces. To address this challenge, this paper proposes a multi-objective optimization-based local navigation approach (MONA) that transforms the local navigation problem into a multi-objective optimization problem. The approach uses the knee-guided multi-objective evolutionary algorithm to optimize the control input for the mobile robot, ensuring safe and fast navigation in complex environments. The proposed approach has been validated through simulation and real scenario experiments, demonstrating its effectiveness.

Collaborative robots are predicted to interact physically with humans in human-cyber-physical systems (HCPSs). Robotic hands are able to record force and temperature simultaneously through soft and conformable sensors applied over finger surfaces and conforming to the complex curved geometries of automatic machines with tactile perception. Here, skin-inspired triple tactile (SITT) sensors are integrated into robotic fingers to enable precise bimanual grasping. The SITT sensor has skin-inspired multilayer microstructures, which integrate three sensors, namely, an interdigital electrode sensor, a flexible force sensor, and a temperature sensor. The SITT sensor can simultaneously or independently measure a material’s dielectric property, tactile force and temperature. An HCPS based on SITT sensors, a data acquisition board, bimanual robotic hands, and human-robot interaction software is developed for efficient bimanual manipulation. Through 3C assembly experiments, the designed HCPS is demonstrated to execute complex tasks. This research presents a novel methodology for constructing robust tactile sensors for robotic fingers in a bimanual manipulation system, and it presents significant potential across various aspects of intelligent production, including sensitive object handling, adaptive manipulation, and interactive robotics applications.Note to Practitioners—This work aims to overcome the challenge of skin-inspired triple tactile (SITT) sensors developed for robotic fingers in a human-robot collaborative assembly scenario, which can also be used in many other similar human-robot/machine collaborations (e.g., wearable prosthetics with tactile feedback) with practical value. Its capability to accurately measure the touched objects’ information (e.g., material dielectric property, force, temperature) is crucial for the bimanual robot to successfully interact with human operators. HCPS is valuable for its potential to achieve complicated interactions among humans, cyber systems, and physical resources. Collaborative robots with tactile sensors are expected to interact physically with humans in the HCPS. In this paper, we report SITT sensors integrated on robotic fingers to enable precise bimanual grasping. They can simultaneously or independently measure material dielectric properties and tactile forces and record temperature. By combining tactile perception information and a related control method, our smart bimanual robotic hands with the developed HCPS demonstrates the capability to execute 3C assembly in intelligent manufacturing. In the future, additional application scenarios will be designed for the HCPS platform, and some traditional tasks (such as a single arm for collaborative assembly) will be replaced by our hardware and software systems.

Human-robot collaboration plays an important role in intelligent manufacturing. However, the main challenge is how the robot can make online reactive changes to the plan based on the observed human behavior to ensure the completion of user-defined tasks. Such a challenge is further exacerbated if eye-in-hand manipulation is considered since the local field of the camera view cannot capture global observations. Different from existing planning approaches that separate the perception and planning modules, and make strong assumptions about perception abilities, we develop a framework of real-time local reactive planning that enables the robot to quickly adapt its actions if necessary through its limited perception of surroundings using an eye-in-hand camera. Specifically, we develop a locally observable transition system (LOTS) and interpretably express the task using linear temporal logic (LTL). To improve the grasping performance using local visual perception, we propose a high-resolution grasp network (HRG-Net) that achieves state-of-the-art results on multiple datasets (99.50% in Cornell and 97.50% in Jacquard and 96% in Graspnet-1Billion) for the task. A physical experiment using a 7DoF Franka Emika Panda robot demonstrates the effectiveness of the reactive planning framework.Note to Practitioners—Intelligent manufacturing often requires the human operator to work collaboratively with the robot in a shared workspace. Due to possible (assistive or non-assistive) interference of human operators, it is highly desired that the robot can perceive human behaviors and react properly to ensure task accomplishment. Hence, this work is particularly motivated to develop a reactive planning framework that relies on real-time local visual perception (i.e., eye-in-hand camera) to quickly react to its dynamic surroundings and replan its motion when necessary. In future work, rather than using the observed human behavior, we will investigate how to predict human intentions to further improve human-robot collaboration.

Despite remarkable progress in robotic exoskeletons, exoskeletons are still far from matching human-level guidance and locomotion performance in gait training or movement enhancement. A desirable exoskeleton would first provide a standard gait profile by learning from human operators while requiring body weight support with active human-following to govern dynamic locomotion synchronization. To address these issues, in this article, we propose a human operator-involved dynamic locomotion synchronization control framework for the lower limb exoskeleton actively following gait training with gravity-supporting. First, we designed a human motion capture system based on a five-link model for the locomotion of a human operator. To reproduce human-level motor skills, we use whole-body teleoperation to leverage human control intelligence to command the locomotion of a robotic exoskeleton system. Specifically, using the linear inverted pendulum (LIP) model, the human operator's divergent component of motion (DCM) is obtained by the human motion capture system. The dynamic similarity is used to generate a reference DCM for the robotic exoskeleton to synchronize the human operator's movement. Finally, a fuzzy-based adaptive controller is designed to track the synchronous trajectory for the exoskeleton in the presence of robotic dynamics uncertainties with input saturation. Experiments on the human subject are carried out to demonstrate the effectiveness of the proposed method.

The enhancement of underwater images has emerged as a significant technological challenge in advancing marine research and exploration tasks. Due to the scattering of suspended particles and absorption of light in underwater environments, underwater images tend to present blurriness and predominantly color distortion. In this study, we propose a novel approach utilizing denoising diffusion models to improve underwater degraded images. After training the noise estimation network of the denoising diffusion models, we accelerate the deterministic sampling process with denoising diffusion implicit models. We also propose a patch-based method by implementing average sampling between overlapping image patches at each sampling step, enabling the generation of images at arbitrary resolution while preserving their natural appearance and details. Through benchmark experiments, we illustrate that our method outperforms or closely approaches state-of-the-art techniques in terms of effectiveness and performance. We demonstrate that our approach reduces the interference of underwater environments with the semantic information of the images by salient object detection experiments.

Underwater areas have harsh environments with poor light, limited visibility, and high levels of noise. Humans have a weak perception of position, surroundings, and exterior information when staying underwater, which makes it difficult for humans to carry out complex underwater tasks, such as rescue, observation, and construction. Wearable devices have shown good results in enhancing human sensory function on land, thus they could potentially play a role in enhancing human underwater perception ability. This perspective aims to analyze the state-of-the-art of underwater wearable systems for human perception enhancement. This work discusses the core technology and challenges of human underwater perceptual enhancement, including wearable underwater navigation, underwater environment reconstruction, and underwater sensorial information delivery. Future research could focus on designing waterproof flexible human-machine interfaces for sensing and feedback, exploiting advanced sensors and fusion algorithms for wearable underwater positioning, and studying multimodal information interaction strategies of wearable systems.

For leg prosthesis user, the soft tissue and skin under the stump of are not accustomed to weight bearing, excessive continuous contact pressure can lead to the risk of degenerative tissue ulceration. This article presents a novel human-robot collaborative control scheme that achieves control weight self-adjustment for robotic prostheses to minimize interaction torque. To establish the human-robot interaction relationship, we regard the contact pressure between human residual limb and the prosthetic receiving cavity as the interaction force. We aim at reducing the interaction force under the premise of minimally changing the original motion trajectory of the robotic prosthesis. The control scheme mainly includes trajectory optimization based on a dual-agent game control scheme under a cooperative relationship, and a fuzzy logic system for improving the control accuracy of trajectory tracking of robotic prostheses with unknown dynamic parameters. Experiments were carried out on two amputee participants to verify the proposed human-robot interactive control scheme in a robotic prosthesis. The results show that the interaction torque could be reduced while maintaining minimal trajectory tracking error. The proposed control scheme could potentially facilitate the dexterous manipulation of leg prostheses, thus benefiting amputees.

Conventional lower limb exoskeletons (LLEs) and their corresponding rehabilitation protocols can hardly provide safe and customizable gait rehabilitation training for different patients and scenarios. Thus, this study presents an 8-DoF rehabilitation LLE equipped with a cable-driven body weight support (BWS) mobile mechanism. The mobile BWS mechanism is designed to follow the wearer and offer preset supportive forces and balance protection. A whole body motion planning approach is proposed, wherein iterative null-space projection is employed to solve the task-space trajectories of gait training into the joint-space trajectories of the LLE. For better control performance, dynamic parameters of the human-LLE coupling system are estimated. A control scheme combining robust model predictive control (MPC) and disturbance observer is then designed to manipulate the system against dynamics uncertainty and disturbance during trajectory tracking. In the validation experiments, the nominal model of robust MPC is discretized into quadratic programming problems and solved online by the neuro-dynamics optimization. The experimental results demonstrate the rationality of our system design and motion planning method as well as the effectiveness and stability of the control scheme.

Many previous works on wearable soft exosuits have primarily focused on assisting human motion, while overlooking safety concerns during movement. This article introduces a novel single-motor, altering bi-directional transfer soft exosuit based on impedance optimization and adaptive neural control, which provides assistance to the lower limbs using Bowden cables. This innovative soft exosuit integrates control barrier functions into the impedance optimization, allowing multiple safety constraints to be considered simultaneously, enabling the system to adaptively learn the impedance of the human ankle joint by analyzing the measured interaction forces at the ankle joint, so that the updated reference trajectories comply with safety requirements. To effectively track the updated reference trajectories, we have introduced an adaptive neural controller based on the integral barrier Lyapunov function. This controller is designed to perform the control task under strict safety constraints. The stability of this control approach is meticulously demonstrated through extensive Lyapunov analysis. In contrast to traditional soft exosuits designed purely for assistance, the key advantage of this technology is its ability to adapt to different terrains while ensuring the safety of human movement during assistance. Through experimental testing, we obtain average tracking errors of 0.0062, 0.0062, and 0.0063 rad for flat, grass, and gravel surfaces, respectively, demonstrating the effectiveness of the proposed strategy.

With the increasing popularity of autonomous driving systems and their applications in complex transportation scenarios, collaborative perception among multiple intelligent agents has become an important research direction. Existing single-agent multimodal fusion approaches are limited by their inability to leverage additional sensory data from nearby agents. In this article, we present the collaborative multimodal fusion network (CMMFNet) for distributed perception in multiagent systems. CMMFNet first extracts modality-specific features from LiDAR point clouds and camera images for each agent using dual-stream neural networks. To overcome the ambiguity in-depth prediction, we introduce a collaborative depth supervision module that projects dense fused point clouds onto image planes to generate more accurate depth ground truths. We then present modality-aware fusion strategies to aggregate homogeneous features across agents while preserving their distinctive properties. To align heterogeneous LiDAR and camera features, we introduce a modality consistency learning method. Finally, a transformer-based fusion module dynamically captures cross-modal correlations to produce a unified representation. Comprehensive evaluations on two extensive multiagent perception datasets, OPV2V and V2XSet, affirm the superiority of CMMFNet in detection performance, establishing a new benchmark in the field.

This article mainly investigates the problem of vibration suppression and angle cooperative tracking control of a multiple flexible manipulators described by partial differential equations (PDEs) with input quantization, actuator failures, and unmodeled system dynamics. An intermediate control law is designed, and a smooth function with a positive integrable time-varying function is introduced. Besides, a new smooth function is constructed in the control law to handle the influence of quantization and actuator faults. Under the designed controller, the angles of all flexible manipulators can reach consensus through mutual communication, and the elastic deformation of each flexible manipulator can also be suppressed. Furthermore, the asymptotic stability of a closed-loop system is realized based on the Lyapunov function. Finally, numerical simulation validates the effectiveness of the method.

Dempster-Shafer (DS) evidence theory provides a powerful framework for modeling uncertainty, reasoning, and combining information from multiple sources. However, it may yield counter-intuitive results when handling conflicting evidence, thereby affecting decision reliability and limiting practical applications. To address this issue, this work proposes a novel Evidential Reasoning rule with Divisive Hierarchical Clustering (ER-DHC), consisting of two main modules: evidence clustering and cluster fusion. At first, a new divisive hierarchical algorithm is introduced for evidence clustering, comprising coarse-grained and fine-grained division. In the coarse-grained stage, evidence with different decision preferences is grouped into separate clusters, thus preventing high intra-cluster conflicts and laying a solid foundation for evidence clustering. The fine-grained division adaptively refines cluster structures using an inflection point detection method, thereby enhancing clustering quality. On this basis, a new cluster fusion strategy is developed, involving intra-cluster fusion via classical Dempster's rule and inter-cluster fusion using a fuzzy preference relation-based weighted approach. This fusion strategy can degenerate into classical DS fusion and weighted fusion, while also introducing a new clustering fusion perspective, offering better flexibility. Finally, the proposed ER-DHC method is applied to the multi-source information fusion system, with experimental results demonstrating improved performance of target classification.

Adversarial attacks present a severe threat to the trustworthiness of remote sensing uncrewed systems. Existing detection methods are mostly limited to binary classification, lacking fine-grained traceability and relying heavily on adversarial example (AE) training. To overcome these challenges, we propose a training-free hierarchical adversarial attack traceability framework leveraging the zero-shot transfer ability of contrastive language-image pretraining (CLIP), eliminating the dependency on AE training. For the first time, we establish a multigranularity attack propagation path analysis system. Specifically, the framework leverages CLIP for fine-grained adversarial attack classification without fine-tuning, constructing a hierarchical traceability system (HTS) from coarse-grained to fine-grained semantic levels. Based on Bayesian inference, we design a hierarchical probability fusion method that improves coarse-grained results through maximum a posteriori (MAP) estimation of fine-grained classifiers, collaboratively optimizing hierarchical probability distributions. To capture frequency-domain characteristics of adversarial attacks, we propose a frequency-aware kernel approximation combining high-frequency-enhanced radial basis function (RBF) kernels with ridge regression, improving sensitivity to subtle perturbations in the reproducing kernel Hilbert space (RKHS). This study establishes a comprehensive traceability system for attack propagation chains, providing an interpretable, training-free paradigm for remote sensing security, significantly enhancing fine-grained detection capabilities of uncrewed systems. Extensive experiments on two public remote sensing datasets, RSSSN7 and DWEFS, and one proprietary dataset demonstrate that our method significantly improves adversarial classification accuracy across two victim models, achieving gains of 17.13%/18.89%/13.44% and 17.91%/19.58%/12.17%, respectively, compared with state-of-the-art (SOTA) training-free baselines.

Deep learning has demonstrated exceptional performance in fault diagnosis tasks that rely on large-scale datasets. However, the high cost of annotating such datasets has led to the emergence of various automatic annotation methods. While these methods reduce labeling costs, they inevitably introduce noisy labels, which pose significant challenges to the generalization and accuracy of deep learning-based diagnostic models. Although Learning with Noisy Labels (LNL) methods mitigate the adverse effects of noisy labels through strategies such as sample separation or label correction, many rely heavily on their own prediction results to guide subsequent training, which often introduces confirmation bias, limiting the effectiveness of the trained models. To address this limitation, this article draws inspiration from quantum collapse and proposes a novel LNL strategy named Multigranularity Labeling (MgL). By integrating observed labels, pseudolabels, and collapsed labels, MgL constructs the multigranularity labels, designed to suppress confirmation bias and improve the model's tolerance to noisy labels. Extensive experiments validate the effectiveness and superiority of MgL, particularly on training datasets with high noise intensity, such as those with 90% symmetric noise. This advancement offers promising opportunities for applying datasets with lower annotation costs in real-world scenarios, ultimately contributing to intelligent diagnostic systems.

Unsupervised domain adaptation (UDA) is vital for alleviating the workload of labeling 3D point cloud data and mitigating the absence of labels when facing an unseen domain. Various methods have recently emerged to utilize images along with point clouds to enhance the performance of cross-domain 3D segmentation. However, the pseudo labels, which are generated from models trained on the source domain and provide additional supervised signals for the target domain, are inadequate when utilized for 3D segmentation due to their inherent noisiness and consequently restrict the accuracy of neural networks. With the advent of 2D Visual Foundation Models (VFMs) and their abundant knowledge prior, we propose a novel pipeline VFMSeg to further enhance the cross-modal UDA framework by leveraging these models. In this work, we study how to harness the knowledge priors learned by VFMs to produce more accurate labels for unlabeled target domains and improve overall performance. We first utilize a multi-modal VFM, which is pre-trained on large-scale image-text pairs, to provide supervised labels (VFM-PL) for images and point clouds from the target domain. Then, we adopt another VFM to generate fine-grained 2D masks for guiding the generation of augmented images and point clouds, which mix the data from source and target domains like view frustums (FrustumMixing). Finally, we merge class-wise prediction across modalities to produce more accurate annotations for unlabeled target domains. Our method is evaluated on various autonomous driving datasets and the results demonstrate a significant improvement in 3D segmentation task. Our code is available athttps://github.com/EtronTech/VFMSeg

In partial-to-complete point cloud completion, it is imperative that enabling every patch in the output point cloud faithfully represents the corresponding patch in partial input, ensuring similarity in terms of geometric content. To achieve this objective, we propose a straightforward method dubbed PPCL that aims to maximize the mutual information between two point patches from the encoder and decoder by leveraging a contrastive learning framework. Contrastive learning facilitates the mapping of two similar point patches to corresponding points in a learned feature space. Notably, we explore multi-layer point patches contrastive learning (MPPCL) instead of operating on the whole point cloud. The negatives are exploited within the input point cloud itself rather than the rest of the datasets. To fully leverage the local geometries present in the partial inputs and enhance the quality of point patches in the encoder, we introduce Multi-level Feature Learning (MFL) and Hierarchical Feature Fusion (HFF) modules. These modules are also able to facilitate the learning of various levels of features. Moreover, Spatial-Channel Transformer Point Up-sampling (SCT) is devised to guide the decoder to construct a complete and fine-grained point cloud by leveraging enhanced point patches from our point patches contrastive learning. Extensive experiments demonstrate that our PPCL can achieve better quantitive and qualitative performance over off-the-shelf methods across various datasets.

The promotion of large-scale applications of reinforcement learning (RL) requires efficient training computation. While existing parallel RL frameworks encompass a variety of RL algorithms and parallelization techniques, the excessively burdensome communication frameworks hinder the attainment of the hardware's limit for final throughput and training effects on a single desktop. In this article, we propose Spreeze, a lightweight parallel framework for RL that efficiently utilizes a single desktop hardware resource to approach the throughput limit. We asynchronously parallelize the experience sampling, network update, performance evaluation, and visualization operations, and employ multiple efficient data transmission techniques to transfer various types of data between processes. The framework can automatically adjust the parallelization hyperparameters based on the computing ability of the hardware device in order to perform efficient large-batch updates. Based on the characteristics of the “Actor-Critic” RL algorithm, our framework uses dual GPUs to independently update the network of actors and critics in order to further improve throughput. Simulation results show that our framework can achieve up to 15,000 Hz experience sampling and 370,000 Hz network update frame rate using only a personal desktop computer, which is an order of magnitude higher than other mainstream parallel RL frameworks, resulting in a 73% reduction of training time. Our work on fully utilizing the hardware resources of a single desktop computer is fundamental to enabling efficient large-scale distributed RL training.

The stability criterion is critical for the design of legged robots' motion planning and control algorithms. If these algorithms cannot theoretically ensure legged robots' stability, we need many trials to identify suitable parameters for stable locomotion. However, most existing stability criteria are tailored to robots driven solely by legs and cannot be applied to thruster-assisted legged robots. Here, we propose a stability criterion for a thruster-assisted underwater hexapod robot by finding maximum and minimum allowable thruster forces and comparing them with the current thrusts to check its stability. On this basis, we propose a method to increase the robot's stability margin by adjusting the value of thrusts. This process is called stability enhancement. The criterion uses the optimization method to transform multiple variables such as attitude, velocity, acceleration of the robot body, and the angle and angular velocity of leg joints into one kind of variable (thrust) to judge the stability directly. In addition, the stability enhancement method is straightforward to implement because it only needs to adjust the thrusts. These provide insights into how multiclass forces such as inertia force, fluid force, thrust, gravity, and buoyancy affect the robot's stability.

This paper proposes a hybrid long short-term motor (HLSM) optimization and control approach for a walking exoskeleton. It consists of long-term global optimization, short-term local optimization, human-in-the-loop trajectory adaptation, and hybrid cerebellar model articulation controller (HCMAC). In the long-term global optimization, a graphic Spiking Neural Network (SNN) is utilized for an optimal global path. Along the path, the short-term motor optimization includes footstep optimization and obtains a sequence of footsteps. While in response to the unexpected obstacles along the footstep sequence, a human-in-the-loop planning strategy is designed by a virtual impedance model between the Centers of Mass (COMs) of the human and the exoskeleton, regulating the COM of the exoskeleton and generating footstep adaptation of the exoskeleton such that the exoskeleton can avoid obstacles and maintain its original global trajectory. Moreover, considering the unmodeled dynamics, we propose an HCMAC based on an integral Lyapunov function, which is exploited to counteract the system's nonlinear uncertainties, external disturbances, and reduces a relatively high computational cost. We validate the effectiveness of the HLSM planner and controller in a practical indoor setting. The results demonstrate the effectiveness of HLSM planning and control in a real scenario for a walking exoskeleton.

Outdoor sports pose a challenge for people with impaired vision. The demand for higher-speed mobility inspired us to develop a vision-based wearable steering assistance. To ensure broad applicability, we focused on a representative sports environment, the athletics track. Our efforts centered on improving the speed and accuracy of perception, enhancing planning adaptability for the real world, and providing swift and safe assistance for people with impaired vision. In perception, we engineered a lightweight multitask network capable of simultaneously detecting track lines and obstacles. Additionally, due to the limitations of existing datasets for supporting multi-task detection in athletics tracks, we diligently collected and annotated a new dataset (MAT) containing 1000 images. In planning, we integrated the methods of sampling and spline curves, addressing the planning challenges of curves. Meanwhile, we utilized the positions of the track lines and obstacles as constraints to guide people with impaired vision safely along the current track. Our system is deployed on an embedded device, Jetson Orin NX. Through outdoor experiments, it demonstrated adaptability in different sports scenarios, assisting users in achieving free movement of 400meter at an average speed of 1.34 m/s, meeting the level of normal people in jogging. Our MAT dataset is publicly available from https://github.com/snoopy-l/MAT

Collaborative perception is dedicated to tackling the constraints of single-agent perception, such as occlusions, based on the multiple agents' multi-view sensor inputs. However, most existing works assume an ideal condition that all agents' multi-view cameras are continuously available. In reality, cameras may be highly noisy, obscured or even failed during the collaboration. In this work, we introduce a new robust camera-insensitivity problem: how to overcome the issues caused by the failed camera perspectives, while stabilizing high collaborative performance with low calibration cost? To address above problems, we propose RCDN, a Robust Camera-insensitivity collaborative perception with a novel Dynamic feature-based 3D Neural modeling mechanism. The key intuition of RCDN is to construct collaborative neural rendering field representations to recover failed perceptual messages sent by multiple agents. To better model collaborative neural rendering field, RCDN first establishes a geometry BEV feature based time-invariant static field with other agents via fast hash grid modeling. Based on the static background field, the proposed time-varying dynamic field can model corresponding motion vector for foregrounds with appropriate positions. To validate RCDN, we create OPV2V-N, a new large-scale dataset with manual labelling under different camera failed scenarios. Extensive experiments conducted on OPV2V-N show that RCDN can be ported to other baselines and improve their robustness in extreme camera-insensitivity setting. Our code and datasets will be available soon.

An amputee cannot directly use the perceptual visual information to control the movements and gait patterns of his worn prosthesis. In order to help an amputee walk and cross over obstacles smoothly, the paper proposes a vision-locomotion coordination control method for a powered lower-limb prosthesis (PLLP), in which a vision system is proposed to detect obstacles, and a complete vision-locomotion loop is then constructed. With deep learning techniques, the vision system can recognize common obstacles (e.g., garbage cans, bricks and boxes) and obtain the features of obstacles (e.g., distance from obstacles to the depth camera and height of obstacles). Through integrating the vision system into the locomotion control system, the PLLP can make obstacle avoidance decisions and use dynamic movement primitives with type-2 fuzzy models (T2FDMPs) to help amputees cross over obstacles simultaneously. Utilizing the type-2 fuzzy models, smooth trajectories for crossing over obstacles can be obtained. The experimental results show that the PLLP with visual information can switch an amputee’s gaits between level walking and obstacle avoidance adaptively, which demonstrates the effectiveness of the vision-locomotion coordination control system.Note to Practitioners—The paper is motivated by the challenge of obstacle avoidance of prostheses. Traditional prostheses cannot achieve autonomous obstacle avoidance because they lack of environmental perception capability. In addition, most prostheses always utilize use the human-robot interaction between amputees and prostheses to recognize the environments. However, due to noise and individual differences, recognition results are not accurate enough. We found that the integration of vision can improve the accuracy and efficiency of environmental recognition. Therefore, it is necessary to construct a complete vision and locomotion closed-loop. In the paper, a vision-locomotion coordination control is proposed, and to make the PLLP cross over obstacles smoothly, a novel trajectory shaping with fuzzy-based dynamic movement primitives is developed. The coordination control is partitioned into the obstacle detection, trajectory shaping and joint control, which can help the PLLP fulfill several obstacle avoidance tasks.

Industries, such as manufacturing, are accelerating their embrace of the metaverse to achieve higher productivity, especially in complex industrial scheduling. In view of the growing parking challenges in large cities, high-density vehicle spatial scheduling is one of the potential solutions. Stack-based parking lots utilize parking robots to densely park vehicles in the vertical stacks like container stacking, which greatly reduces the aisle area in the parking lot, but requires complex scheduling algorithms to park and take out the vehicles. The existing high-density parking (HDP) scheduling algorithms are mainly heuristic methods, which only contain simple logic and are difficult to utilize information effectively. We propose a hybrid residual multiexpert (HIRE) reinforcement learning (RL) approach, a method for interactive learning in the digital industrial metaverse, which efficiently solves the HDP batch space scheduling problem. In our proposed framework, each heuristic scheduling method is considered as an expert. The neural network trained by RL assigns the expert strategy according to the current parking lot state. Furthermore, to avoid being limited by heuristic expert performance, the proposed hierarchical network framework also sets up a residual output channel. Experiments show that our proposed algorithm outperforms various advanced heuristic methods and the end-to-end RL method in the number of vehicle maneuvers, and has good robustness to the parking lot size and the estimation accuracy of vehicle exit time. We believe that the proposed HIRE RL method can be effectively and conveniently applied to practical application scenarios, which can be regarded as a key step for RL to enter the practical application stage of the industrial metaverse.

Wearable walking exoskeletons show great potentials in helping patients with neuro musculoskeletal stroke. Key to the successful applications is the design of effective walking trajectories that enable smooth walking for exoskeletons. This work proposes a walking planning method based on the divergent component of motion to obtain a stable joint angle trajectory. Since periodic and nonperiodic disturbances are ubiquitous in the repeating walking motion of an exoskeleton system, a major challenge in the walking control of wearable exoskeleton is the joint angle drift problem, that is, the joint angle motion trajectories are not necessarily periodic due to the presence of disturbance. To address this challenge, this work develops an adaptive repetitive control strategy to guarantee that the motion trajectories of joint angle are repetitive. In particular, by treating the disturbance as system uncertainties, an adaptive controller is designed to compensate for the uncertainties based on an integral-type Lyapunov function. A fully saturated learning approach is then developed to achieve asymptotic tracking of repetitive walking trajectories. Extensive experiments are carried out to demonstrate the effectiveness of the tracking performance.

This article presents a human-in-the-loop cooperative control of a walking exoskeleton to provide assistance to the user and enhance human mobility. First, a dynamic mathematical model of the human-exoskeleton system is derived, and then, a human-in-the-loop cooperative control framework is proposed in two ways: separable cooperative control (SCC) and interactive cooperative control (ICC), respectively. The SCC introduces a space division of the human and the exoskeleton, while the ICC allows the robot to perceive the human intention and follows human motor, thereby improving the collaborative performance. The ICC formulates the impedance connection between the human and the exoskeleton in the divided orthogonal subspaces of walking, such that the robot is able to modify its position of center of mass (COM) when its motor trajectory deviates the one of the human. In addition, a novel adaptation control is proposed to deal with the unmodeled dynamics and trajectory tracking. Finally, to validate the effectiveness of our proposed controller, a series of experiments are conducted in three adults in gait at different speeds. It shows that the proposed controller can preserve the periodic walking gait and inherit the robustness of dealing with perturbations during walking.

Human-centered environments provide affordance for the use of two-handed mobile manipulators. Yet robots designed to function in and physically interact with such environments are not yet capable of meeting human users' requirements. This work proposes a whole body control framework of a two-handed mobile manipulator driven by series elastic actuators (SEAs) for cart pushing tasks. A whole body dynamic model for an integrated mobile platform and on-board arms is revealed, which takes into account the interaction forces with the cart. Then, the explicit force/position control of the mobile manipulator is performed. It enables the robot to interact dynamically with the environment while providing motion, i.e., the manipulators provide both output force control and motion control for pushing a cart. To cope with the highly nonlinear system dynamics and parameter variation of a SEA-driven mobile manipulator, this work proposes an adaptive robust controller based on a novel integral barrier Lyapunov function for cart pushing tasks by considering model uncertainty. The proposed controller enables the mobile manipulator to complete cart pushing tasks by regulating the position and output force of the mobile base and arms. The experimental results show the effectiveness of this approach in cart pushing tasks.

This article presents U 2PNet, a novel unsupervised underwater image restoration network using polarization for improving signal-to-noise ratio and image quality in underwater imaging environments. Traditional methods for underwater image restoration using polarization require specific cues or pairs of underwater polarization datasets, which limit their practical applications. Our proposed method requires only one mosaicked polarized image of the scene and does not require datasets for pretraining or specific cues. We design two subnetworks (T-net and B textsubscript ∞ -net) to accurately estimate the transmission map and background light, and unique nonreference loss functions to ensure effective restoration. Our experiments are based on an indoor polarization simulated dataset and a real polarization image dataset constructed from our underwater robotic platform equipped with polarization cameras. Experiment results demonstrate that our proposed method achieves state-of-the-art performance on both simulated and real underwater polarization images. The code and datasets will be available at https://github.com/polwork/U-2Pnet.

Driven by the rise of collaborative robots, a lot of work has focused on the transparency and stability of physical human–robot interaction (pHRI), in which most of the efforts do not take the requirement of multiple tasks into account. However, the spectrum of applications for collaborative robots has been continuously broadened, and robots without the ability to perform multiple tasks simultaneously may not be capable of collaborating in certain scenarios. In this article, we provide a fuzzy-based multitask intelligent control framework of collaborative robots for pHRI. Our controller formulation consists of “outer-loop” and “inner-loop.” In the “outer-loop,” a fuzzy logic system predicts human desired motion trajectory for the robot to track. In the “inner-loop,” the robot is driven by a hierarchical multitask controller to track the trajectory generated by the “outer-loop” and perform other subtasks simultaneously. With null space projections, the whole task stack can be implemented in a strict task hierarchy in order of priority. The weight-tuning law of the FLSs and the hierarchical multitask control law are given based on Lyapunov stability analysis. The proposed control framework is applied to a mobile manipulator and the effectiveness is verified by exploratory experiments. Results confirm the effectiveness of the proposed control framework and compare its performance with other approaches.

Many multi-view camera-based 3D object detection models transform the image features into Bird’s-Eye-View (BEV) via the Lift-Splat-Shoot (LSS) mechanism, which “lifts” 2D camera-view features to the 3D voxel space based on the predicted depth distribution and then “splats” 3D features into a BEV plane for subsequent 3D object detection. However, the BEV feature in such a one-stage view transformation scheme heavily relies on the quality of the predicted depth distribution and 2D camera-view features, which further determines the final detection performance. In this paper, we propose a BEVRefiner model which performs dual refinement for both depth prediction and 2D camera-view features. On the one hand, we perform light-weight depth refinement in the depth distribution frustum space by incorporating 3D context and depth distribution prior. On the other hand, we reproject the BEV feature back to each camera view to enhance 2D image features. In this way, the original camera-view features can be enhanced by implicitly incorporating 3D contexts and multi-view contexts, which cannot be achieved in the original 2D camera view. We also propose to use dominant depth bins only for the reprojection to save computational burden. Finally, we generate the refined BEV feature using the refined depth distribution and camera-view features for more accurate 3D object detection. Our BEVRefiner can be plugged into LSS-based BEV detectors and we perform extensive experiments on the representative model BEVDet, which strongly verified the efficiency of our proposed approach under several settings.

Surface electromyography (sEMG) signal has been used in upper limb rehabilitation robots (ULRR). However, existing ULRR based on myoelectric controllers suffers from limited generalization ability in estimating three-dimensional (3-D) motion intention. This article proposes a muscle-synergy-inspired approach to enhance the generalization ability of the myoelectric controller of a cable-driven ULRR. Low-dimensional commands are extracted from sEMG signals based on an EMG-to-muscle activation model and non-negative matrix factorization. The extracted commands are used to estimate the 3-D human force. Two different trajectory tracking tasks are selected to test the generalization ability. The system is trained based on training sets where participants perform one task. Then the system is tested using testing sets where participants perform the other task. Finally, the system is verified on real-time robotic control experiment. Results show that the proposed controller achieves better force estimating accuracy, better trajectory tracking accuracy, and lower interaction force than the myoelectric controller without considering muscle synergies, which means the proposed controller yields better generalization performance.

Event cameras have the ability to record continuous and detailed trajectories of objects with high temporal resolution, thereby providing intuitive motion cues for optical flow estimation. Nevertheless, most existing learning-based approaches for event optical flow estimation directly remould the paradigm of conventional images by representing the consecutive event stream as static frames, ignoring the inherent temporal continuity of event data. In this paper, we argue that temporal continuity is a vital element of event-based optical flow and propose a novel Temporal Motion Aggregation (TMA) approach to unlock its potential. Technically, TMA comprises three components: an event splitting strategy to incorporate intermediate motion information underlying the temporal context, a linear lookup strategy to align temporally fine-grained motion features and a novel motion pattern aggregation module to emphasize consistent patterns for motion feature enhancement. By incorporating temporally fine-grained motion information, TMA can derive better flow estimates than existing methods at early stages, which not only enables TMA to obtain more accurate final predictions, but also greatly reduces the demand for a number of refinements. Extensive experiments on DSEC-Flow and MVSEC datasets verify the effectiveness and superiority of our TMA. Remarkably, compared to E-RAFT, TMA achieves a 6% improvement in accuracy and a 40% reduction in inference time on DSEC-Flow. Code will be available at https://github.com/ispc-lab/TMA.

Offline safe reinforcement learning (RL) algorithms promise to learn policies that satisfy safety constraints directly in offline datasets without interacting with the environment. This arrangement is particularly important in scenarios with high sampling costs and potential dangers, such as autonomous driving and robotics. However, the influence of safety constraints and out-of-distribution (OOD) actions have made it challenging for previous methods to achieve high reward returns while ensuring safety. In this work, we propose a Variational Optimization with Conservative Eestimation algorithm (VOCE) to solve the problem of optimizing safety policies in the offline dataset. Concretely, we reframe the problem of offline safe RL using probabilistic inference, which introduces variational distributions to make the optimization of policies more flexible. Subsequently, we utilize pessimistic estimation methods to estimate the Q-value of cost and reward, which mitigates the extrapolation errors induced by OOD actions. Finally, extensive experiments demonstrate that the VOCE algorithm achieves competitive performance across multiple experimental tasks, particularly outperforming state-of-the-art algorithms in terms of safety.

Navigating independently is a challenge for visually impaired vision due to the demand of obstacles avoiding, recognizing desired objects, and wayfinding in complicated environments. In this paper, we present an augmented wearable E-Glasses with a set of sensors, where an object detection neural network based on visual-auditory fusion method is employed to search desired targets, thus addressing navigation challenges and improving the mobility and independence of the visually impaired. We demonstrate advanced navigation capabilities: indoor wayfinding, recognizing and steering the users to desired goals, and a sequence of indoor challenges. The fusion network adopts a feature-level fusion strategy, which is capable to align two modalities automatically and effectively integrate visual features and audio features. Across all experiments, the developed fusion algorithm has a 94.67% success rate. The wearable E-Glasses supply a platform that helps to improve the mobility and quality of life of people with impaired vision.

Point cloud registration is a fundamental problem in 3D computer vision. Previous learning-based methods for LiDAR point cloud registration can be categorized into two schemes: dense-to-dense matching methods and sparse-to-sparse matching methods. However, for large-scale outdoor LiDAR point clouds, solving dense point correspondences is time-consuming, whereas sparse keypoint matching easily suffers from keypoint detection error. In this paper, we propose SDMNet, a novel Sparse-to-Dense Matching Network for large-scale outdoor LiDAR point cloud registration. Specifically, SDMNet performs registration in two sequential stages: sparse matching stage and local-dense matching stage. In the sparse matching stage, we sample a set of sparse points from the source point cloud and then match them to the dense target point cloud using a spatial consistency enhanced soft matching network and a robust outlier rejection module. Furthermore, a novel neighborhood matching module is developed to incorporate local neighborhood consensus, significantly improving performance. The local-dense matching stage is followed for fine-grained performance, where dense correspondences are efficiently obtained by performing point matching in local spatial neighborhoods of high-confidence sparse correspondences. Extensive experiments on three large-scale outdoor LiDAR point cloud datasets demonstrate that the proposed SDMNet achieves state-of-the-art performance with high efficiency.

In this paper, an integrated framework consisting of the sensing, navigation and control is proposed for an autonomous mobile manipulator driven by series elastic actuators (SEAs) to preform mobile manipulation tasks in unknown environments. First, ORB-SLAM2 technique is combined into the environment sensing by extracting the ORB features, automatic initialization, repositioning and loop detection for real-time posture estimation. Then, the navigation function is designed for generating collision-free trajectory in an environment with obstacles. To realize kinematic and dynamic control of the mobile manipulator with the developed SEA joints, the whole body dynamics is considered and described. And to handle dynamic uncertainties and the SEA inherent saturation limits, a novel adaptive neural network control considering the whole body dynamics is proposed. Without knowing the exact parameters of the whole body model, the designed controller merely requires the position and velocity of the actuators and links, which can make the tracking errors converge to zero and keep all signals uniformly bounded in the closed-loop system. The performance and efficiency of the proposed method are verified by extensive experiments. Note to Practitioners—This paper is motivated by issues of manipulation control of autonomous unmanned system. Traditional manipulation frameworks focus either on sensing or control by assuming that the environment is known, which would result in lacking of autonomy for a specified task. Since mobile manipulation tasks often consist of nonholonomic and holonomic constraints for wheeled mobile manipulators with differential steering. In addition, most current works for whole body control are based on the condition that robot dynamic parameters are known beforehand. Therefore, it is necessary to establish an enhanced framework to simultaneously deal with these problems. In this paper, an integrated navigation and control framework is proposed. To make the mobile manipulator work in the unknown environment, task sensing and whole body control for mobile manipulators are also developed. The framework is partitioned into the task sensing, navigation and control, where the mobile manipulator can fulfill the mobile manipulation tasks in the unstructured environments.

Robot learning through kinesthetic teaching is a promising way of cloning human behaviors, but it has its limits in the performance of complex tasks with small amounts of data, due to compounding errors. In order to improve the robustness and adaptability of imitation learning, a hierarchical learning strategy is proposed: low-level learning comprises only behavioral cloning with supervised learning, and high-level learning constitutes policy improvement. First, the Gaussian mixture model (GMM)-based dynamical system is formulated to encode a motion from the demonstration. We then derive the sufficient conditions of the GMM parameters that guarantee the global stability of the dynamical system from any initial state, using the Lyapunov stability theorem. Generally, imitation learning should reason about the motion well into the future for a wide range of tasks; it is significant to improve the adaptability of the learning method by policy improvement. Finally, a method based on exponential natural evolution strategies is proposed to optimize the parameters of the dynamical system associated with the stiffness of variable impedance control, in which the exploration noise is subject to stability conditions of the dynamical system in the exploration space, thus guaranteeing the global stability. Empirical evaluations are conducted on manipulators for different scenarios, including motion planning with obstacle avoidance and stiffness learning.

Soft exo-suit could facilitate walking assistance activities (such as level walking, upslope, and downslope) for unimpaired individuals. In this article, a novel human-in-the-loop adaptive control scheme is presented for a soft exo-suit, which provides ankle plantarflexion assistance with unknown human-exosuit dynamic model parameters. First, the human-exosuit coupled dynamic model is formulated to express the mathematical relationship between the exo-suit actuation system and the human ankle joint. Then, a gait detection approach, including plantarflexion assistance timing and planning, is proposed. Inspired by the control strategy that is used by the human central nervous system (CNS) to handle interaction tasks, a human-in-the-loop adaptive controller is proposed to adapt the unknown exo-suit actuator dynamics and human ankle impedance. The proposed controller can emulate human CNS behaviors which adapt feedforward force and environment impedance in interaction tasks. The resulting adaptation of actuator dynamics and ankle impedance is demonstrated with five unimpaired subjects and implemented on a developed soft exo-suit. The human-like adaptivity is performed by the exo-suit in several human walking speeds and illustrates the promising potential of the novel controller.

This article presents an active human-following control of the lower limb exoskeleton for gait training. First, to improve safety, considering the human balance, the OpenPose-based visual feedback is used to estimate the individual's pose, then, the active human-following algorithm is proposed for the exoskeleton robot to achieve the body weight support and active human-following. Second, taking the human's intention and voluntary efforts into account, we develop a long short-term memory (LSTM) network to extract surface electromyography (sEMG) to build the estimation model of joints' angles, that is, the multichannel sEMG signals can be correlated with flexion/extension (FE) joints' angles of the human lower limb. Finally, to make the robot motion adapt to the locomotion of subjects under uncertain nonlinear dynamics, an adaptive control strategy is designed to drive the exoskeleton robot to track the desired locomotion trajectories stably. To verify the effectiveness of the proposed control framework, several recruited subjects participated in the experiments. Experimental results show that the proposed joints' angles estimation model based on the LSTM network has a higher estimation accuracy and predicted performance compared with the existing deep neural network, and good simultaneous locomotion tracking performance is achieved by the designed control strategy, which indicates that the proposed control can assist subjects to perform gait training effectively.

Open-set domain adaptation (OSDA) aims to achieve knowledge transfer in the presence of both domain shift and label shift, which assumes that there exist additional unknown target classes not presented in the source domain. To solve the OSDA problem, most existing methods introduce an additional unknown class to the source classifier and represent the unknown target instances as a whole. However, it is unreasonable to treat all unknown target instances as a group since these unknown instances typically consist of distinct categories and distributions. It is challenging to identify all unknown instances with only one additional class. In addition, most existing methods directly introduce marginal distribution alignment to alleviate distribution shift between the source and target domains, failing to learn discriminative class boundaries in the target domain since they ignore categorical discriminative information in the adaptation. To address these problems, in this article, we propose a novel prototype-based shared-dummy classifier (PSDC) model for the OSDA. Specifically, our PSDC introduces an auxiliary dummy classifier to calibrate the source classifier and simultaneously develops a weighted adaptation procedure to align class-wise prototypes for adaptation. We further design a pseudo-unknown learning algorithm to reduce the open-set risk. Extensive experiments on Office-31, Office-Home, and VisDA datasets show that the proposed PSDC can outperform existing methods and achieve the new state-of-the-art performance. The code will be made public.

Currently, type-2 fuzzy systems, fuzzy control strategies, and origami structures have been used in robots to assist in the comfortable and smooth operation of such robots. Recent advances in these various fields have been employed to improve the key technologies of wearable robots, such as providing more efficient decision making, improved maneuverability, increased control intelligence, and more lightweight structures. The current advances have highlighted the potential for these various methods, both separately and in combination with each other, to achieve further significant advances. Hence, this article summarizes the latest research results in these three key aspects, elaborates on some of the challenges that remain, and discusses potential development directions of wearable robots in the future.

The integration and interaction of proprioception and exteroception in the human multisensory network facilitate high-level cognitive functionalities, such as cross-modal integration, recognition, and imagination for accurate evaluation and comprehensive understanding of the multimodal world. In this article, we propose a novel cross-modal integration framework for the upper limb prosthesis based on type-2 fuzzy logic system (FLS), which can facilitate the high-level cognitive and dexterous manipulation of the prosthesis by combing human's surface electromyography (sEMG) with the computer vision. First, a transfer learning approach is proposed to improve the decoding of human's intent and enhance the effectiveness of skill transition. Then the sEMG signals and image information are integrated to jointly determine the grasp posture of the bionic hand based on fuzzy decision strategy. Fusing multisensory data and using cross-modal integration, the system is capable of crossmodally recognizing multimodal information. In order to realize the prosthesis automatically reaching the target position under the guidance of computer vision, an interval type-2 fuzzy logic controller considering uncertain dynamic parameters and disturbance is designed. Experiments are performed in some typical 3C assembly scenarios, and results show that our proposed strategy can obviously improve the accuracy of grasp posture selection and trajectory tracking effect, which brings more potential job opportunities with hope to amputees, and provides a promising approach toward robotic sensing and perception.

Many previous studies of soft exosuits improved human locomotion performance. However, there is no example to control a soft exosuit using human ankle impedance adaption in assistance tasks compliantly. In this article, the human–environment interaction information is exploited into the exosuit control. A novel fuzzy-based optimization and control method of soft exosuit is proposed to provide plantarflexion assistance for human walking by changing the human–robot interaction. In particular, a fuzzy neurodynamics optimization is developed to learn the unknown human ankle impedance parameters automatically. A fuzzy approximation technique is applied to improve the control performance of the exosuit when a human is walking with unknown human–robot interaction model parameters. This control scheme guarantees that the human–robot dynamics follows a target human ankle impedance model to obtain the compliant interaction performance. Experiments on different participants verify the effectiveness of the control scheme. Results show that a compliant human–robot interaction is achieved by learning the human–environment interaction parameters, i.e., the human ankle parameters. It indicates that our proposed method can facilitate exosuit control to achieve compliant robot–human–environment interaction.

This article presents a framework to enable a human and robot to perform collaborative tasks safely and efficiently. It consists of three functions. First, human motion is predicted by utilizing Gaussian mixture regression. Second, the motion planning of an online robot is performed such that the robot can appropriately react to a human coworker while executing a task. In our proposed framework, the predicted human motion is transferred to a virtual force acting on a robot's end effector. Its initial trajectory is modified so as to avoid any collisions with the human. To obtain a smooth, collision-free, and energy-minimized trajectory, a constrained optimization problem is formulated. A neural dynamics optimization algorithm is then adopted to solve it. Third, an adaptive fuzzy controller is proposed to track the robot's desired trajectory with uncertain dynamics parameters. We provide the rigorous proof of stability for the proposed methods. The physical experiments are conducted to demonstrate the effectiveness of the proposed collaborative strategy.

The existing hand gesture classification research based on surface electromyogram (sEMG) faces the challenges of low classification accuracy, weak real-time ability, weak robustness, few categories, and lack of explainability. In this article, we investigate how to classify sEMG signals for grasp recognition and human–robot interaction to consider these issues. A novel interval type-2 (IT2) fuzzy classifier based on explainable neural network is proposed for sEMG gesture recognition. Based on fully connected neural network, the adaptive moment estimation is applied to tune the antecedent parameters. The Ninapro data is adopted to test the performance of the proposed model, which realizes recognition of 52 gestures and achieves 95.04% categorization accuracy. Moreover, grasping experiments are conducted on computer, communication, and consumer electronics (3C) experiment platform to test the ability of the classifier in real scenarios. The experiment recognizes six gestures. The results of the 3C grasping experiment show that the proposed method achieves 99.4% offline training accuracy as well as 96.07% online test accuracy. Meanwhile, 89.4% of the classification results can be obtained within 0.5 s. The overall results demonstrate great potential for real-world applications, such as human intent detection and manipulator control.

Forecasting the future environment is an essential and fundamental capability of autonomous driving systems. In contrast to the widely studied video prediction, only a few literatures have explored LiDAR point cloud prediction. To achieve future point cloud generation, most existing methods are based on free-form 3D scene flow prediction. However, these simple scene flow prediction-based methods may cause distortions since the motions of 3D scenes can be seen as a combination of rigid (static background) and flow (dynamic foreground) motions. To address this issue, we propose a simple but effective rigid constrained point cloud prediction network named RPP-Net. The key component of the proposed method is the hybrid motion decoder, which generates motion masks and combines flow motions and rigid motions to produce hybrid motions without additional annotations and sensors. Besides, to reduce running time, we provide a hierarchical cell, which uses a feature encoder to extract deep features and an RPP-RNN module to capture temporal correlations across frames. To evaluate the effectiveness of our RPP-Net, we have conducted extensive experiments on both KITTI dataset and Argoverse dataset and the results show that our RPP-Net significantly outperforms existing methods and achieves a new state-of-the-art.

When a mobile robot is required to perform tasks in the unknown and complex environment, it is critical to have the ability of dynamic obstacle avoidance. However, conventional deep deterministic policy gradient (DDPG) for collision-free navigation can only perceive a fixed number of dynamic obstacles, and thus it cannot adapt to the stochastic working scenario. To overcome the limitation, an improved DDPG algorithm is proposed in this study. It is an exploration to implement the DDPG with long short-term memory (LSTM) network-based encoder to achieve dynamic obstacle avoidance for the mobile robot in the stochastic working scenario, which can encode the variable number of obstacles into a fixed-length representation. Specifically, to facilitate the LSTM network-based encoder, one safe processing rule is designed to guarantee the entire information of the observable obstacles can be represented completely. The LSTM network-based encoder takes the latest environment information of observable obstacles by employing the safe processing rule and generates the fixed length state vector. In addition, continuous state space for mobile robot and obstacles, as well as reward function and action space are designed. Both simulations and experiments are conducted and the results verify that the improved DDPG algorithm can achieve collision-free trajectory with multiple dynamic obstacles well. It helps to reduce the path distance and motion time effectively.

Robots for high-speed manipulation require to produce motions beyond the performance limitations set by the traditional approaches. Recent results integrate the properties associated with dynamic coupling driving and structural mechanics to compute optimal smooth arm motions; however, when accelerating the convergence speed of potential solutions, those approaches cannot avoid premature convergence. In this article, we propose an autonomous motion planning method at the torque level for a class of robots considering multiple conflicting performance metrics. Specifically, we focus on the hyper dynamic manipulation of a golf swing robot using a knee-guided multiobjective optimization algorithm. Compared with traditional planning methods in position or velocity level, it can study motor performance limitations with strong nonlinear dynamic coupling beyond the motion limits designed by the manufacturers. First, the robot’s joint torque is approximated by the B-spline method using the solution at each iteration. Then, we transform the motion planning problem into a multiobjective optimization problem with soft constraints of torque limits and hard constraints of joint stops, and develop a knee-guided evolutionary algorithm to find the optimization solution with the quality tradeoffs between the scale of parameters and metrics. Finally, we conduct the simulation to demonstrate the dynamics performance of the golf swing robot. The results indicate that our approach can generate superior dynamics performance beyond limits with low energy consumption and high precision.

In order to enable intelligent robots to recognize unknown objects as accurately as human beings, object perception research is of great significance in service and industrial robot application scenarios. However, object perception using spectral measurements under few-shot learning usually leads to a poor result because of inadequate training samples. To overcome this problem, this work proposes a novel few-shot learning with coupled dictionary learning (FSL-CDL) framework. First, a hybrid feature fusion method is developed to extract the multiple dimension-reduced features of original spectral measurements to build the hybrid features. Then, based on the hybrid features, a multitask coupled learning method is developed to effectively recognize unknown objects under few-shot learning. In this method, two coupling patterns, i.e., interspectroscopy coupling and intraspectroscopy coupling, effectively bridge the gap between two spectral measurements. Finally, the proposed FSL-CDL is compared with other advanced algorithms on the SMM50 dataset, and reaches 97.5% and 98.4% recognition accuracy under one-shot and five-shot learning, respectively, which are better than other algorithms. Besides, FSL-CDL can be extended to other perception tasks which contains multiple heterogeneous measurements.

Stereo matching is an indispensable function that enables machine vision system to obtain depth information of its environment. However, most of existing algorithms rely on conventional camera, which follows the frame-based scheme and has several shortcomings: low dynamic range, low temporal resolution and high power consumption. To address these issues, we propose two novel patch-based stereo matching methods that exploit the output from a pair of neuromorphic vision sensors. Compared to frame-based camera, neuromorphic vision sensor has independent pixels that generates events at the time intensity changes occur. Based on this unique output, we first construct event representations and present a novel encoding method, which integrates with attention mechanism to encode rich spatial-temporal information of event streams. Then, we design efficient and accuracy networks and propose corresponding loss to train them, which are used to extract event-based descriptors from representations. Finally, the disparity maps are calculated based on local features and refined by two simple smoothing methods. Extensive experiments on the Multi Vehicle Stereo Event Camera Dataset demonstrate the effectiveness of our methods.

Falling down is a serious problem for health and has become one of the major etiologies of accidental death for the elderly living alone. In recent years, many efforts have been paid to fall recognition based on wearable sensors or standard vision sensors. However, the prior methods have the risk of privacy leaks, and almost all these methods are based on video clips, which cannot localize where the falls occurred in long videos. For these reasons, in this article, the bioinspired vision sensor-based falls temporal localization framework is proposed. The bioinspired vision sensors, such as dynamic and active-pixel vision sensor (DAVIS) camera applied in this work responds to pixels' brightness change, and each pixel works independently and asynchronously compared to the standard vision sensors. This property makes it have a very high dynamic range and privacy preserving. First, to better represent event data, compared with the typical constant temporal window mechanism, an adaptive temporal window conversion mechanism is developed. The temporal localization framework follows a proven proposal and classification paradigm. Second, for the high-efficient and recall proposal generation, different from the traditional sliding window scheme, the event temporal density as the actionness score is set and the 1D-watershed algorithm to generate proposals is applied. In addition, we combine the temporal and spatial attention mechanism with our feature extraction network to temporally model the falls. Finally, to evaluate the performance of our framework, 30 volunteers are recruited to join the simulated fall experiments. According to the results of experiments, our framework can realize precise falls temporal localization and achieve the state-of-the-art performance.

The exoskeleton is mainly used by subjects who suffer muscle injury to enhance motor ability in the daily life environment. Previous research seldom considers extending human collaboration skills to human-robot collaborations. In this article, two models, that is: 1) the following the better model and 2) the interpersonal goal integration model, are designed to facilitate the human-human collaborative manipulation in tracking a moving target. Integrated with dual-arm exoskeletons, these two models can enable the robot to successfully perform target tracking with two human partners. Specifically, the manipulation workspace of the human-exoskeleton system is divided into a human region and a robot region. In the human region, the human acts as the leader during cooperation, while, in the robot region, the robot takes the leading role. A novel region-based Barrier Lyapunov function (BLF) is then designed to handle the change of leader roles between the human and the robot and ensures the operation within the constrained human and robot regions when driving the dual-arm exoskeleton to track the moving target. The designed adaptive controller ensures the convergence of tracking errors in the presence of region switches. Experiments are performed on the dual-arm robotic exoskeleton for the subject with muscle damage or some degree of motor dysfunctions to evaluate the proposed controller in tracking a moving target, and the experimental results demonstrate the effectiveness of the developed control.

Robotic grasping ability lags far behind human skills and poses a significant challenge in the robotics research area. According to the grasping part of an object, humans can select the appropriate grasping postures of their fingers. When humans grasp the same part of an object, different poses of the palm will cause them to select different grasping postures. Inspired by these human skills, in this article, we propose new grasping posture prediction networks (GPPNs) with multiple inputs, which acquire information from the object image and the palm pose of the dexterous hand to predict appropriate grasping postures. The GPPNs are further combined with grasping rectangle detection networks (GRDNs) to construct multilevel convolutional neural networks (ML-CNNs). In this study, a force-closure index was designed to analyze the grasping quality, and force-closure grasping postures were generated in the GraspIt! environment. Depth images of objects were captured in the Gazebo environment to construct the dataset for the GPPNs. Herein, we describe simulation experiments conducted in the GraspIt! environment, and present our study of the influences of the image input and the palm pose input on the GPPNs using a variable-controlling approach. In addition, the ML-CNNs were compared with the existing grasp detection methods. The simulation results verify that the ML-CNNs have a high grasping quality. The grasping experiments were implemented on the Shadow hand platform, and the results show that the ML-CNNs can accurately complete grasping of novel objects with good performance.

The generation of motor adaptation in response to mechanical perturbation during human walking is seldom considered in an exoskeleton system. Reshaping step trajectory over consecutive gait cycles for a walking exoskeleton is investigated in this article. Step adjustment of a walking exoskeleton can adapt to human walking intention by shaping step trajectory. This work develops an admittance adaptive fuzzy control strategy for a walking exoskeleton robot to provide assistance for human lower limb movement. Considering human walking intention and utilizing an admittance model, it shapes a reference trajectory to ensure that the walking exoskeleton follows it according to the human–robot force produced by its wearer. Considering a nonlinear and dynamic model with uncertainties, this work designs an integral-type Lyapunov function controller to track a reference trajectory. A disturbance observer is integrated into the controller design to compensate for uncertain disturbance in order to achieve an effective tracking performance. Finally, this work conducts experiments on two healthy subjects with the proposed method on a walking exoskeleton to validate its effectiveness. The results show that it can be applied to walking exoskeletons to enhance human mobility.

The papers in this special section focus on cyborg intelligence. Well-known scientists and experts have expressed concern that robots may take over the world. More generally, there is a concern that robots could take over human jobs and leave billions of people suffering long-term unemployment. Yet, such concerns ignored the potential of intelligence techniques to enhance the natural capabilities of human beings with in-the-body technologies and so become cyborgs with superior capabilities to robots. Cyborg intelligence is dedicated to improving the natural capabilities of human beings by integrating artificial intelligence (AI) with biological intelligence and in-the-body technologies through tight integrations of machines and biological beings.

This article proposes a novel Stackelberg-game-oriented optimal control approach to address the bounded constraint-following control problem for uncertain mechanical systems. First, the uncertainties (possibly fast time-varying) in the system are assumed to be bounded with an unknown boundary, which lies in a specified fuzzy evidence number. In practical engineering, bounded system performance is always demanded, such as the inequality constraint. A diffeomorphism transformation approach is proposed to transform the constrained system into a restructured one satisfying the bounded constraint. Second, we propose an adaptive robust control oriented by the constraint-following control to render the restructured system to follow the specified constraints accurately with deterministic performance (guaranteeing uniform boundedness and uniform ultimate boundedness). The self-adjusting adaptive law (leakage-type) can compensate for the uncertainties and avoid overcompensation. Third, a Stackelberg-game-oriented optimization approach is proposed to obtain the optimal control parameters based on the fuzzy evidence theory. In the optimization approach, the two control parameters$\sigma$and$\varepsilon$are considered as two players with respective cost functions related to system performance and control cost. Furthermore, the optimization problem is solved by obtaining the Stackelberg strategy, which is proved to exist in analytic form. Ultimately, the permanent magnet synchronous linear motor system simulation is presented to show the design process and the excellent performance of the proposed optimal control scheme.

Point cloud completion is a generation and estimation issue derived from the partial point clouds, which plays a vital role in the applications of 3D computer vision. The progress of deep learning (DL) has impressively improved the capability and robustness of point cloud completion. However, the quality of completed point clouds is still needed to be further enhanced to meet the practical utilization. Therefore, this work aims to conduct a comprehensive survey on various methods, including point-based, view-based, convolution-based, graph-based, generative model-based, transformer-based approaches, etc. And this survey summarizes the comparisons among these methods to provoke further research insights. Besides, this review sums up the commonly used datasets and illustrates the applications of point cloud completion. Eventually, we also discussed possible research trends in this promptly expanding field.

Predicting the future can significantly improve the safety of intelligent vehicles, which is a key component in autonomous driving. 3D point clouds can accurately model 3D information of surrounding environment and are crucial for intelligent vehicles to perceive the scene. Therefore, prediction of 3D point clouds has great significance for intelligent vehicles, which can be utilized for numerous further applications. However, due to point clouds are unordered and unstructured, point cloud prediction is challenging and has not been deeply explored in current literature. In this paper, we propose a novel motion-based neural network named MoNet. The key idea of the proposed MoNet is to integrate motion features between two consecutive point clouds into the prediction pipeline. The introduction of motion features enables the model to more accurately capture the variations of motion information across frames and thus make better predictions for future motion. In addition, content features are introduced to model the spatial content of individual point clouds. A recurrent neural network named MotionRNN is proposed to capture the temporal correlations of both features. Moreover, an attention-based motion align module is proposed to address the problem of missing motion features in the inference pipeline. Extensive experiments on two large-scale outdoor LiDAR point cloud datasets demonstrate the performance of the proposed MoNet. Moreover, we perform experiments on applications using the predicted point clouds and the results indicate the great application potential of the proposed method.

Many previous works of soft wearable exoskeletons (exosuit) target at improving the human locomotion assistance, without considering the impedance adaption to interact with the unpredictable dynamics and external environment, preferably outside the laboratory environments. This article proposes a novel hierarchical human-in-the-loop paradigm that aims to produce suitable assistance powers for cable-driven lower limb exosuits to aid the ankle joint in pushing off the ground. It includes two primary loop layers: impedance learning in the external loop and human-in-the-loop adaptive management in the inner loop. Considering unknown terrains, its impedance model can be transferred to a quadratic programming problem with specified constraints, which a designed primal-dual optimization prototype then solves. Then, the presented impedance learning strategy is introduced to regulate the impedance model with the adaptive assistant powers for humans on different terrains. An adaptive controller is designed in the inner loop to balance the nonlinearities and compliance existing in the human-exosuit coexistence, while the robust mechanism compensates for disturbances to facilitate trajectory management without employing the general regressor. The advantage of the proposed technique over conventional solutions with fixed impedance parameters is that it can improve human walking performance over different terrains. Experiments demonstrate the significance of the approach.

Although great progress has been made in generic object detection by advanced deep learning techniques, detecting small objects from images is still a difficult and challenging problem in the field of computer vision due to the limited size, less appearance, and geometry cues, and the lack of large-scale datasets of small targets. Improving the performance of small object detection has a wider significance in many real-world applications, such as self-driving cars, unmanned aerial vehicles, and robotics. In this article, the first-ever survey of recent studies in deep learning-based small object detection is presented. Our review begins with a brief introduction of the four pillars for small object detection, includingmultiscale representation, contextual information, super-resolution, and region-proposal. Then, the collection of state-of-the-art datasets for small object detection is listed. The performance of different methods on these datasets is reported later. Moreover, the state-of-the-art small object detection networks are investigated along with a special focus on the differences and modifications to improve the detection performance comparing to generic object detection architectures. Finally, several promising directions and tasks for future work in small object detection are provided. Researchers can track up-to-date studies on this webpage available at:https://github.com/tjtum-chenlab/SmallObjectDetectionList.

The upper-limb prosthesis has been extensively studied using electromyography (EMG) signals to overcome the physical and functional deficiencies of amputees in recent years. However, most studies focus on the discrete classification of gestures and ignore the interconnection between the classification results and the neuroprosthesis control interface, which plays a vital role in system development. In this article, a new continuous control scheme is proposed to achieve an effective control of the developed upper-limb prosthesis. It utilizes eight channels of EMG signals of the human upper limb to model and control the developed prosthesis. A continuous control scheme is proposed that combines the state of the system and the decoding results to dynamically produce the expected angular velocity of the joint based on the results of the classification. Finally, experiments are performed to demonstrate the effectiveness of the proposed algorithm using an upper-limb neuroprosthesis, achieving the reach and grasp tasks. The results showed that it improves performance with a regular angular velocity of the joint, which underlines the importance of an adequate control scheme for the EMG-guided prosthesis.

Achieving high-precision position control while maintaining positive cable tensions is the most challenging issue for the motion control of cable-driven parallel robots, which should be considered significantly. Different from the existing control schemes with online tension distribution that needs real-time computing in each control cycle, a novel dual-loop dynamic control scheme is proposed in this article, where a paralleled dual-loop tracking strategy is introduced to provide a more compatible scheme, which consists of two tracking loops: 1) the tension control loop and 2) the position control loop. In the former loop, the offline tension distribution is adopted to avoid cable hanging loosely and the real-time feasibility of the distribution method is no longer a necessary demand. In the latter loop, due to the complex dynamics characterized by the cable-driven form, the cooperative motion relation among multiple cables and inevitable external disturbances are investigated comprehensively, and the robust synchronization method is included to guarantee the high-precision position control. Afterward, the Lyapunov method is adopted to analyze the strict stability of the whole closed-loop system with both the position and tension control feedback. The experiments indicate that by synthesizing the two control loops, the proposed scheme can dramatically reduce the tracking errors in the trajectory tracking while avoiding the cable relaxation, and particularly, has a satisfactory control effect when the velocity and acceleration of the trajectory have significant oscillations. Additionally, the strong disturbance rejection ability is also validated via robustness experiments.

LiDAR point cloud streams are usually sparse in time dimension, which is limited by hardware performance. Generally, the frame rates of mechanical LiDAR sensors are 10 to 20 Hz, which is much lower than other commonly used sensors like cameras. To overcome the temporal limitations of LiDAR sensors, a novel task named Point Cloud Frame Interpolation is studied in this paper. Given two consecutive point cloud frames, Point Cloud Frame Interpolation aims to generate intermediate frame(s) between them. To achieve that, we propose a novel framework, namely Point Cloud Frame Interpolation Network (PointINet). Based on the proposed method, the low frame rate point cloud streams can be upsampled to higher frame rates. We start by estimating bi-directional 3D scene flow between the two point clouds and then warp them to the given time step based on the 3D scene flow. To fuse the two warped frames and generate intermediate point cloud(s), we propose a novel learning-based points fusion module, which simultaneously takes two warped point clouds into consideration. We design both quantitative and qualitative experiments to evaluate the performance of the point cloud frame interpolation method and extensive experiments on two large scale outdoor LiDAR datasets demonstrate the effectiveness of the proposed PointINet. Our code is available at https://github.com/ispc-lab/PointINet.git.

This paper proposes an efficient, fully convolutional neural network to generate robotic grasps by using 300×300 depth images as input. Specifically, a residual squeeze-and-excitation network (RSEN) is introduced for deep feature extraction. Following the RSEN block, a multi-scale spatial pyramid module (MSSPM) is developed to obtain multi-scale contextual information. The outputs of each RSEN block and MSSPM are combined as inputs for hierarchical feature fusion. Then, the fused global features are upsampled to perform pixel-wise learning for grasping pose estimation. The experimental results on Cornell and Jacquard grasping datasets indicate that the proposed method has a fast inference speed of 5ms while achieving high grasp detection accuracy of 96.4% and 94.8% on Cornell and Jacquard, respectively, which strikes a balance between accuracy and running speed. Our method also gets a 90% physical grasp success rate with a UR5 robot arm.

Under-actuated bionic hands have achieved tremendous popularity in many fields because of their advantages of lightweight, budget-friendly, satisfactory flexibility, and adaptability. Except for the bionic mechanical design, various anthropomorphic control strategies have been proposed and investigated in the last decades. However, due to its under-actuated characteristic, there are still many challenges for anthropomorphic control of all the degrees of freedom (DOFs) using less input. It is challenging to map the human hand kinematic synergies on robotic hands, particularly for a dynamic environment. Therefore, it is worth studying how to control the under-actuated bionic hand effectively in a dynamic environment. In this paper, an anthropomorphic control method is proposed using sensor fusion of hand kinematic inputs to control the under-actuated bionic hand. In order to map the kinematics of human fingers to the bionic hand, a novel finger bending angle is defined to represent the posture of human fingers. Multiple Leap Motion Controllers (LMC) are fused to estimate the stable and accurate finger bending angles to avoid the occlusion problem. Finally, experiments with real-time control of the under-actuated bionic hand are implemented to demonstrate the proposed approach’s effectiveness.

The hand gesture recognition system is a noncontact and intuitive communication approach, which, in turn, allows for natural and efficient interaction. This work focuses on developing a novel and robust gesture recognition system, which is insensitive to environmental illumination and background variation. In the field of gesture recognition, standard vision sensors, such as CMOS cameras, are widely used as the sensing devices in state-of-the-art hand gesture recognition systems. However, such cameras depend on environmental constraints, such as lighting variability and the cluttered background, which significantly deteriorates their performances. In this work, we propose an event-based gesture recognition system to overcome the detriment constraints and enhance the robustness of the recognition performance. Our system relies on a biologically inspired neuromorphic vision sensor that has microsecond temporal resolution, high dynamic range, and low latency. The sensor output is a sequence of asynchronous events instead of discrete frames. To interpret the visual data, we utilize a wearable glove as an interaction device with five high-frequency (>100 Hz) active LED markers (ALMs), representing fingers and palm, which are tracked precisely in the temporal domain using a restricted spatiotemporal particle filter algorithm. The latency of the sensing pipeline is negligible compared with the dynamics of the environment as the sensor's temporal resolution allows us to distinguish high frequencies precisely. We design an encoding process to extract features and adopt a lightweight network to classify the hand gestures. The recognition accuracy of our system is comparable to the state-of-the-art methods. To study the robustness of the system, experiments considering illumination and background variations are performed, and the results show that our system is more robust than the state-of-the-art deep learning-based gesture recognition systems. Note to Practitioners-This article addresses the robustness of the hand gesture recognition system that is important for gesture recognition-based applications. Existing methods rely on either the large-volume data to train a deep learning model or to restrict the applied environments (e.g., an ideal environment without dynamic background). However, a vision-based deep learning model requires large computational resources, while the ideal environment limits the practicality of the system. In this work, we introduce a biologically inspired neuromorphic vision sensor and an ALM glove and build a novel gesture recognition system to tackle the above issue. The neuromorphic vision sensor has a microsecond temporal resolution and a high dynamic range. With these properties, the sensing system of our prototype operates in a very low-latency space, which, in turn, ensures that our gesture recognition system is robust to illumination variance and dynamic background. Thus, this work is valuable to the research of illumination-robust gesture recognition systems. Preliminary experiments suggest that our system prototype is feasible, but it has not yet been incorporated into an online gesture recognition system nor tested with complex gestures. In future work, we will concentrate on the improvement of the signal processing methods that advance the current system to complex and practical applications.

More natural and intuitive control is expected to maximize the auxiliary effect of the powered prosthetic leg for lower limb amputees. In order to realize the stable and flexible walking of prosthetic legs in different terrains according to human intention, a brain-computer interface (BCI) based on motor imagery (MI) is developed. For the raw electroencephalogram (EEG) signals, discrete wavelet transform (DWT) is utilized to extract the time-frequency domain features, which are used as the input signals of the common spatial pattern (CSP) to obtain the time-frequency-space domain features of EEG signals. Then, a support vector machine (SVM) classifier and a directed acyclic graph (DAG) structure are combined to classify multiclass imaginary tasks. According to the result of human intention recognition, the prosthetic leg performs the corresponding gait trajectory generated by coding the ground reaction force (GRF). In addition, a sensory feedback loop is established by functional electrical stimulation (FES), which feeds back the movement of the prosthetic leg to human in real time. The effectiveness and feasibility of the developed EEG-based volitional control of powered prosthetic legs have been validated by three subjects, all of whom were able to fulfill smoothly walking on the floor, ascending stairs, and descending stairs according to their own intentions using prosthetic legs.

To achieve high-accuracy grasping of unknown objects, we present novel multilevel convolutional neural networks (CNNs) for robotic grasping with a parallel gripper or multifingered dexterous hand. The multilevel CNNs include four levels with different structures and functions. The first level is constructed to get the approximate position of the grasped object. The second level aims to obtain the preselected grasping rectangles. The third level is constructed to re-evaluate the preselected grasping rectangles and obtain substantially detailed features with quite a large network, so as to assess each preselected grasping rectangle exactly. By using a selection algorithm, the optimal grasping rectangle can be determined and unknown object grasping can be achieved with a parallel gripper. The purpose of the fourth level is to obtain the finger position distribution to complete the accurate grasping of unknown objects with a multifingered dexterous hand. The test results indicate that, compared to state-of-the-art methods, the proposed multilevel CNNs can greatly increase the precision of the grasping rectangle. Grasping experiments were implemented on a Youbot arm with five degrees of freedom and a Shadow four-fingered dexterous hand. The results show that the multilevel CNNs can determine the optimal grasping rectangle and finger position distribution, thereby achieving high-accuracy grasping of various unknown objects, even under several complex environmental conditions.Note to Practitioners—Robot grasping of objects lags far behind human experiences and poses a significant challenge in the robotics area. To solve it, we present new multilevel convolutional neural networks (CNNs) to process red green blue-depth (RGB-D) images and realize optimal grasping detection of unknown objects. Moreover, we provide details of the network structure, network training, and network testing. The testing results obtained from the open grasping data set show that the multilevel CNNs can significantly increase the accuracy of the grasping rectangle compared to state-of-the-art methods. Experiments were implemented on different robotic platforms, including a five-degrees-of-freedom Youbot arm with a parallel gripper and a UR5 robot arm with a Shadow multifingered dexterous hand. The results validate that the multilevel CNNs offer excellent generalization and robustness for handling different sizes and shapes of unknown objects, as well as background disturbances, which are key problems in robotic manipulation.

In recent decades, there has been an increasing interest in the use of robotic powered exoskeletons to assist patients with movement disorders in rehabilitation and daily life. Providing assistive torque that compensates for the user's remaining muscle contributions is a growing and challenging field within exoskeleton control. In this article, ankle joint torques were estimated using electromyography (EMG)-driven neuromusculoskeletal (NMS) model and an artificial neural network (ANN) model in seven movement tasks, including fast walking, slow walking, self-selected speed walking, and isokinetic dorsi/plantar flexion at 60°/s and 90°/s. In each method, EMG signals and ankle joint angles were used as input, the models were trained with data from 3-D motion analysis, and ankle joint torques were predicted. Six cases using different motion trials as calibration (for the NMS model)/training (for the ANN) were devised, and the agreement between the predicted and measured ankle joint torques was computed. We found that the NMS model could overall better predict ankle joint torques from EMG and angle data than the ANN model with some exceptions; the ANN predicted ankle joint torques with better agreement when trained with data from the same movement. The NMS model predicted ankle joint torque best when calibrated with trials during which EMG reached maximum levels, whereas the ANN predicted well when trained with many trials and types of movements. In addition, the ANN prediction may become less reliable when predicting unseen movements. Detailed comparative studies of methods to predict ankle joint torque are crucial for determining strategies for exoskeleton control.

Singularity issue has long been a concern of the task-space control design for Euler-Lagrange systems. In classical task-space controls, robots are often assumed to operate in the task space, where singularities do not exist. Such an assumption limits their potential applications in various workspaces. To address the potential singularity issue associated with Euler-Lagrange systems, this article proposes an adaptive fuzzy-region-based control for Euler-Lagrange systems with kinematically singular configurations. Singular regions are described by the potential energy function. The proposed controller includes a joint-space control, which is active when the system approaches singular regions, and a task-space control, which is used to track the desired trajectory. Therefore, the system can smoothly transit from singular regions to nonsingular regions or can achieve singularity avoidance during the tracking task. In order to achieve singularity avoidance while reducing control effort, the coefficients of the potential energy function are adjusted dynamically based on the designed fuzzy system. Rigorous analysis shows that singularity issues can be properly handled, and the asymptotic stability of the system is ensured. Experiments are conducted to demonstrate the effectiveness of the proposed controller.

Cable-driven parallel robots (CDPRs) are a new type of parallel robots that use cables to control a mobile platform. They possess several advantages, including large workspace, low inertia, and high payload capacity. However, there are several problems in the high-precision trajectory tracking control of CDPRs. On the one hand, all the cables must remain in tension during the entire motion process. On the other hand, the controller design is subjected to model uncertainties and external disturbances. Accordingly, this article proposes a robust synchronization control (RSC) scheme in the cable length space to achieve high-precision trajectory tracking. The synchronization control ensures motion coordination among all the cables and prevents cable relaxation, whereas the robust control eliminates modeling errors and restrains external disturbances. The uniformly ultimate boundedness of the tracking and synchronization errors in the closed-loop system equation was proved using the Lyapunov theory. Simulations and experiments of the trajectory tracking control were both implemented on a three-degree-of-freedom CDPR. Compared with the adaptive robust control scheme and the augmented proportional derivative scheme on the premise of the approximate energy consumption, the proposed RSC scheme could reduce not only the tracking errors of the cables but also the synchronization errors between adjacent cables. Moreover, the RSC scheme could significantly improve the trajectory tracking accuracy of the mobile platform. The robustness of this scheme was verified using load experiments and torque-disturbance experiments.

Urban vehicle emission prediction can help the regulation of vehicle pollution and traffic control. However, it is hard to predict the spatiotemporal variation of vehicle emission because of the spatial interactions and temporal correlations between different road segments as well as the high nonlinearity and complexity of vehicle emission variation. The existing methods solve the problem by splitting the region into standard segments or grids based on conventional deep learning methods, without considering that urban vehicle emission varies by graph-structured traffic road network and depends on many complex external environment factors. To address these issues, a spatiotemporal graph convolution multifusion network (ST-MFGCN) is proposed to leverage the graph structural properties as the inherent connectivity of road network for urban vehicle emission prediction, which can capture the vehicle emission spatiotemporal variation patterns and learn the effects of complex environmental factors. The proposed model consists of three parts: 1) a spatiotemporal graph convolution module to capture spatiotemporal dependencies by merging closeness, period, and trend sequences with temporal convolution as well as graph convolution is introduced to model the spatial dependencies; 2) an external factor component to divide multisource external factors into global and individual external features; and 3) a general fusion component to merge the spatiotemporal patterns and the external features as well as fit the mutation of emission measurement data by multifusion strategy. Finally, the proposed model is evaluated on the practical monitoring data of vehicle emission data in Hefei, and the results demonstrate that our proposed model can predict regional vehicle emissions effectively.

This article discusses the control design and experiment validation of a flexible two-link manipulator (FTLM) system represented by ordinary differential equations (ODEs). A reinforcement learning (RL) control strategy is developed that is based on actor–critic structure to enable vibration suppression while retaining trajectory tracking. Subsequently, the closed-loop system with the proposed RL control algorithm is proved to be semi-global uniform ultimate bounded (SGUUB) by Lyapunov’s direct method. In the simulations, the control approach presented has been tested on the discretized ODE dynamic model and the analytical claims have been justified under the existence of uncertainty. Eventually, a series of experiments in a Quanser laboratory platform are investigated to demonstrate the effectiveness of the presented control and its application effect is compared with PD control.

In this paper, an adaptive neural bounded control scheme is proposed for an ${n}$ -link rigid robotic manipulator with unknown dynamics. With the combination of the neural approximation and backstepping technique, an adaptive neural network control policy is developed to guarantee the tracking performance of the robot. Different from the existing results, the bounds of the designed controller are known a priori, and they are determined by controller gains, making them applicable within actuator limitations. Furthermore, the designed controller is also able to compensate the effect of unknown robotic dynamics. Via the Lyapunov stability theory, it can be proved that all the signals are uniformly ultimately bounded. Simulations are carried out to verify the effectiveness of the proposed scheme.

This article addresses the visual regulation problem of a differential-drive mobile robot with an arbitrarily installed monocular camera in the indoor environment. A three-stage controller is designed by using a novel nonadaptive switching approach, where the unknown image depth and the uncalibrated camera-to-robot translation parameters do not need to be estimated. Convergence of the error systems with the designed controller in each stage is analyzed. Moreover, the existence of the switching time instants from each stage is proved. The simulation results are presented to show the effectiveness of the proposed approach.

In this paper, a force sensorless control scheme based on neural networks (NNs) is developed for interaction between robot manipulators and human arms in physical collision. In this scheme, the trajectory is generated by using geometry vector method with Kinect sensor. To comply with the external torque from the environment, this paper presents a sensorless admittance control approach in joint space based on an observer approach, which is used to estimate external torques applied by the operator. To deal with the tracking problem of the uncertain manipulator, an adaptive controller combined with the radial basis function NN (RBFNN) is designed. The RBFNN is used to compensate for uncertainties in the system. In order to achieve the prescribed tracking precision, an error transformation algorithm is integrated into the controller. The Lyapunov functions are used to analyze the stability of the control system. The experiments on the Baxter robot are carried out to demonstrate the effectiveness and correctness of the proposed control scheme.

Service robot navigation must take the humans into account explicitly so as to produce motion behaviors that reflect its social awareness. Generally, the navigation problems of mobile service robot can be summarized to three aspects: 1) human detection; 2) robot real-time localization; and 3) robot motion planning. The purpose of this paper is to provide a feasible strategy to integrate these three aspects to achieve a conscious, safe, accurate, robust, and efficient navigation. We first introduce the human detection system for recognition of human gesture using a weighted dynamic time warping (DTW) with kinematic constraints. Thus, by interpreting the human body language through gesture recognition, robot motion behaviors like heading to the assigned position or following people can be activated. Then, for the robot localization, a simultaneous localization and mapping (SLAM) method based on artificial and natural landmark recognition is employed to provide absolute position feedback in real time. For the motion planning, a novel quadrupole potential field (QPF) method is proposed to plan collision-free trajectories, adequately considering the nonholomic constraint of the mobile robot system. Then, a robust kinematic controller is designed for trajectory tracking to account for slip disturbances. Such a design automatically merges path finding, trajectory generation, and trajectory tracking in a closed-loop fashion, achieving simultaneous motion planning for obstacle avoidance and feedback stabilization to a desired position and orientation even in the presence of slippage. Finally, experiments prove the effectiveness and feasibility of the proposed strategy, showing a good navigation performance on mobile service robot.

The complexity of surgical operation can be released significantly if surgical robots can learn the manipulation skills by imitation from complex tasks demonstrations such as puncture, suturing, and knotting, etc.. This paper proposes a reinforcement learning algorithm based manipulation skill transferring technique for robot-assisted Minimally Invasive Surgery by Teaching by Demonstration. It employed Gaussian mixture model and Gaussian mixture Regression based dynamic movement primitive to model the high-dimensional human-like manipulation skill after multiple demonstrations. Furthermore, this approach fascinates the learning and trial phase performed offline, which reduces the risks and cost for the practical surgical operation. Finally, it is demonstrated by transferring manipulation skills for reaching and puncture using a KUKA LWR4+ robot in a lab setup environment. The results show the effectiveness of the proposed approach for modelling and learning of human manipulation skill.

In this paper, an Internet of Things-based human-robot collaborative control scheme is developed in Robot-assisted Minimally Invasive Surgery scenario. A hierarchical operational space formulation is designed to exploit the redundancies of the 7-DoFs redundant manipulator to handle multiple operational tasks based on their priority levels, such as guaranteeing a remote center of motion constraint and avoiding collision with a swivel motion without influencing the undergoing surgical operation. Furthermore, the concept of the Internet of Robotic Things is exploited to facilitate the best action of the robot in human-robot interaction. Instead of utilizing compliant swivel motion, HTC VIVE PRO controllers, used as the Internet of Things technology, is adopted to detect the collision. A virtual force is applied to the robot elbow, enabling a smooth swivel motion for human-robot interaction. The effectiveness of the proposed strategy is validated using experiments performed on a patient phantom in a lab setup environment, with a KUKA LWR4+ slave robot and a SIGMA 7 master manipulator. By comparison with previous works, the results show improved performances in terms of the accuracy of the RCM constraint and surgical tip.

In this paper, a bilateral teleoperation control of a serial robot manipulator, which guarantees a Remote Center of Motion (RCM) constraint in its kinematic level, is developed. A two-layered approach based on the energy tank model is proposed to achieve haptic feedback on the end effector with a pedal switch. The redundancy of the manipulator is exploited to maintain the RCM constraint using the decoupled Cartesian Admittance Control. Transparency and stability of the proposed bilateral teleoperation are demonstrated using a KUKA LWR4+ serial robot and a Sigma 7 haptic manipulator with an RCM constraint in augmented reality. The results prove that the control can achieve not only the bilateral teleoperation but also maintain the RCM constraint.

In this article, an admittance-based controller for physical human-robot interaction (pHRI) is presented to perform the coordinated operation in the constrained task space. An admittance model and a soft saturation function are employed to generate a differentiable reference trajectory to ensure that the end-effector motion of the manipulator complies with the human operation and avoids collision with surroundings. Then, an adaptive neural network (NN) controller involving integral barrier Lyapunov function (IBLF) is designed to deal with tracking issues. Meanwhile, the controller can guarantee the end-effector of the manipulator limited in the constrained task space. A learning method based on the radial basis function NN (RBFNN) is involved in controller design to compensate for the dynamic uncertainties and improve tracking performance. The IBLF method is provided to prevent violations of the constrained task space. We prove that all states of the closed-loop system are semiglobally uniformly ultimately bounded (SGUUB) by utilizing the Lyapunov stability principles. At last, the effectiveness of the proposed algorithm is verified on a Baxter robot experiment platform. Note to Practitioners-This work is motivated by the neglect of safety in existing controller design in physical human-robot interaction (pHRI), which exists in industry and services, such as assembly and medical care. It is considerably required in the controller design for rigorously handling constraints. Therefore, in this article, we propose a novel admittance-based human-robot interaction controller. The developed controller has the following functionalities: 1) ensuring reference trajectory remaining in the constrained task space: a differentiable reference trajectory is shaped by the desired admittance model and a soft saturation function; 2) solving uncertainties of robotic dynamics: a learning approach based on radial basis function neural network (RBFNN) is involved in controller design; and 3) ensuring the end-effector of the manipulator remaining in the constrained task space: different from other barrier Lyapunov function (BLF), integral BLF (IBLF) is proposed to constrain system output directly rather than tracking error, which may be more convenient for controller designers. The controller can be potentially applied in many areas. First, it can be used in the rehabilitation robot to avoid injuring the patient by limiting the motion. Second, it can ensure the end-effector of the industrial manipulator in a prescribed task region. In some industrial tasks, dangerous or damageable tools are mounted on the end-effector, and it will hurt humans and bring damage to the robot when the end-effector is out of the prescribed task region. Third, it may bring a new idea to the designed controller for avoiding collisions in pHRI when collisions occur in the prescribed trajectory of end-effector.

Exoskeleton robots can assist humans to perform activities of daily living with little effort. In this paper, a hierarchical control scheme is presented which enables an exoskeleton robot to achieve cooperative manipulation with humans. The control scheme consists of two layers. In low-level control of the upper limb exoskeleton robot, an admittance control scheme with an asymmetric barrier Lyapunov function-based adaptive neural network controller is proposed to enable the robot to be back drivable. In order to achieve high-level interaction, a strategy for learning human skills from demonstration is proposed by utilizing Gaussian mixture models, which consists of the learning and reproduction phase. During the learning phase, the robot observes and learns how a demonstrator performs a specific impedance-based task successfully, and in the reproduction phase, the robot can provide the subjects with just enough assistance by extracting human skills from demonstrations to prevent the motion of the robot end-effector deviating far from desired ones, due to variation in the interaction force caused by environmental disturbances. Experimental results of two different tasks show that the proposed control scheme can provide human subjects with assistance as needed during cooperative manipulation.

For human-robot co-manipulation by robotic exoskeletons, the interaction forces provide a communication channel through which the human and the robot can coordinate their actions. In this article, an optimization approach for reshaping the physical interactive trajectory is presented in the co-manipulation tasks, which combines impedance control to enable the human to adjust both the desired and the actual trajectories of the robot. Different from previous studies, the proposed method significantly reshapes the desired trajectory during physical human-robot interaction (pHRI) based on force feedback, without requiring constant human guidance. The proposed scheme first formulates a quadratically constrained programming problem, which is then solved by neural dynamics optimization to obtain a smooth and minimal-energy trajectory similar to the natural human movement. Then, we propose an adaptive neural-network controller based on the barrier Lyapunov function (BLF), which enables the robot to handle the uncertain dynamics and the joint space constraints directly. To validate the proposed method, we perform experiments on the exoskeleton robot with human operators for co-manipulation tasks. The experimental results demonstrate that the proposed controller could complete the co-manipulation tasks effectively.

In the paper, the development, control, and preliminary evaluation of the human-robot coupled walking exoskeleton for weight-support enhancement are presented, which provides the assistance of abduction/adduction and flexion/extension for the hip joint, and the flexion/extension for knee joint of human legs during walking. The trajectory generation strategy for the walking exoskeleton utilizes the inverted pendulum approximation. Considering human in the control loop, the periodic walking trajectory and uncertainties with known periods, we propose a human-cooperative adaptive fuzzy strategy combing virtual tunnels that allows human subjects to change the movement timing of their legs and produce a physiological path. The strategy does not need human-robot coupled model, and the designed compliant virtual constraints keep the legs of the subject within the constrained tunnel around the expected path. The subjects with the assistance of supporting torques can walk effortlessly along the spatial path. The path control strategy has been verified with two healthy subjects. The recorded kinematic data demonstrated that the participants are able to achieve stable walking with larger spatio-temporal variability.

Recent advances in fine manipulation have led to increased interest in both scientific research works and engineering applications. Robot manipulation at a level approaching human skills is gaining attention in both industrial and individual services. A major challenge in fine manipulation is the unavoidable uncertainties and unpredictable conditions encountered in dynamic and unstructured application environments. The employment of biologically inspired (bioinspired) embodiments in fine manipulation shows significant advantages in tackling such problems. The aim of bioinspired embodiment is to improve fine manipulation of robotic systems utilizing the knowledge gained from natural systems with biomimetic methods. Such a method includes sensing, planning, and execution. This article provides a comprehensive survey of the current state of bioinspired technologies in fine manipulation, and outlines new challenges and some potential directions.

In this paper, the complex problems of internal forces and position control are studied simultaneously and a disturbance observer-based radial basis function neural network (RBFNN) control scheme is proposed to: 1) estimate the unknown parameters accurately; 2) approximate the disturbance experienced by the system due to input saturation; and 3) simultaneously improve the robustness of the system. More specifically, the proposed scheme utilizes disturbance observers, neural network (NN) collaborative control with an adaptive law, and full state feedback. Utilizing Lyapunov stability principles, it is shown that semiglobally uniformly bounded stability is guaranteed for all controlled signals of the closed-loop system. The effectiveness of the proposed controller as predicted by the theoretical analysis is verified by comparative experimental studies.

Motion tracking control design of nonholonomic mobile robot systems considering actuator dynamics is addressed in this paper. A trajectory tracking controller is designed at actuator level, which guarantees that the nonholonomic mobile robot tracks a given trajectory. A numerical example is shown to demonstrate and validate the proposed approach in this paper.

Humans can adapt to complex environments by voluntarily adjusting the impedance parameters and interaction force. Traditional robots perform tasks independently without considering their interactions with the external environment, which leads to poor flexibility and adaptability. Comparatively, humans can adapt to complex environments by voluntarily adjusting the impedance parameters and interaction force. In order to solve the problems of human-robot security and adaptability to unknown environment, a human-inspired control with force and impedance adaptation is proposed to interact with unknown environments and exhibit this biological behavior on the developed dual-arm exoskeleton robots. First, we propose a computationally model utilizing the sampled surface electromyogram (sEMG) signals to calculate the human arm endpoint stiffness and define a co-contraction index to describe the dynamic behaviors of the muscular activities in the tasks. Then, the obtained human limb impedance stiffness parameters and the sampling position information are transferred to the slave arm of the exoskeleton as the input variables of the controller in real-time. In addition, a variable stiffness observer is used here to compensate for the errors of the calculated stiffness by sEMG signals. The experimental studies of human impedance transfer control have been conducted to show the effectiveness of the developed approach. Results of the experimental suggest that the proposed controller can achieve human motor adaptation and enable the subjects to execute a skill transfer control by a dual-arm exoskeleton robot.

This paper proposes a brain-robot interface (BRI)-based control strategy in combination with the simultaneous localization and mapping (SLAM) to achieve the navigation and control of a mobile robot in uncertain environments. The BRI is based on steady state visually evoked potentials, utilizing the multivariate synchronization index classification algorithm to analyze the human electroencephalograph (EEG) signals in such a manner that human intentions can be recognized and motion commands can be produced for the brain controlled robot. The entire system is semi-autonomous since the navigation of mobile robot is commanded by the BRI, and the low-level motion of the mobile robot is autonomous with a designed kinematic controller. By utilizing vanishing points and door plates as the environmental features, a global metric map of the environment has been built by a sequential SLAM algorithm. The main contribution of this paper is the combination of an artificial potential field (APF) and the brain signals, which builds up the relationship between the strength of EEG signals and the intensity of the potential field. Through the proposed EEG-APF method, motion commands that would plan an obstacle-free trajectory in un-structured environments, can be obtained. The entire system has been tested with eight volunteer subjects, and all subjects are able to successfully fulfill manipulating mobile robot in the experiments.

The contribution of SiC power devices to the whole power conversion system still lacks comparative analysis verification. This paper experimentally investigates a high-performance All-SiC power module to provide solid evidence for the excellent performance of SiC semiconductor in the high power electrical energy conversion. The All-SiC power module in this paper, featured with high-efficiency and high-power-density, is intended for the power supply rectifier of electrolysis inactivation in the Ballast Water Management System (BWMS). Conventional All-Si power module for BWMS rectifier used low switching frequency Si IGBTs as their power devices, resulting in low power density and limited efficiency. This paper schemes the technical approach based on fast switching SiC MOSFETs, and then experimentally investigates the performance of it. Finally, the comparison between the traditional All-SiC and All-SiC power modules verifies the technical advantages by using SiC MOSFETs as the power devices.

To achieve the better navigation performance of a mobile robot in the unknown environments, a novel human-robot hybrid system incorporating a motor-imagery (MI)-based brain teleoperation control is presented in this paper, where a deep-learning-based active perception is developed in the simultaneous localization and mapping (SLAM) framework. Using the deep-learning-based object recognition in the red-green-blue-depth (RGB-D) data acquisition process, the designed SLAM approach can select the valid feature points effectively, and the speed of displacement tracking can be improved by combining the oriented FAST and rotated BRIEF (ORB) SLAM algorithm with the optical flow method. The global trajectory map can also be mended using graph-based nonlinear error optimization. In addition, to build the connection between human intentions and the robot control commands flexibly in the developed mobile robot, a common spatial pattern (CSP)-based support vector machine (SVM) classification algorithm is proposed so that the control commands can be obtained directly from the human electroencephalograph (EEG) signals, which are preanalyzed and classified using the phenomena of event-related synchronization/desynchronization (ERS/ERD). Experiments involving several operators have verified the effectiveness of the proposed framework in the actual unstructured environments.

In this paper, a hierarchical hybrid motion/force control architecture for the manipulation and grasping of mobile manipulators is presented, where the systems are subject to varieties of physical constraints such as Coulomb friction cones, nonholonomic/holonomic constraints, and actuator saturation limits. The incorporation of a projection-based operation space control and an adaptive controller based on the neural networks used in this paper formulates a novel control scheme, so the system stability is further guaranteed and the uncertain dynamics is handled without redesigning the minimal-order dynamics model. Considering the effects of these constraints, the actuator saturation limits are handled by an auxiliary designed system, and the neural dynamics optimization is applied for the quadratically constrained programing problem of the optimal robotic grasping. The dynamic uncertainties can be estimated online by using the developed motion/force control strategy, and the application of a novel disturbance observer is explored to ensure the good tracking performance. The experimental results are presented to verify the performance and the efficiency of the proposed method.

In this paper, we investigate fuzzy neural network (FNN) control using impedance learning for coordinated multiple constrained robots carrying a common object in the presence of the unknown robotic dynamics and the unknown environment with which the robot comes into contact. First, an FNN learning algorithm is developed to identify the unknown plant model. Second, impedance learning is introduced to regulate the control input in order to improve the environment-robot interaction, and the robot can track the desired trajectory generated by impedance learning. Third, in light of the condition requiring the robot to move in a finite space or to move at a limited velocity in a finite space, the algorithm based on the position constraint and the velocity constraint are proposed, respectively. To guarantee the position constraint and the velocity constraint, an integral barrier Lyapunov function is introduced to avoid the violation of the constraint. According to Lyapunov's stability theory, it can be proved that the tracking errors are uniformly bounded ultimately. At last, some simulation examples are carried out to verify the effectiveness of the designed control.

In this paper, an admittance adaptation method has been developed for robots to interact with unknown environments. The environment to be interacted with is modeled as a linear system. In the presence of the unknown dynamics of environments, an observer in robot joint space is employed to estimate the interaction torque, and admittance control is adopted to regulate the robot behavior at interaction points. An adaptive neural controller using the radial basis function is employed to guarantee trajectory tracking. A cost function that defines the interaction performance of torque regulation and trajectory tracking is minimized by admittance adaptation. To verify the proposed method, simulation studies on a robot manipulator are conducted.

Due to strongly coupled nonlinearities of the grasped dual-arm robot and the internal forces generated by grasped objects, the dual-arm robot control with uncertain kinematics and dynamics raises a challenging problem. In this paper, an adaptive fuzzy control scheme is developed for a dual-arm robot, where an approximate Jacobian matrix is applied to address the uncertain kinematic control, while a decentralized fuzzy logic controller is constructed to compensate for uncertain dynamics of the robotic arms and the manipulated object. Also, a novel finite-time convergence parameter adaptation technique is developed for the estimation of kinematic parameters and fuzzy logic weights, such that the estimation can be guaranteed to converge to small neighborhoods around their ideal values in a finite time. Moreover, a partial persistent excitation property of the Gaussian-membership-based fuzzy basis function was established to relax the conventional persistent excitation condition. This enables a designer to reuse these learned weight values in the future without relearning. Extensive simulation studies have been carried out using a dual-arm robot to illustrate the effectiveness of the proposed approach.

This paper addresses a trajectory-tracking control problem for mobile robots by combining tube-based model predictive control (MPC) in handling kinematic constraints and adaptive control in handling dynamic constraints. In order to handle kinematic constraints, the tube-based MPC scheme is introduced, which includes the state feedback controller to suppress the external disturbance in the velocity level. The tube-based MPC is transformed to a constrained quadratic programming (QP) problem, and then the QP problem can be efficiently solved by a primal-dual neural network over a finite receding horizon so as to obtain the optimal control velocity. Besides, an adaptive controller employing the neural network technology is proposed to acquire the approximation of the uncertain robotic dynamics. Moreover, an auxiliary control is developed in order to deal with actuator saturation, and a disturbance observer is designed to reject the external disturbance online in the dynamic level. Subsequently, through Lyapunov function synthesis, the stability of the closed-loop system have been guaranteed. Finally, in order to verify the effectiveness, the experimental studies are carried out using an actual mobile robot.

The papers in this special section focus on robotic manipulation based on bio-inspired computing. It is the goal of this papers to present applications of human manipulation ability in robotic systems, and to outline key strategies for robotic dexterous manipulation in next generation.

This paper addresses the control design for an upper limb exoskeleton in the presence of input saturation. An adaptive controller employing the neural network technology is proposed to approximate the uncertain robotic dynamics. Also, an auxiliary system is designed to deal with the effect of input saturation. Furthermore, we develop both the state feedback and the output feedback control strategies, which effectively estimates the uncertainties online from the measured feedback errors, instead of the model-based control. In addition to the proposed control, a disturbance observer is designed to reject the unknown disturbance online for achieving the trajectory tracking. The method requires a minimal amount of a priori knowledge of system dynamics. Subsequently, the principle of Lyapunov synthesis ensures the stability of the closed-loop system. Finally, the experimental studies are carried out on this robotic exoskeleton.

In this paper, a closed-loop control has been developed for the exoskeleton robot system based on brain-machine interface (BMI). Adaptive controllers in joint space, a redundancy resolution method at the velocity level, and commands that generated from BMI in task space have been integrated effectively to make the robot perform manipulation tasks controlled by human operator's electroencephalogram. By extracting the features from neural activity, the proposed intention decoding algorithm can generate the commands to control the exoskeleton robot. To achieve optimal motion, a redundancy resolution at the velocity level has been implemented through neural dynamics optimization. Considering human-robot interaction force as well as coupled dynamics during the exoskeleton operation, an adaptive controller with redundancy resolution has been designed to drive the exoskeleton tracking the planned trajectory in human brain and to offer a convenient method of dynamics compensation with minimal knowledge of the dynamics parameters of the exoskeleton robot. Extensive experiments which employed a few subjects have been carried out. In the experiments, subjects successfully fulfilled the given manipulation tasks with convergence of tracking errors, which verified that the proposed brain-controlled exoskeleton robot system is effective.

This paper proposes an enhanced robot skill learning system considering both motion generation and trajectory tracking. During robot learning demonstrations, dynamic movement primitives (DMPs) are used to model robotic motion. Each DMP consists of a set of dynamic systems that enhances the stability of the generated motion toward the goal. A Gaussian mixture model and Gaussian mixture regression are integrated to improve the learning performance of the DMP, such that more features of the skill can be extracted from multiple demonstrations. The motion generated from the learned model can be scaled in space and time. Besides, a neural-network-based controller is designed for the robot to track the trajectories generated from the motion model. In this controller, a radial basis function neural network is used to compensate for the effect caused by the dynamic environments. The experiments have been performed using a Baxter robot and the results have confirmed the validity of the proposed methods.

This paper has developed a coordination control method for a dual-arm exoskeleton robot based on human impedance transfer skills, where the left (master) robot arm extracts the human limb impedance stiffness and position profiles, and then transfers the information to the right (slave) arm of the exoskeleton. A computationally efficient model of the arm endpoint stiffness behavior is developed and a co-contraction index is defined using muscular activities of a dominant antagonistic muscle pair. A reference command consisting of the stiffness and position profiles of the operator is computed and realized by one robot in real-time. Considering the dynamics uncertainties of the robotic exoskeleton, an adaptive-robust impedance controller in task space is proposed to drive the slave arm tracking the desired trajectories with convergent errors. To verify the robustness of the developed approach, a study of combining adaptive control and human impedance transfer control under the presence of unknown interactive forces is conducted. The experimental results of this paper suggest that the proposed control method enables the subjects to execute a coordination control task on a dual-arm exoskeleton robot by transferring the stiffness from the human arm to the slave robot arm, which turns out to be effective.

In this paper, an active control approach using two piezoelectric actuators (PZTAs) and an adaptive controller is investigated for suppressing the two-DOF regenerative chatter in micromilling. The PZTAs are utilized as active control elements to provide force compensation for chatter suppression. First, the dynamical model of micromilling process is demonstrated. Then, an adaptive controller is developed by employing neural networks to approximate the unknown dynamics of the cutting system and the unknown bounding functions related to the time-delayed tool vibrations, and applying the Lyapunov-Krasovskii functional to aid in treating the time-delayed effect of the regenerative mechanism of chatter. By employing the developed control approach, the tool vibrations in two directions vertical to each other are successfully suppressed. Finally, simulations are presented to validate the effectiveness of the developed control approach.

It has been established that the transfer of human adaptive impedance is of great significance for physical human-robot interaction (pHRI). By processing the electromyography (EMG) signals collected from human muscles, the limb impedance could be extracted and transferred to robots. The existing impedance transfer interfaces rely only on visual feedback and, thus, may be insufficient for skill transfer in a sophisticated environment. In this paper, physical haptic feedback mechanism is introduced to result in muscle activity that would generate EMG signals in a natural manner, in order to achieve intuitive human impedance transfer through a designed coupling interface. Relevant processing methods are integrated into the system, including the spectral collaborative representation-based classifications method used for hand motion recognition; fast smooth envelop and dimensionality reduction algorithm for arm endpoint stiffness estimation. The tutor's arm endpoint motion trajectory is directly transferred to the robot by the designed coupling module without the restriction of hands. Haptic feedback is provided to the human tutor according to skill learning performance to enhance the teaching experience. The interface has been experimentally tested by a plugging-in task and a cutting task. Compared with the existing interfaces, the developed one has shown a better performance.

This paper addresses the problem of robotic manipulators with unknown deadzone. In order to tackle the uncertainty and the unknown deadzone effect, we introduce adaptive neural network (NN) control for robotic manipulators. State-feedback control is introduced first and a high-gain observer is then designed to make the proposed control scheme more practical. One radial basis function NN (RBFNN) is used to tackle the deadzone effect, and the other RBFNN is also proposed to estimate the unknown dynamics of robot. The proposed control is then verified on a two-joint rigid manipulator via numerical simulations and experiments.

This paper reports the development of an intelligent shared control system for a robotic manipulator that is commanded by the user's mind. The target objects are detected by a vision system and then displayed to the user in a video that shows them fused with flicking diamonds that are designed to excite electroencephalograph (EEG) signals at different frequency bands. Through the analysis of the invoked EEG signals, a brain-computer interface is developed to infer the exact object that is required by the user. These results are then transferred to the shared control system, which is enabled by visual servoing techniques to achieve accurate object manipulation. The task motion and self-motion (CTS) methods are coordinated to enhance the intelligence of the shared control system by equipping the robot with an autonomous obstacle avoidance function. Extensive experimental studies are performed to verify that the adaptive object tracking algorithm, the CTS method, and the least-squares method are helpful in improving the performance of the intelligent robotic system.

In this paper, a neural-dynamic optimization-based nonlinear model predictive control (NMPC) is developed for the multiple nonholonomic mobile robots formation. First, a model-based monocular vision method is developed to obtain the location information of the leader. Then, a separation-bearing-orientation scheme (SBOS) control strategy is proposed. During the formation motion, the leader robot is controlled to track the desired trajectory and the desired leader-follower relationship can be maintained through the SBOS method. Finally, the model predictive control (MPC) is utilized to maintain the desired leader-follower relationship. To solve the MPC generated constrained quadratic programming problem, the neural-dynamic optimization approach is used to search for the global optimal solution. Compared to other existing formation control approaches, the proposed solution is that the NMPC scheme exploit prime-dual neural network for online optimization. Finally, by using several actual mobile robots, the effectiveness of the proposed approach has been verified through the experimental studies.

In this paper, two upper limbs of an exoskeleton robot are operated within a constrained region of the operational space with unidentified intention of the human operator's motion as well as uncertain dynamics including physical limits. The new human-cooperative strategies are developed to detect the human subject's movement efforts in order to make the robot behavior flexible and adaptive. The motion intention extracted from the measurement of the subject's muscular effort in terms of the applied forces/torques can be represented to derive the reference trajectory of his/her limb using a viable impedance model. Then, adaptive online estimation for impedance parameters is employed to deal with the nonlinear and variable stiffness property of the limb model. In order for the robot to follow a specific impedance target, we integrate the motion intention estimation into a barrier Lyapunov function based adaptive impedance control. Experiments have been carried out to verify the effectiveness of the proposed dual-arm coordination control scheme, in terms of desired motion and force tracking.

This paper is concerned with a model predictive control (MPC) technique together with an adaptive robust scheme for the trajectory tracking of a wheeled inverted pendulum vehicle in the absence of platform velocity information, in addition to taking dynamic uncertainty and external disturbance into consideration. Specifically, to deal with velocity information loss and dynamic uncertainty, an extended state observer is introduced to evaluate the velocity vectors and model dynamics, where the uniformly ultimately bounded property of observer system can be guaranteed by using the Lyapunov stability theorem. With these observations, MPC is employed for the underactuated longitudinal subsystem to achieve longitudinal velocity tracking, as well as holding the pendulum-like vehicle body stability, and in particular taking the state and input saturation into account; at the same time, an adaptive robust controller is constructed for the rotational subsystem to realize the rotational velocity tracking, in which the adaptive laws can enhance the vehicle adaptability in diverse environment. In addition, a saturated trajectory generator with closed-loop characteristic is introduced so as to properly handle the velocity limitation and nonholonomic constraint simultaneously. The simulation results validate that the control system is robust against the external disturbance and model uncertainty, thereby demonstrating the effectiveness and robustness of the proposed control strategy.

In this paper, a mind controlled multi-task manipulator based on motor imagery electroencephalogram (EEG) is proposed. Describe the system function first: In the case of only two types of control signal, the implementation of multi-task Manipulator relies on a toggle-confirmation mode of operation: the task is switched when imagining the left-hand movement, and the task is confirmed when the right-hand movement is imagined. In the BCI system, common spatial pattern (CSP) is used for feature extraction, mutual information for feature selection, and linear discriminant analysis (LDA) for pattern classification. The EEG signal is processed and classified into two categories, imagery of left-hand and right-hand movement. In this way, we can achieve the multi-task control of the manipulator under the premise of ensuring the accuracy of EEG recognition.

We develop a novel disturbance observer-based adaptive fuzzy control approach in this paper for a class of uncertain multi-input-multi-output mechanical systems possessing unknown input nonlinearities, i.e., deadzone and saturation and time-varying external disturbance. It is shown that the input nonlinearities can be represented by a nominal part and a nonlinear disturbance term. High-dimensional integral-type Lyapunov function is used to construct the controller. Fuzzy logic system is employed to cancel model uncertainties, and disturbance observer is also integrated into control design to compensate the fuzzy approximation error, external disturbance, and nonlinear disturbance caused by the unknown input nonlinearities. Semiglobally uniformly ultimately boundness of the closed-loop control system is guaranteed with tracking errors keeping bounded. Experimental studies on a robotic exoskeleton using the proposed control demonstrate the effectiveness of the approach.

This paper presents a teleoperation control for an exoskeleton robotic system based on the brain-machine interface and vision feedback. Vision compressive sensing, brain-machine reference commands, and adaptive fuzzy controllers in joint-space have been effectively integrated to enable the robot performing manipulation tasks guided by human operator's mind. First, a visual-feedback link is implemented by a video captured by a camera, allowing him/her to visualize the manipulator's workspace and movements being executed. Then, the compressed images are used as feedback errors in a nonvector space for producing steady-state visual evoked potentials electroencephalography (EEG) signals, and it requires no prior information on features in contrast to the traditional visual servoing. The proposed EEG decoding algorithm generates control signals for the exoskeleton robot using features extracted from neural activity. Considering coupled dynamics and actuator input constraints during the robot manipulation, a local adaptive fuzzy controller has been designed to drive the exoskeleton tracking the intended trajectories in human operator's mind and to provide a convenient way of dynamics compensation with minimal knowledge of the dynamics parameters of the exoskeleton robot. Extensive experiment studies employing three subjects have been performed to verify the validity of the proposed method.

Robots with coordinated dual arms are able to perform more complicated tasks that a single manipulator could hardly achieve. However, more rigorous motion precision is required to guarantee effective cooperation between the dual arms, especially when they grasp a common object. In this case, the internal forces applied on the object must also be considered in addition to the external forces. Therefore, a prescribed tracking performance at both transient and steady states is first specified, and then, a controller is synthesized to rigorously guarantee the specified motion performance. In the presence of unknown dynamics of both the robot arms and the manipulated object, the neural network approximation technique is employed to compensate for uncertainties. In order to extend the semiglobal stability achieved by conventional neural control to global stability, a switching mechanism is integrated into the control design. Effectiveness of the proposed control design has been shown through experiments carried out on the Baxter Robot.

An adaptive neural control strategy for multiple input multiple output nonlinear systems with various constraints is presented in this paper. To deal with the nonsymmetric input nonlinearity and the constrained states, the proposed adaptive neural control is combined with the backstepping method, radial basis function neural network, barrier Lyapunov function (BLF), and disturbance observer. By ensuring the boundedness of the BLF of the closed-loop system, it is demonstrated that the output tracking is achieved with all states remaining in the constraint sets and the general assumption on nonsingularity of unknown control coefficient matrices has been eliminated. The constructed adaptive neural control has been rigorously proved that it can guarantee the semiglobally uniformly ultimate boundedness of all signals in the closed-loop system. Finally, the simulation studies on a 2-DOF robotic manipulator system indicate that the designed adaptive control is effective.

In this paper, a sensory-motor fusion-based manipulation and grasping control strategy has been developed for a robotic hand-eye system. The proposed hierarchical control architecture has three modules: 1) vision servoing; 2) surface electromyography (sEMG)-based movement recognition; and 3) hybrid force and motion optimization for manipulation and grasping. A stereo camera is used to obtain the 3-D point cloud of a target object and provides the desired operational position. The AdaBoost-based motion recognition is employed to discriminate different movements based on sEMG of human upper limbs. The operational space motion planning for bionic arm and force planning for multifingered robotic hand can be both transformed as a convex optimization problem with various constraints. A neural dynamics optimization solution is proposed and implemented online. The proposed formulation can achieve a substantial reduction of computational load. The actual implementation includes a bionic arm with dextrous hand, high-speed active vision, and an EMG sensors. A series of manipulation tasks consisting of tracking/recogniting/grasping of an object are implemented, and experiment results exhibit the responsiveness and flexibility of the proposed sensory motion fusion approach.

This paper develops an image-based visual servoing (IBVS) control strategy using model predictive control (MPC) to stabilize a physically constrained mobile robot. In IBVS strategy, ambiguity, and degeneracy problems of the homography and fundamental matrix-based algorithms can be avoided. Moreover, a synthetic error vector incorporating the advantages of IBVS and position-based visual servoing is defined that includes both the robot angle and image coordinates. By using linear system control theory, the kinematics of nonholonomic chained robotic systems can be transformed into a skew-symmetric form, and through introducing an exponential decay phase, the uncontrollable problem can be solved. Then, an MPC strategy is developed and, thereafter, iteratively transformed into a constrained quadratic programming (QP) problem. Subsequently, we utilize a primal-dual neural network (PDNN) to solve this QP problem. By using PDNN optimization, the cost function of MPC effectively converges to the exact optimal values. Finally, experimental studies on the actual robotic systems have been conducted to demonstrate the performance of the proposed approach.

In this paper, a novel control scheme is developed for a teleoperation system, combining the radial basis function (RBF) neural networks (NNs) and wave variable technique to simultaneously compensate for the effects caused by communication delays and dynamics uncertainties. The teleoperation system is set up with a TouchX joystick as the master device and a simulated Baxter robot arm as the slave robot. The haptic feedback is provided to the human operator to sense the interaction force between the slave robot and the environment when manipulating the stylus of the joystick. To utilize the workspace of the telerobot as much as possible, a matching process is carried out between the master and the slave based on their kinematics models. The closed loop inverse kinematics (CLIK) method and RBF NN approximation technique are seamlessly integrated in the control design. To overcome the potential instability problem in the presence of delayed communication channels, wave variables and their corrections are effectively embedded into the control system, and Lyapunov-based analysis is performed to theoretically establish the closed-loop stability. Comparative experiments have been conducted for a trajectory tracking task, under the different conditions of various communication delays. Experimental results show that in terms of tracking performance and force reflection, the proposed control approach shows superior performance over the conventional methods.

The tutor-tutee hand-in-hand teaching may be the most effective approach for a tutee to acquire new motor skills. Repetitive nature of such procedures in a group setting usually results in a high labour cost and time inefficiency. Potential solution can be utilizing robotic platforms playing the role of tutors for demonstrating and transferring the required skills. This requires an appropriate guidance scheme to integrate the tutor's motor functionalities into the robot's control architecture. For instance, for hand-in-hand supervision of the writing task, the tutor's corrections can be applied when necessary, while a very compliant motion can be achieved if no errors are detected. Inspired by this behavior, we develop a teaching interface using a dual-arm robotic platform. In our setup, one arm is connected to the tutees arm providing guidance through a variable stiffness control approach, and the other to the tutor to capture the motion and to feedback the tutees performance in a haptic manner. The reference stiffness for the tutors arm stiffness is estimated in real-time and replicated by the tutees robotic arm. Comparative experiments have been carried out on a dual-arm Baxter robot. The results imply that the human tutor is able to intuitively transfer writing skills to the tutee and also show superior learning performance over over some conventional teaching by demonstration techniques.

Most studies on bilateral teleoperation assume known system kinematics and only consider dynamical uncertainties. However, many practical applications involve tasks with both kinematics and dynamics uncertainties. In this paper, trilateral teleoperation systems with dual-master-single-slave framework are investigated, where a single robotic manipulator constrained by an unknown geometrical environment is controlled by dual masters. The network delay in the teleoperation system is modeled as Markov chain-based stochastic delay, then asymmetric stochastic time-varying delays, kinematics and dynamics uncertainties are all considered in the force-motion control design. First, a unified dynamical model is introduced by incorporating unknown environmental constraints. Then, by exact identification of constraint Jacobian matrix, adaptive neural network approximation method is employed, and the motion/force synchronization with time delays are achieved without persistency of excitation condition. The neural networks and parameter adaptive mechanism are combined to deal with the system uncertainties and unknown kinematics. It is shown that the system is stable with the strict linear matrix inequality-based controllers. Finally, the extensive simulation experiment studies are provided to demonstrate the performance of the proposed approach.

In this paper, a vision-based adaptive control is designed for a wheeled inverted pendulum (WIP) robot to track a moving human target by integration of multisensor data. A new algorithm is employed in the system to combine an OptiTrack camera and a Kinect camera, such that more robust and efficient performance can be achieved for human target detection and tracking. Robust adaptive control has been developed for the WIP robot to maintain its balance on two wheels and to follow the human target using visual feedback. Leader-follower control, dynamic balance control and visual tracking are efficiently combined together to achieved desired tracking and balancing performance. Extensive experiment studies have been performed to test the effectiveness of the proposed control strategies.

Mobile robots tracking a reference trajectory are constrained by the motion limits of their actuators, which impose the requirement for high autonomy driving capabilities in robots. This paper presents a model predictive control (MPC) scheme incorporating neural-dynamic optimization to achieve trajectory tracking of nonholonomic mobile robots (NMRs). By using the derived tracking-error kinematics of nonholonomic robots, the proposed MPC approach is iteratively transformed as a constrained quadratic programming (QP) problem, and then a primal-dual neural network is used to solve this QP problem over a finite receding horizon. The applied neural-dynamic optimization can make the cost function of MPC converge to the exact optimal values of the formulated constrained QP. Compared with the existing fast MPC, which requires repeatedly calculating the Hessian matrix of the Langragian and then solves a quadratic program. The computation complexity reaches O(n3), while the proposed neural-dynamic optimization contains O(n2) operations. Finally, extensive experiments are provided to illustrate that the MPC scheme has an effective performance on a real mobile robot system.

This paper investigates dynamic balance optimization and control of quadruped robots with compliant/flexible joints under perturbing external forces. First, we formulate a constrained dynamic model of compliant/flexible joints for quadruped robots and a reduced-order dynamic model is developed considering the robot interaction with the environment through multiple contacts. A dynamic force distribution approach based on quadratic objective function is proposed for evaluating the optimal contact forces to cope with the external wrench, and fuzzy-based adaptive control of compliant/flexible joints for quadruped robots is proposed to suppress uncertainties in the dynamics of the robot and actuators. The dynamic surface control approaches and fuzzy learning algorithms are combined in the proposed framework. All the signals of the closed-loop system have proven to be uniformly ultimately bounded through Lyapunov synthesis. Simulation experiments were performed for a quadruped robot with compliant/flexible joints. The benefits of its tracking accuracy and robustness indicate that the proposed framework is promising for the robots with payload uncertainties and external disturbances.

This paper studies the optimal distribution of feet forces and control of multilegged robots with uncertainties in both kinematics and dynamics. First, a constrained dynamics for multilegged robots and the constrained environment model are established by considering both kinematic and dynamic uncertainties. Under an external wrench for multilegged robots, the foot forces and moments of the supporting legs can be formulated as quadratic programming problems subject to linear and nonlinear constraints. The neurodynamics of recurrent neural network is developed for foot force optimization. For the obtained optimized tip-point force and the motion of legs, we propose a hybrid task-space trajectory and force tracking based on fuzzy system and adaptive mechanism that are used to compensate for the external perturbation, kinematics, and dynamics uncertainties. The tracking of task-space trajectory and constraint force is achieved under unknown dynamical parameters, constraints, and disturbances. Extensive simulations have been provided to verify the effectiveness of the proposed scheme.

The studies on bio-mechatronics systems and applications have been carried out for more than three decades, to overcome the challenges raised from both theoretical and experimental sides, especially those posed by the application of mechatronics and robotics in healthcare and medical fields. The research on biomedical robotics and bio-mechatronics covers a diverse spectrum of rapid rising interdisciplinary areas including bio-inspired robots for industrial, military, medical, and rehabilitation applications. This special issue aims at showcasing the most exciting and recent advances in the application of robotics and mechatronics in various fields and brings together a broad spectrum of topics covering various definition, development, control, and deployment of bio-mechatronics/robot systems, including social robots, wearable robot systems such as exoskeleton, rehabilitation robot, tele-robot, and a numbers of systems engineering approaches such as modeling, optimization and control. This special issue is to give analysis to the biological systems from a “bio-mechatronic” point of view, and to investigate the engineering and scientific principles behind their remarkable performance. High-quality original papers of innovative ideas and concepts have been included in the special issue of biomedical robotics and bio-mechatronics systems and application. While the design and development of bio-inspired machines and systems with novel and high performance in various applications have been investigated as well. The recent development of multidisciplinary research shall contribute to the promotion of the research on biomedical robotics and bio-mechatronics systems and application, with application to transportation, diagnosis, surgery, assistive technology, prosthetics, personal assistance, rehabilitation, health care, in laboratory, hospital, and the real world.

This paper presents fuzzy approximation-based adaptive backstepping control of an exoskeleton for human upper limbs to provide forearm movement assistance so that a human forearm can track any continuous desired trajectory (or constant setpoint) in the presence of parametric/functional uncertainties, unmodeled dynamics, actuator dynamics, and/or disturbances from environments. Given the desired trajectories of human forearm positions, in the developed control, adaptive fuzzy approximators are used to estimate the dynamical uncertainties of the human-robot system, and an iterative learning scheme is utilized to compensate for unknown time-varying periodic disturbances. With the synthesis of the backstepping, iterative learning, and Lyapunov function approaches, the developed controller does not require exact knowledge of the exoskeleton model, and the close-loop system can be proven to be semiglobally uniformly bounded. Three comparison experiments are conducted to illustrate the effectiveness of the proposed control scheme by tracking periodic/repeated trajectories.

In this paper, a decentralized adaptive fuzzy control has been developed for two cooperating robotic manipulators moving an object with impedance interaction. The contact forces are described using gradients of nonlinear potentials; then, the deformations of the contact surface can be obtained by an impedance approach. The cooperating manipulators are considered as a combination of subsystems, and the decentralized local dynamics coupled with physical interactions among the subsystems are developed. To compensate for the effect of dynamics uncertainties and external disturbances, decentralized fuzzy control combining parameter adaptations and disturbance observers is constructed. It guarantees the motion trajectories and impedance forces of the constrained object converging to the desired manifolds. It is theoretically established that the disturbance observers compensate for unparameterizable uncertainties, while the adaptive fuzzy mechanism compensates for the fast-changing components of the uncertainties that go beyond the disturbance observers. Moreover, unknown nonlinear dynamics such as the inertia matrix, Coriolis/centripetal matrix, and frictions, as well as interconnections with nonlinear bounds, can be accommodated through online learning. The experiments on two real robots have been carried out to verify the effectiveness of the proposed theoretical results.

In this brief, the utilization of robust model-based predictive control is investigated for the problem of missile interception. Treating the target acceleration as a bounded disturbance, novel guidance law using model predictive control is developed by incorporating missile inside constraints. The combined model predictive approach could be transformed as a constrained quadratic programming (QP) problem, which may be solved using a linear variational inequality-based primal-dual neural network over a finite receding horizon. Online solutions to multiple parametric QP problems are used so that constrained optimal control decisions can be made in real time. Simulation studies are conducted to illustrate the effectiveness and performance of the proposed guidance control law for missile interception.

We propose a dual-arm cyclic-motion-generation (DACMG) scheme by a neural-dynamic method, which can remedy the joint-angle-drift phenomenon of a humanoid robot. In particular, according to a neural-dynamic design method, first, a cyclic-motion performance index is exploited and applied. This cyclic-motion performance index is then integrated into a quadratic programming (QP)-type scheme with time-varying constraints, called the time-varying-constrained DACMG (TVC-DACMG) scheme. The scheme includes the kinematic motion equations of two arms and the time-varying joint limits. The scheme can not only generate the cyclic motion of two arms for a humanoid robot but also control the arms to move to the desired position. In addition, the scheme considers the physical limit avoidance. To solve the QP problem, a recurrent neural network is presented and used to obtain the optimal solutions. Computer simulations and physical experiments demonstrate the effectiveness and the accuracy of such a TVC-DACMG scheme and the neural network solver.

In this paper, a class of nonholonomic chained systems is first converted into two subsystems, and then an explicit exponential decaying term is introduced into the input of the first subsystem to guarantee its controllability. After a state-scaling transformation, a model predictive control (MPC) scheme is proposed for the nonholonomic chained systems. The proposed MPC scheme employs a general projection neural network (GPN) to iteratively solve a quadratic programming (QP) problem over a finite receding horizon. The GPN employed in this paper is proved to be stable in the sense of Lyapunov, and its global convergence to the optimal solution is guaranteed for the reformulated QP. A simulation study is performed to show stable and convergent control performance under the proposed method, irrespective of whether the control input $\boldsymbol {u_{1}}$ vanishes or not.

This paper investigates optimal feet forces distribution and control of quadruped robots with uncertainties in both kinematics and dynamics. First, a constrained dynamics of quadruped robots is established. The distribution of required forces and moments on the supporting legs of a quadruped robot can be formulated as a problem for minimizing an objective function subject to form-closure constraints and balance constraints of external force. The dynamics of recurrent neural network for realtime force optimization are proposed. For the obtained optimized tip-point force and the motion of legs, we propose the hybrid motion/force control based on adaptive fuzzy system to compensate for the external perturbation and the task-space tracking errors in the environment. The proposed control can confront the uncertainties including approximation task space error and external perturbation. The verification of the proposed control is conducted using the extensive simulations.

In this paper, an adaptive fuzzy control is investigated for multilateral teleoperation of two cooperating robotic manipulators that manipulate an object with constrained trajectory/force in the presence of dynamics uncertainties and random network-induced delays. First, the interconnected dynamics that consist of two master robots and cooperating slave robots are formulated. To consider multiple stochastic delays in communication channels, Markov processes are used to model these random network-induced delays. The interconnected dynamics of the teleoperation are divided into a local master/slave position/force subsystem and a stochastic-delayed motion synchronization subsystem. Then, an adaptive fuzzy control strategy, which is based on linear matrix inequalities (LMIs) that combine adaptive update techniques, is proposed to suppress the dynamics uncertainties, the external disturbances, and the multiple stochastic delays in communication channels. The control approach ensures that the defined synchronization errors converge to zero. The stochastic stability in mean square of the closed-loop system is proved using LMIs based on Lyapunov-Krasovskii functional synthesis. The proposed controls are validated using extensive simulation studies.

This paper investigates two surface electromyogram (sEMG)-based control strategies developed for a power-assist exoskeleton arm. Different from most of the existing position control approaches, this paper develops force control methods to make the exoskeleton robot behave like humans in order to provide better assistance. The exoskeleton robot is directly attached to a user's body and activated by the sEMG signals of the user's muscles, which reflect the user's motion intention. In the first proposed control method, the forces of agonist and antagonist muscles pair are estimated, and their difference is used to produce the torque of the corresponding joints. In the second method, linear discriminant analysis-based classifiers are introduced as the indicator of the motion type of the joints. Then, the classifier's outputs together with the estimated force of corresponding active muscle determine the torque control signals. Different from the conventional approaches, one classifier is assigned to each joint, which decreases the training time and largely simplifies the recognition process. Finally, the extensive experiments are conducted to illustrate the effectiveness of the proposed approaches.

This paper investigates optimal feet forces' distribution and control of quadruped robots under external disturbance forces. First, we formulate a constrained dynamics of quadruped robots and derive a reduced-order dynamical model of motion/force. Consider an external wrench on quadruped robots; the distribution of required forces and moments on the supporting legs of a quadruped robot is handled as a tip-point force distribution and used to equilibrate the external wrench. Then, a gradient neural network is adopted to deal with the optimized objective function formulated as to minimize this quadratic objective function subjected to linear equality and inequality constraints. For the obtained optimized tip-point force and the motion of legs, we propose the hybrid motion/force control based on an adaptive neural network to compensate for the perturbations in the environment and approximate feedforward force and impedance of the leg joints. The proposed control can confront the uncertainties including approximation error and external perturbation. The verification of the proposed control is conducted using a simulation.

In this paper, automatic motion control is investigated for one of wheeled inverted pendulum (WIP) models, which have been widely applied for modeling of a large range of two wheeled modern vehicles. First, the underactuated WIP model is decomposed into a fully actuated second order subsystem Σa consisting of planar movement of vehicle forward and yaw angular motions, and a nonactuated first order subsystem Σb of pendulum motion. Due to the unknown dynamics of subsystem Σa and the universal approximation ability of neural network (NN), an adaptive NN scheme has been employed for motion control of subsystem Σa . The model reference approach has been used whereas the reference model is optimized by the finite time linear quadratic regulation technique. The pendulum motion in the passive subsystem Σb is indirectly controlled using the dynamic coupling with planar forward motion of subsystem Σa , such that satisfactory tracking of a set pendulum tilt angle can be guaranteed. Rigours theoretic analysis has been established, and simulation studies have been performed to demonstrate the developed method.

This paper develops a learning control algorithm adapting the reference point and force to interact with an object of unknown geometry and elasticity. The controller is inspired by neuroscience studies that investigated the neural mechanisms when human adapt to virtual objects of different properties. The learning control algorithm estimates the shape and stiffness of the given object while maintaining a specified contact force with the environment. Simulations demonstrate the efficiency of the algorithm to identify the geometry and impedance of an unknown object without requiring force sensing. These properties are attractive for robotic haptic exploration with little demand on the sensing.

In this paper, we investigate optimized adaptive control and trajectory generation for a class of wheeled inverted pendulum (WIP) models of vehicle systems. Aiming at shaping the controlled vehicle dynamics to be of minimized motion tracking errors as well as angular accelerations, we employ the linear quadratic regulation optimization technique to obtain an optimal reference model. Adaptive control has then been developed using variable structure method to ensure the reference model to be exactly matched in a finite-time horizon, even in the presence of various internal and external uncertainties. The minimized yaw and tilt angular accelerations help to enhance the vehicle rider's comfort. In addition, due to the underactuated mechanism of WIP, the vehicle forward velocity dynamics cannot be controlled separately from the pendulum tilt angle dynamics. Inspired by the control strategy of human drivers, who usually manipulate the tilt angle to control the forward velocity, we design a neural-network-based adaptive generator of implicit control trajectory (AGICT) of the tilt angle which indirectly "controls" the forward velocity such that it tracks the desired velocity asymptotically. The stability and optimal tracking performance have been rigorously established by theoretic analysis. In addition, simulation studies have been carried out to demonstrate the efficiency of the developed AGICT and optimized adaptive controller.

In this paper, an adaptive fuzzy control scheme is proposed for hybrid motion/force of trilateral teleoperation systems with a dual-master-single-slave configuration under stochastic time-varying delays in communication channels. Different from previous works on bilateral teleoperation systems, this paper addresses dual-master trilateral control of a single holonomic-constrained robotic manipulator, where the communication delays are modeled as multiple Markov chains, and the motion/force controls are investigated under consideration of unsymmetric stochastic time-varying delays and system dynamical uncertainties. Using partial feedback linearization, the whole trilateral teleoperation system, which consists of both master and slave manipulator dynamics, is transformed into three subsystems. By integrating Markov jump systems to handle random delays, adaptive fuzzy control strategies are developed for the nonlinear teleoperators with modeling uncertainties and external disturbances by using the approximation property of the fuzzy logic systems (FLSs). It is proven that the trilateral teleoperation system is stochastically stable in mean square under specific linear matrix inequality (LMI) conditions, and all the signals of the resulting closed-loop system are uniformly bounded. The proposed scheme is validated by extensive simulations.

The high conventional accuracy of pattern recognition-based surface myoelectric classification in laboratory experiments does not necessarily result in high accessibility to practical protheses. An obvious reason is the effect of signals of untrained classes caused by the relatively small training dataset. In order to make the classifier robust to untrained classes, a classification scheme is developed based on boosting and random forest classifiers in this paper. Meanwhile, a threshold, the post probability of the prediction, is introduced as a balance (i.e., adjust) between the accurate classification and the rejection of the samples belonging to some untrained classes. The experiments are conducted to compare with other two schemes using linear discriminant analysis and support vector machines. Surface electromyogram signals, labeled with seven isometric movements, are collected from six healthy subjects' forearm. It is shown that the proposed scheme can reach up to about 92% accuracy in recognizing trained classes and 20% for untrained classes. Through adjusting the threshold, the accuracy of rejecting untrained classes reaches up to around 80%, with small decrease in recognizing trained classes (down to 80%). In the analysis of experiments' results, we also find that the proposed scheme has better error distribution among the classes.

In this paper, adaptive neural network control is investigated for single-master-multiple-slaves teleoperation in consideration of time delays and input dead-zone uncertainties for multiple mobile manipulators carrying a common object in a cooperative manner. Firstly, concise dynamics of teleoperation systems consisting of a single master robot, multiple coordinated slave robots, and the object are developed in the task space. To handle asymmetric time-varying delays in communication channels and unknown asymmetric input dead zones, the nonlinear dynamics of the teleoperation system are transformed into two subsystems through feedback linearization: local master or slave dynamics including the unknown input dead zones and delayed dynamics for the purpose of synchronization. Then, a model reference neural network control strategy based on linear matrix inequalities (LMI) and adaptive techniques is proposed. The developed control approach ensures that the defined tracking errors converge to zero whereas the coordination internal force errors remain bounded and can be made arbitrarily small. Throughout this paper, stability analysis is performed via explicit Lyapunov techniques under specific LMI conditions. The proposed adaptive neural network control scheme is robust against motion disturbances, parametric uncertainties, time-varying delays, and input dead zones, which is validated by simulation studies.

Wheeled inverted pendulum (WIP) models have been widely used in the field of autonomous robotics and intelligent vehicles. A novel transportation system, WIP-car is proposed in this paper, which is composed of a mobile wheeled inverted pendulum system, a driven chair, an acceleration pedal and a deceleration pedal, which are used to drive the chair forward or backward such that the car can be accelerated or decelerated. The neural-adaptive implicit control is designed for dynamic balance and stable tracking of desired trajectories of WIP-car. Neither the dynamics nor the dimension of the regulated system is required to be known, while the relative degree of the regulated output is assumed to be known. Under the assumption that WIP-car is feedback linearizable, adaptive neural network is introduced to cancel the inversion dynamics error. Simulation results demonstrate that the system is able to track reference signals satisfactorily with all closed loop signals uniformly bounded.

In this paper, adaptive dynamic coupling control is considered for hybrid joint, which could be switched to either active (actuated) or passive (under-actuated) mode, for human-symbiotic wheeled mobile manipulators. Based on Lyapunov synthesis, adaptive coupling control using physical properties of wheeled mobile manipulators proposed for passive hybrid joints ensures that the system outputs track the given bounded reference signals within a small neighborhood of zero, and guarantees semi-global uniform boundedness of all closed loop signals. The effectiveness of the proposed controls is verified through extensive simulations.

In this paper, coupled dynamics are presented for two cooperating mobile robotic manipulators manipulating an object with relative motion in the presence of uncertainties and external disturbances. Centralized robust adaptive controls are introduced to guarantee the motion, and force trajectories of the constrained object converge to the desired manifolds with prescribed performance. The stability of the closed-loop system and the boundedness of tracking errors are proved using Lyapunov stability synthesis. The tracking of the constraint trajectory/force up to an ultimately bounded error is achieved. The proposed adaptive controls are robust against relative motion disturbances and parametric uncertainties and are validated by simulation studies.

In this paper, adaptive robust force/motion control strategies are presented for mobile manipulators under both holonomic and nonholonomic constraints in the presence of uncertainties and disturbances. The proposed control is robust not only to parameter uncertainties such as mass variations but also to external ones such as disturbances. The stability of the closed-loop system and the boundedness of tracking errors are proved using Lyapunov stability synthesis. The proposed control strategies guarantee that the system motion converges to the desired manifold with prescribed performance and the bounded constraint force. Simulation results validate that the motion of the system converges to the desired trajectory, and the constraint force converges to the desired force.

Motion control is investigated for a nonholonomic mobile manipulator using hybrid joints, which could be switched to either active (actuated) or passive (underactuated) mode. The mobile platform is driven with two independent wheels and the arm has hybrid joints in the horizontal plane. The dynamic constraints are shown to be a first-order nonholonomic for the mobile base and a second-order nonholonomic for the link when one of hybrid joints is underactuated. Two Different motion control methods for mobile underactuated manipulator with one passive joint are proposed. Simulation and experimental studies show the effectiveness of the proposed methods

a human-symbiotic safe mobile manipulator with nonholonomic constraint is described to realize human safety, impact force absorbing and task fulfillment. The robot consists of an arm covered with soft materials and hybrid joints, which can be put into active or passive mode as needed. In an unexpected or expected collision with human, the arising impulse force is attenuated effectively by the physical model formed with the hybrid joint and the soft material. Owing to the displacement movement of the link when the joint is passive, a recovery control algorithm has been developed for the end-effector to maintain its desired task position after the collision. Simulation and experiment results confirm that the proposed physical model is suitable for robot working in the human-robot symbiotic environment and the control method are useful for robot’s task fulfillment.

In this paper, a human-symbiotic robot is described to realize human safety, impact force absorbing and task fulfillment. The robot consists of an arm covered with soft materials and hybrid joints, which can be put into active or passive mode as needed. In an unexpected or expected collision with human, the arising impulse force is attenuated effectively by the physical model formed with the hybrid joint and the soft material. Owing to the displacement movement of the link when the joint is passive, a recovery control algorithm has been developed for the end-effector to maintain its desired task position after the collision. Simulation results confirm that the proposed physical model is suitable for robot working in the human-robot symbiotic environment and the control method is useful for robot's task fulfillment.