The task performance of mobile manipulators can be significantly enhanced by whole-body control and optimization in complex scenarios. Due to the nonlinear properties of whole-body dynamics and parameter uncertainty, modeling accurate system dynamics is essential in addition to designing an effective control strategy. However, traditional control methods have high computational costs and fail to deal with the parameter errors caused by model linearization. To address these issues, we propose a model predictive control (MPC)-Net, a learning-based approach that facilitates rapid online optimization by combining deep learning with multiple MPCs. Firstly, we develop a parameter identification algorithm based on a deep neural model to estimate the unknown dynamics parameters. Although the control performance of the MPC approach positively correlated with the prediction horizon, a long horizon would result in additional computational costs. Thus, MPC-Net is constructed by combining multiple sub-MPC issues, and the nonlinear coefficients are obtained by using a deep neural network-based optimization framework. Furthermore, MPC-Net generates the solution by combining the outputs of multiple sub-MPC problems using the nonlinear transformation of learned coefficients. Experiments are conducted on a mobile manipulator, which demonstrates the proposed MPC-Net-based optimization control offers fast efficient computation and low tracking error performance.

Humans can naturally learn and adapt to walking patterns in a variety of terrains. To simulate this learning characteristic, this article introduces a neural dynamics-based impedance optimization and trajectory adaptation approach for our designed soft exosuit, with a dual-driven configuration to assist both ankles of individuals. This method adaptively learns the impedance of the human ankle joint using measured interaction forces and dynamically adjusts trajectories to align with real-time human-robot interaction. Additionally, an adaptive control framework integrating neural dynamics-based optimization with several adaptive laws is developed to achieve stable tracking of updated reference trajectories, with Lyapunov stability analysis confirming uniform ultimate boundedness (UUB) of the closed-loop system. The designed controller offers the benefit of concurrently addressing trajectory adaptation, force control, and impedance tuning for soft exosuits. Experimental validation on human subjects across various terrains demonstrates that the proposed method reduces maximum trajectory tracking error to 0.016 rad (lower than PID and ADRC controllers) and enables impedance parameters to converge within 3 gait cycles. The controller concurrently addresses trajectory adaptation, force control, and impedance tuning, offering a lightweight (8 kg) and wearability-optimized solution for walking assistance.

Wearable flexible lower-limb exoskeletons, commonly referred to as soft exosuits, have emerged as a promising technology for enhancing mobility and assisting gait rehabilitation. However, most conventional designs provide assistance failing to adapt to variations in walking cadence or gait phase subdivision. This study presents a planning and control frame work for a cable-driven exosuit that delivers personalised ankle assistance in the triadic interaction encompassing the human, the robot, and the environment. The framework enables automatic gait phase detection and adaptive assistance, dynamically responding to changes in gait, environmental conditions, and metabolic demands. Inspired by the periodic nature of human walking, a gait cycle is divided into eight phases using a designed transformer classification model that processes foot force and inertial measurement unit (IMU) data. Heart rate is incorporated to provide feedback on metabolic changes. A human-in-the-loop control strategy based on an event-triggered mechanism utilizing fine gait classification and heart rate is proposed to achieve ankle joint assistance adaptation. When the deviations in gait phases or metabolic conditions exceed thresholds, the ankle joint assistive trajectory is replanned online via optimization with safety constraints. An adaptive fuzzy controller ensures stable tracking under uncertain dynamics and external disturbances. The stability of the control system is analytically verified using Lyapunov theory, and experimental results demonstrate the effectiveness of the proposed approach.

This paper proposes a hierarchical task-oriented whole-body locomotion framework for the exoskeleton walking-cart pushing (EWCP) task, which includes a straight gait and a bypassing gait, allowing the exoskeleton robot to avoid obstacles during walking. In this framework, the core components are gait planning and phase estimation for locomotion in unstructured environments. Notably, our mobile redundancy exoskeleton system can provide more flexibility and versatility in manipulation when performing complex tasks. For the hierarchical task-oriented whole-body locomotion, the detour gait consists of straight lines and turning shapes so that the EWCP system can avoid obstacles on the ground. For gait planning, we use the dynamic motion primitives to learn the joint motion trajectory of whole-body locomotion, which has good generalization ability and adaptability with respect to the gait. For phase estimation, the current gait phase can be estimated from the joint angles. Additionally, we design an task switching mechanism, where the exoskeleton system can switch different configurations flexibly for different scenarios, such as track switching only with both feet supported. And the phase estimation and gait switching strategy ensure the stability of task switching. The experimental results show that the exoskeleton can effectively accomplish EWCP tasks in an environment with obstacles. Our work has also shown that even some challenging motion tasks can be implemented with relatively simple controllers, which greatly simplifies the design of control systems.Note to Practitioners—This paper is motivated by issues of hierarchical tasks of the lower limb exoskeleton. Traditional exoskeletons cannot achieve obstacle avoidance in complex scenes because they do not have enough degrees of freedom or they do not use hierarchical locomotion. In this paper, a hierarchical task-oriented whole-body locomotion is proposed, which includes a straight gait and a bypassing gait, allowing the exoskeleton robot to avoid obstacles during walking. For gait planning, we use DMP to generate the gait trajectory, which can ensure the smoothness of trajectory. For phase estimation, the current gait phase can be estimated from the joint angles. Additionally, we design an task switching mechanism, where the exoskeleton system can switch different configurations flexibly for different scenarios, such as track switching only with both feet supported. And the phase estimation and gait switching strategy ensure the stability of task switching. The proposed hierarchical task-oriented whole-body locomotion framework is expected to be applied to exoskeletons to assist patients in rehabilitation training and users in mobility in daily life. Additionally, the proposed framework uses a series of motion primitives to learn and reproduce the trajectory and update it online, which requires a heavy computation load calculation to ensure the update speed of the trajectory. Therefore, we expect to reduce computation complexity for a walking exoskeleton in the future.

Enhancing energy efficiency is pivotal for advancing robotic lower limb prostheses. However, the conventional control strategy in powered prostheses displays notable variance in energy economy across terrains during walking. This challenge could be managed by developing control optimization methods to effectively reduce the subject’s metabolic cost and the device’s energy expenditure, thereby improving the walking economy in diverse environments. In this study, we investigate a multi-objective optimization problem (MOP) for controlling above-knee prostheses, intending to reduce prosthetic power consumption and the subject’s metabolic costs during walking activities. To bolster real-time performance in the optimization loop, we introduce a knee-guided evolutionary algorithm (KGEA) for MOP, which efficiently reduces exploration space and time complexity, thus enabling the practical implementation of a limited number of solutions in each iteration. The effectiveness of the proposed optimization-based strategy was evaluated through experiments involving two lower limb amputees engaging in slope, stair, and flat walking. Our results demonstrated that the optimization-based strategy significantly reduced the subjects’ metabolic consumption, with an average reduction of 16.53% for subject 1 and 10.23% for subject 2 across different terrains, compared to using the prosthesis without the optimization strategy. This promising outcome marks a substantial advancement in the development of energy-efficient, multi-degree-of-freedom power prostheses.Note to Practitioners—Lower Limb amputees using powered prostheses experience energy efficiency challenges due to limited battery life, particularly with multi-degree-of-freedom prostheses over varied terrains. Additionally, compared to passive alternatives, the increased weight of powered prosthetics poses a challenge in managing users’ metabolic costs, requiring suitable optimization strategies to offset the additional burden and optimize body economy during locomotion. The proposed KGEA-based multi-objective optimization method, designed explicitly for above-knee prostheses, targets the reduction of prosthetic power consumption and subject’s metabolic costs during walking activities. This innovation will likely benefit individuals with lower limb amputations, offering them improved mobility and comfort across various terrains. By addressing variance in varied terrains and inefficiencies in energy consumption caused by traditional control strategies, this work can assist practitioners in creating more effective and user-friendly prosthetic devices. Furthermore, we explore continuous refinement of the multi-objective optimization model and KGEA to accommodate a broader range of user-profiles and walking scenarios, ensuring its applicability in real-world situations.

Visual impairments significantly affect individuals’ ability to perform essential tasks such as communication, object search, and navigation. Traditional wearable augmented vision devices rely on modular designs that separate functions like perception and path planning, leading to cumulative errors and inefficiencies in real-world applications. To address these challenges, we propose an end-to-end multimodal large model framework, UniANS, specifically designed for wearable augmented vision devices. UniANS integrates visual perception, speech interaction, and path planning into a unified framework. Such integration improves task coordination, reduces error propagation, and enhances overall performance. We also propose a prompt design strategy with a mixture of cluster-conditional low-rank adaptation experts architectures and dual-branch encoders, combined with advanced preprocessing techniques for visual and speech modules. The framework has been validated through ablation studies, showing superior performance in accuracy and task effectiveness compared to existing methods. We further showcase its capabilities in addressing challenges related to communication, object search, and indoor navigation tasks. The design of UniANS enhances mobility and quality of life for visually impaired individuals.

Visually impaired individuals face limited mobility and restricted independent navigation in complex environments. Improving mobility and independent navigation for people with visual impairments is essential. In this paper, we introduce wearable electronic glasses (E-Glasses) that utilize a target detection network to fuse visual and auditory information for searching desired targets. Integrated with the electronic glasses, we present a neural path planning that combines spiking neural and convolutional neural networks. Several participants took part in experiments that showcased the remarkable capabilities of the developed system in target detection and navigation. The experimental results revealed an impressive success rate of 95.46% for the target detection network, providing participants with more accurate target information. Additionally, the neural path planning network achieved a success rate of 92.60%, demonstrating a significant speed advantage compared to the enhanced$A^{\ast}$algorithm.Note to Practitioners—This article aims to address the mobility and navigation challenges faced by visually impaired individuals in complex environments. Many solutions have been proposed to assist with wearable navigation by utilizing sensors to perceive the environment and provide path information to users. In this article, we introduce wearable electronic glasses (E-Glasses) with advanced target detection and path planning capabilities to help visually impaired individuals navigate more effectively in complex environments. We have developed a target detection network that integrates visual and auditory information to effectively search for desired targets. Additionally, we have developed a neural path planning algorithm that combines spiking neural networks and convolutional neural networks. Furthermore, experiments conducted in indoor navigation challenges demonstrate the feasibility of this approach. In future research, we will focus on further exploring the target detection network and refining the neural path planning algorithms to improve the overall performance of the system.

Despite remarkable progress in robotic exoskeletons, exoskeletons are still far from matching human-level guidance and locomotion performance in gait training or movement enhancement. A desirable exoskeleton would first provide a standard gait profile by learning from human operators while requiring body weight support with active human-following to govern dynamic locomotion synchronization. To address these issues, in this article, we propose a human operator-involved dynamic locomotion synchronization control framework for the lower limb exoskeleton actively following gait training with gravity-supporting. First, we designed a human motion capture system based on a five-link model for the locomotion of a human operator. To reproduce human-level motor skills, we use whole-body teleoperation to leverage human control intelligence to command the locomotion of a robotic exoskeleton system. Specifically, using the linear inverted pendulum (LIP) model, the human operator's divergent component of motion (DCM) is obtained by the human motion capture system. The dynamic similarity is used to generate a reference DCM for the robotic exoskeleton to synchronize the human operator's movement. Finally, a fuzzy-based adaptive controller is designed to track the synchronous trajectory for the exoskeleton in the presence of robotic dynamics uncertainties with input saturation. Experiments on the human subject are carried out to demonstrate the effectiveness of the proposed method.

Conventional lower limb exoskeletons (LLEs) and their corresponding rehabilitation protocols can hardly provide safe and customizable gait rehabilitation training for different patients and scenarios. Thus, this study presents an 8-DoF rehabilitation LLE equipped with a cable-driven body weight support (BWS) mobile mechanism. The mobile BWS mechanism is designed to follow the wearer and offer preset supportive forces and balance protection. A whole body motion planning approach is proposed, wherein iterative null-space projection is employed to solve the task-space trajectories of gait training into the joint-space trajectories of the LLE. For better control performance, dynamic parameters of the human-LLE coupling system are estimated. A control scheme combining robust model predictive control (MPC) and disturbance observer is then designed to manipulate the system against dynamics uncertainty and disturbance during trajectory tracking. In the validation experiments, the nominal model of robust MPC is discretized into quadratic programming problems and solved online by the neuro-dynamics optimization. The experimental results demonstrate the rationality of our system design and motion planning method as well as the effectiveness and stability of the control scheme.

Many previous works on wearable soft exosuits have primarily focused on assisting human motion, while overlooking safety concerns during movement. This article introduces a novel single-motor, altering bi-directional transfer soft exosuit based on impedance optimization and adaptive neural control, which provides assistance to the lower limbs using Bowden cables. This innovative soft exosuit integrates control barrier functions into the impedance optimization, allowing multiple safety constraints to be considered simultaneously, enabling the system to adaptively learn the impedance of the human ankle joint by analyzing the measured interaction forces at the ankle joint, so that the updated reference trajectories comply with safety requirements. To effectively track the updated reference trajectories, we have introduced an adaptive neural controller based on the integral barrier Lyapunov function. This controller is designed to perform the control task under strict safety constraints. The stability of this control approach is meticulously demonstrated through extensive Lyapunov analysis. In contrast to traditional soft exosuits designed purely for assistance, the key advantage of this technology is its ability to adapt to different terrains while ensuring the safety of human movement during assistance. Through experimental testing, we obtain average tracking errors of 0.0062, 0.0062, and 0.0063 rad for flat, grass, and gravel surfaces, respectively, demonstrating the effectiveness of the proposed strategy.

This paper proposes a hybrid long short-term motor (HLSM) optimization and control approach for a walking exoskeleton. It consists of long-term global optimization, short-term local optimization, human-in-the-loop trajectory adaptation, and hybrid cerebellar model articulation controller (HCMAC). In the long-term global optimization, a graphic Spiking Neural Network (SNN) is utilized for an optimal global path. Along the path, the short-term motor optimization includes footstep optimization and obtains a sequence of footsteps. While in response to the unexpected obstacles along the footstep sequence, a human-in-the-loop planning strategy is designed by a virtual impedance model between the Centers of Mass (COMs) of the human and the exoskeleton, regulating the COM of the exoskeleton and generating footstep adaptation of the exoskeleton such that the exoskeleton can avoid obstacles and maintain its original global trajectory. Moreover, considering the unmodeled dynamics, we propose an HCMAC based on an integral Lyapunov function, which is exploited to counteract the system's nonlinear uncertainties, external disturbances, and reduces a relatively high computational cost. We validate the effectiveness of the HLSM planner and controller in a practical indoor setting. The results demonstrate the effectiveness of HLSM planning and control in a real scenario for a walking exoskeleton.

This article presents a human-in-the-loop cooperative control of a walking exoskeleton to provide assistance to the user and enhance human mobility. First, a dynamic mathematical model of the human-exoskeleton system is derived, and then, a human-in-the-loop cooperative control framework is proposed in two ways: separable cooperative control (SCC) and interactive cooperative control (ICC), respectively. The SCC introduces a space division of the human and the exoskeleton, while the ICC allows the robot to perceive the human intention and follows human motor, thereby improving the collaborative performance. The ICC formulates the impedance connection between the human and the exoskeleton in the divided orthogonal subspaces of walking, such that the robot is able to modify its position of center of mass (COM) when its motor trajectory deviates the one of the human. In addition, a novel adaptation control is proposed to deal with the unmodeled dynamics and trajectory tracking. Finally, to validate the effectiveness of our proposed controller, a series of experiments are conducted in three adults in gait at different speeds. It shows that the proposed controller can preserve the periodic walking gait and inherit the robustness of dealing with perturbations during walking.

Driven by the rise of collaborative robots, a lot of work has focused on the transparency and stability of physical human–robot interaction (pHRI), in which most of the efforts do not take the requirement of multiple tasks into account. However, the spectrum of applications for collaborative robots has been continuously broadened, and robots without the ability to perform multiple tasks simultaneously may not be capable of collaborating in certain scenarios. In this article, we provide a fuzzy-based multitask intelligent control framework of collaborative robots for pHRI. Our controller formulation consists of “outer-loop” and “inner-loop.” In the “outer-loop,” a fuzzy logic system predicts human desired motion trajectory for the robot to track. In the “inner-loop,” the robot is driven by a hierarchical multitask controller to track the trajectory generated by the “outer-loop” and perform other subtasks simultaneously. With null space projections, the whole task stack can be implemented in a strict task hierarchy in order of priority. The weight-tuning law of the FLSs and the hierarchical multitask control law are given based on Lyapunov stability analysis. The proposed control framework is applied to a mobile manipulator and the effectiveness is verified by exploratory experiments. Results confirm the effectiveness of the proposed control framework and compare its performance with other approaches.

Surface electromyography (sEMG) signal has been used in upper limb rehabilitation robots (ULRR). However, existing ULRR based on myoelectric controllers suffers from limited generalization ability in estimating three-dimensional (3-D) motion intention. This article proposes a muscle-synergy-inspired approach to enhance the generalization ability of the myoelectric controller of a cable-driven ULRR. Low-dimensional commands are extracted from sEMG signals based on an EMG-to-muscle activation model and non-negative matrix factorization. The extracted commands are used to estimate the 3-D human force. Two different trajectory tracking tasks are selected to test the generalization ability. The system is trained based on training sets where participants perform one task. Then the system is tested using testing sets where participants perform the other task. Finally, the system is verified on real-time robotic control experiment. Results show that the proposed controller achieves better force estimating accuracy, better trajectory tracking accuracy, and lower interaction force than the myoelectric controller without considering muscle synergies, which means the proposed controller yields better generalization performance.

Navigating independently is a challenge for visually impaired vision due to the demand of obstacles avoiding, recognizing desired objects, and wayfinding in complicated environments. In this paper, we present an augmented wearable E-Glasses with a set of sensors, where an object detection neural network based on visual-auditory fusion method is employed to search desired targets, thus addressing navigation challenges and improving the mobility and independence of the visually impaired. We demonstrate advanced navigation capabilities: indoor wayfinding, recognizing and steering the users to desired goals, and a sequence of indoor challenges. The fusion network adopts a feature-level fusion strategy, which is capable to align two modalities automatically and effectively integrate visual features and audio features. Across all experiments, the developed fusion algorithm has a 94.67% success rate. The wearable E-Glasses supply a platform that helps to improve the mobility and quality of life of people with impaired vision.

Soft exo-suit could facilitate walking assistance activities (such as level walking, upslope, and downslope) for unimpaired individuals. In this article, a novel human-in-the-loop adaptive control scheme is presented for a soft exo-suit, which provides ankle plantarflexion assistance with unknown human-exosuit dynamic model parameters. First, the human-exosuit coupled dynamic model is formulated to express the mathematical relationship between the exo-suit actuation system and the human ankle joint. Then, a gait detection approach, including plantarflexion assistance timing and planning, is proposed. Inspired by the control strategy that is used by the human central nervous system (CNS) to handle interaction tasks, a human-in-the-loop adaptive controller is proposed to adapt the unknown exo-suit actuator dynamics and human ankle impedance. The proposed controller can emulate human CNS behaviors which adapt feedforward force and environment impedance in interaction tasks. The resulting adaptation of actuator dynamics and ankle impedance is demonstrated with five unimpaired subjects and implemented on a developed soft exo-suit. The human-like adaptivity is performed by the exo-suit in several human walking speeds and illustrates the promising potential of the novel controller.

This article presents an active human-following control of the lower limb exoskeleton for gait training. First, to improve safety, considering the human balance, the OpenPose-based visual feedback is used to estimate the individual's pose, then, the active human-following algorithm is proposed for the exoskeleton robot to achieve the body weight support and active human-following. Second, taking the human's intention and voluntary efforts into account, we develop a long short-term memory (LSTM) network to extract surface electromyography (sEMG) to build the estimation model of joints' angles, that is, the multichannel sEMG signals can be correlated with flexion/extension (FE) joints' angles of the human lower limb. Finally, to make the robot motion adapt to the locomotion of subjects under uncertain nonlinear dynamics, an adaptive control strategy is designed to drive the exoskeleton robot to track the desired locomotion trajectories stably. To verify the effectiveness of the proposed control framework, several recruited subjects participated in the experiments. Experimental results show that the proposed joints' angles estimation model based on the LSTM network has a higher estimation accuracy and predicted performance compared with the existing deep neural network, and good simultaneous locomotion tracking performance is achieved by the designed control strategy, which indicates that the proposed control can assist subjects to perform gait training effectively.

The exoskeleton is mainly used by subjects who suffer muscle injury to enhance motor ability in the daily life environment. Previous research seldom considers extending human collaboration skills to human-robot collaborations. In this article, two models, that is: 1) the following the better model and 2) the interpersonal goal integration model, are designed to facilitate the human-human collaborative manipulation in tracking a moving target. Integrated with dual-arm exoskeletons, these two models can enable the robot to successfully perform target tracking with two human partners. Specifically, the manipulation workspace of the human-exoskeleton system is divided into a human region and a robot region. In the human region, the human acts as the leader during cooperation, while, in the robot region, the robot takes the leading role. A novel region-based Barrier Lyapunov function (BLF) is then designed to handle the change of leader roles between the human and the robot and ensures the operation within the constrained human and robot regions when driving the dual-arm exoskeleton to track the moving target. The designed adaptive controller ensures the convergence of tracking errors in the presence of region switches. Experiments are performed on the dual-arm robotic exoskeleton for the subject with muscle damage or some degree of motor dysfunctions to evaluate the proposed controller in tracking a moving target, and the experimental results demonstrate the effectiveness of the developed control.

The upper-limb prosthesis has been extensively studied using electromyography (EMG) signals to overcome the physical and functional deficiencies of amputees in recent years. However, most studies focus on the discrete classification of gestures and ignore the interconnection between the classification results and the neuroprosthesis control interface, which plays a vital role in system development. In this article, a new continuous control scheme is proposed to achieve an effective control of the developed upper-limb prosthesis. It utilizes eight channels of EMG signals of the human upper limb to model and control the developed prosthesis. A continuous control scheme is proposed that combines the state of the system and the decoding results to dynamically produce the expected angular velocity of the joint based on the results of the classification. Finally, experiments are performed to demonstrate the effectiveness of the proposed algorithm using an upper-limb neuroprosthesis, achieving the reach and grasp tasks. The results showed that it improves performance with a regular angular velocity of the joint, which underlines the importance of an adequate control scheme for the EMG-guided prosthesis.

Under-actuated bionic hands have achieved tremendous popularity in many fields because of their advantages of lightweight, budget-friendly, satisfactory flexibility, and adaptability. Except for the bionic mechanical design, various anthropomorphic control strategies have been proposed and investigated in the last decades. However, due to its under-actuated characteristic, there are still many challenges for anthropomorphic control of all the degrees of freedom (DOFs) using less input. It is challenging to map the human hand kinematic synergies on robotic hands, particularly for a dynamic environment. Therefore, it is worth studying how to control the under-actuated bionic hand effectively in a dynamic environment. In this paper, an anthropomorphic control method is proposed using sensor fusion of hand kinematic inputs to control the under-actuated bionic hand. In order to map the kinematics of human fingers to the bionic hand, a novel finger bending angle is defined to represent the posture of human fingers. Multiple Leap Motion Controllers (LMC) are fused to estimate the stable and accurate finger bending angles to avoid the occlusion problem. Finally, experiments with real-time control of the under-actuated bionic hand are implemented to demonstrate the proposed approach’s effectiveness.