This article investigates the system modeling problem for the dynamical process of human brain activity in human-robot cognitive interaction (HRCI). An important novelty of the proposed approaches is to build a computational model of a human-distributed robot-lumped parameter system (HDRLPS) that describes the inherent dynamical principle of human brain activity (with spatiotemporal-varying characteristic) undergoing the interaction between the intrinsic cognitive dynamics and extrinsic robot stimuli. A deterministic learning (DL)-based spatiotemporal dynamics identification scheme is proposed to accurately identify the spatiotemporal dynamics of HDRLS and obtain the associated knowledge as a constant radial basis functional neural network (RBF NN) model. A spatiotemporal dynamics estimator is designed with this model, which can accurately evaluate and monitor the dynamical process of human brain activity in real-time HRCI by the generated dynamics-synchronized state. The effectiveness and practicability of the approaches in the dynamics identification and evaluation for the human brain activity in HRCI are validated by the thorough analysis, including the mathematical proof, the simulation study, and the brain-computer interface (BCI) experiment using publicly available datasets. Our method is compared with state-of-the-art (SOTA) methods, such as LGGNet, EEGNet, Tsception, EEG-Deformer, EEG-Transformer, and EEGViT. The results show that our method can outperform these methods with better recognition accuracy and macro- $F1$ scores. The source code can be found at: https://github.com/alonexing/source_code/tree/master.

Focus of attention is one of the most influential factors facilitating motor training performance. Most of robotic training methods have not well solved the negative effect of divided-attention on motor execution performance, resulting in limited rehabilitation efficiency for motor-cognitive dysfunction. In this study, we propose a novel visuomotor human-robot interaction framework by integrating a gaze-visual game and force-movement robot, to realize more efficient training for both attentional and motor function. An important novelty of this framework is to design a dynamical pattern recognition scheme for the hierarchical-coupled behavior of attentional and motor execution, to facilitate efficient human-robot interaction in both cognitive and motor perspectives. Specifically, an attentional-motor dynamical system modeling method is first developed by using the gaze, force and movement data collected from the human under different attentional-motor behavior. Then, an online dynamical pattern recognition scheme can be design with these models to online recognizing the human’s attentional and motor behavior states. The training robot system can dynamically adjust the parameters according to the recognition results, to guide the collaboration of both attentional and motor training. Experimental study are conducted to demonstrate the desired accuracy and efficiency of our designed approaches in attentional-motor behavior recognition and training.

Lower limb exoskeletons (LLEs) play a crucial role in assisting paraplegic patients with walking in outdoor environments characterized by complex terrains, including various stairs, slopes, and uneven grounds. However, most existing control methods for LLEs rely on predefined joint angles, lacking the flexibility to adapt to diverse terrains. This deficiency often leads to unexpected contacts between the feet of the LLEs and the ground, thereby disrupting the walking balance of the LLEs. In this paper, a novel force-sensor-free contact estimation method is proposed to tackle this problem. This method utilizes only the sensors already present on the LLEs, eliminating the need for any additional force sensors. The proposed approach is founded on the probabilistic modeling of gait phases, knee joint torques, foot heights, and the displacement of the center of mass. Moreover, Kalman filtering is employed to enhance the contact estimation accuracy by integrating multiple probabilistic models. Experiments were carried out on both robot simulation platforms and real exoskeleton robots. The experimental results demonstrate that the proposed approach can accurately estimate contacts during walking on flat ground and stairs. Specifically, it achieves an accuracy of 99% with a time deviation of 8 ms on the flat ground and an accuracy of 95% with a time deviation of 10 ms on stairs.

Motor imagery-based brain–computer interfaces (MI-BCIs) hold significant promise for rehabilitation training in individuals with neurological impairments such as stroke and spinal cord injury (SCI). Achieving precise and robust lower limb movement prediction for each patient is crucial. However, the variability in MI response frequencies and brain activation patterns among subjects presents a great challenge to the generalizability of MI-BCIs. This paper proposes a Tuned Heuristic Fusion Graph Convolutional Network (THFGCN) for limb movement prediction in rehabilitation scenarios. THFGCN innovatively designs a learnable EEG frequency band tuned module and a heuristic space topology module. These two modules allow for the intricate extraction of both frequency and spatial topological features, utilizing graph adjacency matrices that encapsulate channel correlations and spatial relationships, hence fostering individualized analysis and enhanced generalizability across subjects. Furthermore, a spatio-temporal convolution module paired with a feature map attention mechanism is proposed to extract the critical spatio-temporal features of electroencephalogram (EEG) data. Validation experiments on the PhysioNet and LLM-BCImotion datasets against six mainstream methods demonstrate that THFGCN outperforms state-of-theart methods, achieving 88.41% and 82.82% accuracy in the within-subject case, and 65.93% and 60.56% accuracy in the cross-subject case, respectively. Detailed frequency band weight and T-distributed Stochastic Neighbor Embedding visualization validate the effectiveness of proposed modules. Furthermore, feature interpretability analysis proves the extracted features’ profound MI task relevance, underlining THFGCN’s exceptional interpretability.

Gait training with human-like gait patterns can be provided by lower limb exoskeletons (LLEs) for patients with gait impairments. For patients with little effort to keep balance, using a mobile robotic walker to assist gait training with LLEs is an effective way. Since gait patterns are varying with walking speeds, it is a critical issue to coordinated control the robotic walker and the exoskeleton to obtain a natural and human-like walking posture. In this paper, a novel adaptive assistance approach named SQG-OPT is proposed to tackle the problem, which comprises of two parts: the Spatially Quantized Gait (SQG) and the optimization. The SQG generates reference joint angles and reference trajectory of the Center Of Mass (COM) for the human-exoskeleton system in space domain. The optimization part is constructed to convert the reference joint angles from the space domain to the time domain, which is based on the dynamics model of the human-exoskeleton-walker system and adaptive to different walking speeds. The proposed approach has been tested on the robot simulation platform CoppeliaSim, the experimental results indicate that the proposed approach can generate human-like gait patterns for different walking speeds from 0 to 0.8 m/s. Additionally, in comparison with other methods, the proposed approach has a better performance on the movement tracking of the COM for a natural walking posture.Note to Practitioners—The coordinated control is important for the exoskeleton robot with a mobile robotic walker, one of the potential challenge is the adaptive coordinated motion planning for these robotic devices. The proposed approach is for the coordinated control of the exoskeleton robots with a robotic walker and adaptive to different walking speeds, which may inspires more extended coordinated control strategies for human-exoskeleton systems in more gait training applications. The proposed approach is also potential for the coordinated motion planning of the other human-centered assistance robots, such as the wheeled walking assistance robots for the elderly.

6D object pose estimation plays a crucial role in robot grasping and manipulation. However, the prevalent methods for 6D object pose estimation heavily rely on 6D annotated data to train deep neural networks, which poses challenges due to the difficulty in obtaining sufficient pose annotations. To address this limitation, this paper presents a self-supervised pose estimation method based on a novel pixelwise weighted dense fusion architecture. This method allows for direct learning from unannotated RGB-D data facilitated by an Iterative Annotation Resolver. Furthermore, a self-supervised pose refinement method based on joint loss is proposed to enhance the pose estimation accuracy. This refinement method employs a differentiable renderer to construct joint optimization constraints. The experimental results demonstrate that our approach achieves a level of pose estimation accuracy that closely rivals that of supervised methods.

Object pose estimation has extensive applications in various industrial scenarios. However, the heavy reliance on dense 6-D annotation and textured object models has become a significant obstacle to the widespread industrial application of 6-D object pose estimation methods. In this work, we presentSS-Pose, a self-supervised learning framework for estimating 6-D object poses without annotated 6-D data and textured model.SS-Poseproposes thecoordinate system datum reinitializerstage to dynamically establish a sequence-level pose representation datum, and thetemporal–spatial constraint resolvermodule to obtain the self-supervised learning target through interframe constraints. We introduce a one-shotcross-coordinate transformationthat establishes the relationship between the 6-D representation and the object poses, which can be further utilized in real-world tasks. We evaluated the proposedSS-Poseon the challenging YCB-Video dataset and texture-less T-LESS dataset. Our approach achieves competitive performance with significantly lower data dependency, making it suitable for visual perception in industrial applications.

Estimating the center of mass (CoM) is essential for both gait planning and controlling of lower limb exoskeletons. Different from CoM estimation in human and humanoid robots, a critical issue in human-exoskeleton systems pis how to describe the effect of physical human-exoskeleton interactions in estimating the CoM of lower limb exoskeletons. This paper presents a novel center of mass estimation method Physically Coupled Serial Chain (PCSC) for human-coupled lower limb exoskeleton systems. Different from traditional serial chain methods, the proposed PCSC involves physical human-exoskeleton models to describe physical interactions between the pilot and the lower limb exoskeleton. We demonstrated the effectiveness of proposed PCSC model in the AIDER lower limb exoskeleton system. Experimental results indicate that the proposed PCSC model is more accuracy than traditional serial chain methods.

Lower Limb Exoskeletons (LLEs) are promising in gait rehabilitation for stroke survivors. In gait training of post-stroke patients with LLEs, one of the main challenges is how to generate appropriate gait patterns from the sound leg to the paretic leg for different patients with varying walking speeds. In this paper, we proposed a Synergetic Gait Prediction (SGP) model for rehabilitation LLEs with post-stroke patients, which can generate adaptive synergetic gait patterns for different patients with varying walking speeds. The proposed SGP model is based on Sequence-to-Sequence (Seq2Seq) neural networks with temporal attention mechanisms. In the training procedure of the proposed SGP model, a gait database with collected gait patterns from healthy subjects is employed to learn the parameters of SGP model. The SGP model takes current joint angles from the sound leg and a segment of observed history joint angles from both legs as input and predicts the future joint angles for the paretic leg. We compared the effectiveness of the SGP model with the Long Short Term Memory (LSTM) model, experimental results indicate that SGP model can generate synergetic gait patterns for different subjects via varying walking speeds with less prediction error.

In recent years, lower limb exoskeletons have gained considerable interest in applications of walking assistance for paraplegic patients. In daily lives, the exoskeleton should have the ability to help the patients to walk over different terrains. For sloped terrains, how to plan the stepping locations on slopes with different gradients and generate stable human-like gaits for patients is a critical issue. In this article, we proposed a slope gradient estimator (SGE) based on the sensor data fusion of the exoskeleton and combined SGE with the capture point theory and dynamic movement primitives (DMP) to construct an adaptive gait planning approach for slopes. After learning from demonstrated gaits sampled from healthy subjects, adaptive gait trajectories can be reproduced online to adapt to slopes with different gradients. The efficiency of the proposed approach was demonstrated on an exoskeleton system named AIDER. Experimental results indicate that the proposed approach can endow exoskeletons with the ability to generate appropriate gaits for different slopes. Note to Practitioners-For lower limb exoskeletons, it is a vital problem to plan the gait for sloped terrains. Considering different gradients among slopes, fixed predefined gait planning cannot cover all cases; thus, a slope gradient adaptive gait planning approach is necessary. The slope gradient estimator proposed in this article provides a possible slope gradient estimation method for exoskeletons or humanoid bipedal robots; it is easy to estimate the slope gradient only based on the local sensor data of the robot. The proposed dynamic gait generator provides lower limb exoskeletons and humanoid bipedal robots a possible adaptive gait planning framework and some flexibility for different slopes. The proposed approach may inspire more extended gait planning strategies for other terrains, such as stairs.

Lower-limb exoskeleton has gained considerable interests in walking assistance applications for paraplegic patients. In walking assistance of paraplegic patients, the exoskeleton should have the ability to help patients to walk over different terrains in the daily life, such as slope terrains. One critical issue is how to plan the stepping locations on slopes with different gradients, and generate stable and human-like gaits for patients. This paper proposed an adaptive gait planning approach which can generate gait trajectories adapt to slopes with different gradients for lower-limb walking assistance exoskeletons. We modeled the human-exoskeleton system as a 2D Linear Inverted Pendulum Model (2D-LIPM) with an external force in the two-dimensional sagittal plane, and proposed a Dynamic Gait Generator (DGG) based on an extension of the conventional Capture Point (CP) theory and Dynamic Movement Primitives (DMPs). The proposed approach can dynamically generate reference foot locations for each step on slopes, and human-like adaptive gait trajectories can be reproduced after the learning from demonstrated trajectories that sampled from level ground walking of normal healthy human. We demonstrated the efficiency of the proposed approach on both the Gazebo simulation platform and an exoskeleton named AIDER. Experimental results indicate that the proposed approach is able to provide the ability for exoskeletons to generate appropriate gaits adapt to slopes with different gradients.

Lower exoskeleton has gained considerable interests in walking assistance applications for both paraplegia and hemiplegia patients. In walking assistance of hemiplegia patients, the exoskeleton should have the ability to control the affected leg to follow the unaffected leg's motion naturally. One critical issue of walking assistance for hemiplegia patients is how to adapt the controller of both lower limbs with different patients. This paper presents a novel learning-based walking assistance control strategy for lower exoskeleton with hemiplegia patients. In the proposed control strategy, we modeled the control system of lower exoskeleton with hemiplegia patient as a Leader-Follower Multi-Agent System (LF -MAS). In order to adapt different patients with different conditions, reinforcement learning framework is utilized to adapt controllers online. In reinforcement learning framework with LF-MAS, we employed a Policy Iteration Adaptive Dynamic Programming (PI-ADP) algorithm, which aims to achieve better tracking control performance for lower exoskeleton with hemiplegia patient. We demonstrate the efficiency of proposed learning-based walking assistance control strategy in an exoskeleton system with healthy subjects who simulate hemiplegia patients. Experimental results indicate that the proposed control strategy can adapt different pilots with good tracking performance.