This article investigates the system modeling problem for the dynamical process of human brain activity in human-robot cognitive interaction (HRCI). An important novelty of the proposed approaches is to build a computational model of a human-distributed robot-lumped parameter system (HDRLPS) that describes the inherent dynamical principle of human brain activity (with spatiotemporal-varying characteristic) undergoing the interaction between the intrinsic cognitive dynamics and extrinsic robot stimuli. A deterministic learning (DL)-based spatiotemporal dynamics identification scheme is proposed to accurately identify the spatiotemporal dynamics of HDRLS and obtain the associated knowledge as a constant radial basis functional neural network (RBF NN) model. A spatiotemporal dynamics estimator is designed with this model, which can accurately evaluate and monitor the dynamical process of human brain activity in real-time HRCI by the generated dynamics-synchronized state. The effectiveness and practicability of the approaches in the dynamics identification and evaluation for the human brain activity in HRCI are validated by the thorough analysis, including the mathematical proof, the simulation study, and the brain-computer interface (BCI) experiment using publicly available datasets. Our method is compared with state-of-the-art (SOTA) methods, such as LGGNet, EEGNet, Tsception, EEG-Deformer, EEG-Transformer, and EEGViT. The results show that our method can outperform these methods with better recognition accuracy and macro- $F1$ scores. The source code can be found at: https://github.com/alonexing/source_code/tree/master.

Magnetic resonance imaging (MRI) is highly important for diagnosing knee injuries because of its ability to provide detailed images. However, the key problem in intelligent MRI diagnosis is how to identify and extract the most relevant features from a large amount of data. In this paper, we present a single training stage method. This method can extract features from images, sequences, and anatomical planes at different levels simultaneously. First, we propose a nested Mamba module. This module allows parallel extraction of image-level and sequence-level features from the same anatomical plane in MRI. Second, an anatomical-level Mamba module is designed. It integrates features extracted from different anatomical planes. Finally, self-distillation is applied to improve the model’s feature extraction efficiency. Experimental results show that our proposed model outperforms sub-optimal models by 3.7% in accuracy and 2.4% in F1-score. Moreover, compared with the multi-stage model, the single-stage multi-class classification model greatly improves classification efficiency. In conclusion, the model developed in this paper can effectively and accurately conduct intelligent knee MRI diagnosis.

Focus of attention is one of the most influential factors facilitating motor training performance. Most of robotic training methods have not well solved the negative effect of divided-attention on motor execution performance, resulting in limited rehabilitation efficiency for motor-cognitive dysfunction. In this study, we propose a novel visuomotor human-robot interaction framework by integrating a gaze-visual game and force-movement robot, to realize more efficient training for both attentional and motor function. An important novelty of this framework is to design a dynamical pattern recognition scheme for the hierarchical-coupled behavior of attentional and motor execution, to facilitate efficient human-robot interaction in both cognitive and motor perspectives. Specifically, an attentional-motor dynamical system modeling method is first developed by using the gaze, force and movement data collected from the human under different attentional-motor behavior. Then, an online dynamical pattern recognition scheme can be design with these models to online recognizing the human’s attentional and motor behavior states. The training robot system can dynamically adjust the parameters according to the recognition results, to guide the collaboration of both attentional and motor training. Experimental study are conducted to demonstrate the desired accuracy and efficiency of our designed approaches in attentional-motor behavior recognition and training.

Motor imagery-based brain–computer interfaces (MI-BCIs) hold significant promise for rehabilitation training in individuals with neurological impairments such as stroke and spinal cord injury (SCI). Achieving precise and robust lower limb movement prediction for each patient is crucial. However, the variability in MI response frequencies and brain activation patterns among subjects presents a great challenge to the generalizability of MI-BCIs. This paper proposes a Tuned Heuristic Fusion Graph Convolutional Network (THFGCN) for limb movement prediction in rehabilitation scenarios. THFGCN innovatively designs a learnable EEG frequency band tuned module and a heuristic space topology module. These two modules allow for the intricate extraction of both frequency and spatial topological features, utilizing graph adjacency matrices that encapsulate channel correlations and spatial relationships, hence fostering individualized analysis and enhanced generalizability across subjects. Furthermore, a spatio-temporal convolution module paired with a feature map attention mechanism is proposed to extract the critical spatio-temporal features of electroencephalogram (EEG) data. Validation experiments on the PhysioNet and LLM-BCImotion datasets against six mainstream methods demonstrate that THFGCN outperforms state-of-theart methods, achieving 88.41% and 82.82% accuracy in the within-subject case, and 65.93% and 60.56% accuracy in the cross-subject case, respectively. Detailed frequency band weight and T-distributed Stochastic Neighbor Embedding visualization validate the effectiveness of proposed modules. Furthermore, feature interpretability analysis proves the extracted features’ profound MI task relevance, underlining THFGCN’s exceptional interpretability.

Gait training with human-like gait patterns can be provided by lower limb exoskeletons (LLEs) for patients with gait impairments. For patients with little effort to keep balance, using a mobile robotic walker to assist gait training with LLEs is an effective way. Since gait patterns are varying with walking speeds, it is a critical issue to coordinated control the robotic walker and the exoskeleton to obtain a natural and human-like walking posture. In this paper, a novel adaptive assistance approach named SQG-OPT is proposed to tackle the problem, which comprises of two parts: the Spatially Quantized Gait (SQG) and the optimization. The SQG generates reference joint angles and reference trajectory of the Center Of Mass (COM) for the human-exoskeleton system in space domain. The optimization part is constructed to convert the reference joint angles from the space domain to the time domain, which is based on the dynamics model of the human-exoskeleton-walker system and adaptive to different walking speeds. The proposed approach has been tested on the robot simulation platform CoppeliaSim, the experimental results indicate that the proposed approach can generate human-like gait patterns for different walking speeds from 0 to 0.8 m/s. Additionally, in comparison with other methods, the proposed approach has a better performance on the movement tracking of the COM for a natural walking posture.Note to Practitioners—The coordinated control is important for the exoskeleton robot with a mobile robotic walker, one of the potential challenge is the adaptive coordinated motion planning for these robotic devices. The proposed approach is for the coordinated control of the exoskeleton robots with a robotic walker and adaptive to different walking speeds, which may inspires more extended coordinated control strategies for human-exoskeleton systems in more gait training applications. The proposed approach is also potential for the coordinated motion planning of the other human-centered assistance robots, such as the wheeled walking assistance robots for the elderly.

6D object pose estimation plays a crucial role in robot grasping and manipulation. However, the prevalent methods for 6D object pose estimation heavily rely on 6D annotated data to train deep neural networks, which poses challenges due to the difficulty in obtaining sufficient pose annotations. To address this limitation, this paper presents a self-supervised pose estimation method based on a novel pixelwise weighted dense fusion architecture. This method allows for direct learning from unannotated RGB-D data facilitated by an Iterative Annotation Resolver. Furthermore, a self-supervised pose refinement method based on joint loss is proposed to enhance the pose estimation accuracy. This refinement method employs a differentiable renderer to construct joint optimization constraints. The experimental results demonstrate that our approach achieves a level of pose estimation accuracy that closely rivals that of supervised methods.

Object pose estimation has extensive applications in various industrial scenarios. However, the heavy reliance on dense 6-D annotation and textured object models has become a significant obstacle to the widespread industrial application of 6-D object pose estimation methods. In this work, we presentSS-Pose, a self-supervised learning framework for estimating 6-D object poses without annotated 6-D data and textured model.SS-Poseproposes thecoordinate system datum reinitializerstage to dynamically establish a sequence-level pose representation datum, and thetemporal–spatial constraint resolvermodule to obtain the self-supervised learning target through interframe constraints. We introduce a one-shotcross-coordinate transformationthat establishes the relationship between the 6-D representation and the object poses, which can be further utilized in real-world tasks. We evaluated the proposedSS-Poseon the challenging YCB-Video dataset and texture-less T-LESS dataset. Our approach achieves competitive performance with significantly lower data dependency, making it suitable for visual perception in industrial applications.

Despite the advances in the field of human-robot interface (HRI) based on biological neural signal, the use of the sole electroencephalography (EEG) signal to help robotic exoskeleton predict the limb movement is currently no mature in rehabilitation training, due to its unreliability. Multimodal HRI represents a very recent solution to enhance the performance of single modal HRI. These HRI normally include the EEG signal with surface electromyography (sEMG) signal. However, their use for the lower limb movement prediction in hemiplegia is still limited, and the deep fusion feature of sEMG and EEG signal is ignored. This paper proposes a Dense co-attention mechanism-based Multimodal Enhance fusion Network (DMEFNet) for the lower limb movement prediction in hemiplegia. The DMEFNet can realize the mapping and deep fusion between the sEMG and EEG signal features and get a high accuracy movement prediction of the lower limbs. A sEMG and EEG data acquisition experiment and an incomplete asynchronous data collection paradigm are designed to verify the effectiveness of DMEFNet. The experimental results show that DMEFNet has a good movement prediction performance in both within-subject and cross-subject situations, reaching an accuracy of 82.96% and 88.44% respectively.

6D object pose estimation is an essential task in vision-based robotic grasping and manipulation. Prior works extract spatial features by fusing the RGB image and depth without considering the temporal motion information, limiting their performance in heavy occlusion robotic grasping scenarios. In this paper, we present an end-to-end model named TemporalFusion, which integrates the temporal motion information from RGB-D images for 6D object pose estimation. The core of proposed TemporalFusion model is to embed and fuse the temporal motion information from multi-frame RGB-D sequences, which could handle heavy occlusion in robotic grasping tasks. Furthermore, the proposed deep model can also obtain stable pose sequences, which is essential for real-time robotic grasping tasks. We evaluated the proposed method in the YCB-Video dataset, and experimental results show our model outperforms state-of-the-art approaches. Our code is available at https://github.com/mufengjun260/TemporalFusion21.