Myoelectric prosthetic hands are controlled by surface electromyographic signals from residual muscles in amputees. Recognizing continuous finger movements is crucial for improving dexterity when switching tasks. However, existing finger motion decoding algorithms are limited to recognizing discrete static gestures and are unable to cope with the continuous state switching of fine finger movements. To overcome the challenge, a hybrid fuzzy temporal transformer network is proposed. The network integrates temporal convolutional networks with transformers to extract temporal features of motor unit action potentials. It addresses the challenge of identifying waveform changes during gesture switching. An integrated fuzzy decision layer dynamically adjusts the recognition thresholds for static and switching states to suppress noise interference caused by muscle fatigue. By analyzing the impact of switching duration on waveform distortion, the network achieves duration detection for finger movement switching, enabling prediction of target gestures to enhance operational continuity and dexterity. The experiment involved 10 subjects switching between their thumb, index finger, and little finger for different durations. The results show that the correlation between finger switching recognition and gesture targets is 0.859, which is at least 0.026 above that of the baseline methods, and an F1-score at least 0.069 higher. The detection error for switching duration is reduced to a minimum of 80 ms. This work provides a feasible solution for controlling dexterous prostheses in temporal movement switching tasks. Through the collaborative design of hybrid architecture, it paves the way for novel solutions in the practical application of myoelectric prosthetic hand dexterous control.

Enhancing control precision, mitigating external disturbances, and ensuring real-time responsiveness stand as the cornerstone of autonomous vehicle tracking endeavors, each of which intricately interwoven to uphold operational safety. In pursuit of addressing these issues, this paper presents a triple iterative control method inspired by approximate dynamic programming (ADP) tailored for real-time disturbance avoidance. The control framework orchestrates simultaneous iterations of value function, control policy, and disturbance policy, engineered to optimize tracking control amidst external disturbances cast as a zero-sum differential game, tackled adeptly through deep neural networks. Rigorous mathematical proof underpins its triple iteration, coupled with assurances of residual error convergence, solidifying its safety guarantee ability and algorithmic resilience. To validate its effectiveness, both numerical simulations and experiments on a real micro-vehicle platform were conducted. Results underscore the feasibility of this new method, showcasing its energy-saving capability and a four-times acceleration compared to conventional model predictive control (MPC) approaches when confronted with lateral disturbances. Notably, the single-step calculation time of this method on the Raspberry Pi is only 1.44ms, affirming its practical viability and real-world applicability.

Image compression is increasingly important in applications like intelligent driving and smart surveillance systems. This study presents a novel cross view capture distributed image compression network (CVCDIC) to improve the compression quality by using decoder side information. The CVCDIC’s decoder utilizes feature extraction networks to extract features from both the primary image and the side information. Furthermore, a multi-level cross view attention module is designed to capture interrelated details between images at multiple hierarchical levels. Finally, a spatial refinement module, constructed on the foundation of information distillation networks, is designed to further refine the quality of reconstructed images. The results show that CVCDIC can achieve an MS-SSIM of 0.978 at 0.15 bpp, surpassing DSIN (0.925), NDIC (0.956), and ATN (0.955) on the KITTI Stereo dataset.

Upper limb amputees face significant challenges in their daily lives due to the loss of hand or arm functionality. Researchers have developed upper limb prostheses to restore normal hand movements for them. Most hand movement recognition systems of prostheses use electromyography (EMG) as the input signal source, but ignore the interrelationship with electroencephalography (EEG), which may contain valuable movement-related information as well. In order to enhance the accuracy of hand movement classification, we proposed a hand movement recognition system based on a one-dimensional convolutional neural network (1D-CNN) that combines EEG and EMG as the input signal sources to increase the quantity of accessible information. In this work, we collected the EEG and EMG of five subjects during the hand movements and used a 1D-CNN based model to classify the preprocessed signals. The average accuracy of using EEG-EMG fusion is 96.59±2.63%, significantly higher than 74.99±8.24% of using single EEG and 90.31±7.16% of using single EMG. Then, we applied the model trained by offline experiment for online recognition, and controlled the Pepper robot to complete the corresponding hand movements. The average accuracy of online recognition can reach 93.00±4.85% by using majority voting method. The results indicate that the method of EEG-EMG fusion can effectively enhance the performance of hand movement recognition system, which promote the development of upper limb prostheses and contribute to the rehabilitation of upper limb amputees.

This manuscript presents a novel approach for decoding pre-movement patterns from brain signals using a two-stage-training temporal–spectral neural network (TTSNet). The TTSNet employs a combination of filter bank task-related component analysis (FBTRCA) and convolutional neural network (CNN) techniques to enhance the classification of single-upper limb movements in non-invasive brain–computer interfaces (BCIs). In our previous work, we introduced the FBTRCA method which utilized filter banks and spatial filters to handle spectral and spatial information, respectively. However, we observed limitations in the temporal decoding phase, where correlation features failed to effectively utilize temporal information because of misaligned onset and noisy spikes. To address this issue, our proposed method focuses on analyzing multi-channel signals in the temporal–spectral domain. The TTSNet first divides the signals into various filter banks, employing task-related component analysis to reduce dimensionality and eliminate noise, respectively. Subsequently, a CNN is employed to optimize the temporal characteristics of the signals and extract class-related features. Finally, the class-related features from all filter banks are concatenated and classified using the fully connected layer. To evaluate the effectiveness of our proposed method, we conducted experiments on two publicly available datasets. In binary classification tasks, the TTSNet achieved an improved accuracy of 0.7707 ± 0.1168, surpassing the performance of EEGNet (accuracy: 0.7340 ± 0.1246) and FBTRCA (accuracy: 0.7487 ± 0.1250). In multi-class tasks, TTSNet achieved an accuracy of 0.4588 ± 0.0724, exhibiting a 4.27% and 3.95% accuracy increase over EEGNet and FBTRCA, respectively. The findings of this study suggest that the proposed TTSNet method holds promise for detecting limb movements and assisting in the rehabilitation of stroke patients. The classification of single-side limb movements is expected to facilitate the interaction between patients and external environment by increasing the number of control commands in BCIs.

The classification of limb movements can provide with control commands in non-invasive brain-computer interface. Previous studies on the classification of limb movements have focused on the classification of left/right limbs; however, the classification of different types of upper limb movements has often been ignored despite that it provides more active-evoked control commands in the brain-computer interface. Nevertheless, few machine learning method can be used as the state-of-the-art method in the multi-class classification of limb movements. This work focuses on the multi-class classification of upper limb movements and proposes the multi-class filter bank task-related component analysis (mFBTRCA) method, which consists of three steps: spatial filtering, similarity measuring and filter bank selection. The spatial filter, namely the task-related component analysis, is first used to remove noise from EEG signals. The canonical correlation measures the similarity of the spatial-filtered signals and is used for feature extraction. The correlation features are extracted from multiple low-frequency filter banks. The minimum-redundancy maximum-relevance selects the essential features from all the correlation features, and finally, the support vector machine is used to classify the selected features. The proposed method compared against previously used models is evaluated using two datasets. mFBTRCA achieved a classification accuracy of 0.4193 ± 0.0780 (7 classes) and 0.4032 ± 0.0714 (5 classes), respectively, which improves on the best accuracies achieved using the compared methods (0.3590 ± 0.0645 and 0.3159 ± 0.0736, respectively). The proposed method is expected to provide more control commands in the applications of non-invasive brain-computer interfaces.

The convolutional neural network (CNN) has emerged as a powerful tool for decoding electroencephalogram (EEG), which owns the potential use in the event-related potential-based brain-computer interface (ERP-BCI). However, the intra-individual difference of ERP makes the traditional learning models trained on static EEG data hard to decode when the EEG features vary along the time, which limits the long-time performance of the model. Addressing this problem, this study proposes a three-dimension CNN (3D-CNN)-based model to decode the ERPs dynamically. As input, the EEG is transformed into a brain topographic map stream along time. Then the 3D-CNN applies three-dimension kernels to capture the dynamical characteristic of spatial feature at several time points. Ten subjects participated in a cross-time task for 6 or 12[Formula: see text]h. The 3D-CNN shows higher accuracies and shorter computational cost than the baseline models of the 2D-CNN, the long short term memory (LSTM), the back propagation (BP), and the fisher linear discriminant analysis (FLDA) when detecting the ERPs. In addition, four schemes of the 3D-CNN are compared to explore the influence of the structure on the performance. This result demonstrates advanced robustness of the 3D-CNN kernel to the intra-individual EEG difference, helping to launch a more practical EEG decoding model for a long-time use.

Surface electromyography (sEMG) signals have been applied widely in prosthetic hand controlling. In the sEMG signal acquisition, wireless devices bring convenience, but also introduce signal missing due to interference or failure during data transmission. The missing signal may only last for tens of milliseconds, but have a great impact on the recognition. Researchers have employed various methods to complete missing sEMG data, but the completed signal may not totally fit the origins, and more extra calculation time will be spent. When recognizing hand gestures by sEMG from few sensors, to recognize the slightly or not serious signal missing, this study proposed a data split reorganization (DSR) strategy and a weight-based multiple neural network voting (WMV) method. To validate the proposed methods, controllable missing sEMG signals are generated artificially. Three time domain features are extracted based on non-overlapping sliding windows. The DSR is employed to make full use of the features, and then the WMV is utilized to recognize them. Nine subjects participated in the experiments, and the results indicate that the accuracy of the proposed methods is higher. For 5%, 10%, and 15% data missing ratios, the accuracy is 93.66%, 92.55%, and 91.19%, respectively. The Wilcoxon signed-rank test also demonstrates that these results are significantly superior to the situations in which the proposed methods are not applied. In the future, we will optimize the proposed methods to recognize the seriously missing sEMG signal.

Image restoration with corruptions from combined multiple types of distortion is a challenging and practical problem. Recent studies show that a promising technique is to perform parallel “operations” to handle different types of distortion, which can be modeled by a deep neural network framework. However, the reconstruction may be dominated by a small number of operations due to the heterogeneous features generated by different operations. To handle this issue, we introduce a tensor 1×1 convolutional layer by leveraging high-order tensor fusion, which can not only harmonize the heterogeneous features but also take high order statistical information into account. To efficiently learn the large-scale kernel tensor resulted from the tensor product, we employ tensor network to represent kernels, which is able to convert the exponential growth of the dimension to linear growth. Armed with this new layer, we propose high-order operation-wise attention network for the task of multi-distorted image restoration. The experimental results demonstrated that the proposed method outperforms the method with vanilla 1 × 1 convolutional layer in several typical tasks and is promising for more difficult tasks. Code is available at https://github.com/ZihaoH/High-order-OWAN.

In the growing elderly population globally, patients with severe movement disorders account for a large proportion. Moreover, the development of intelligent service equipment can better assist them in their daily. This paper proposes a new service robot control system. The brain-computer interface (BCI) based on Steady-State Visual Evoked Potentials (SSVEP) is used to acquire and process electroencephalogram(EEG) signals and output various control commands accordingly. Then, considering the visual fatigue of SSVEP-BCI, we added an object detection method based on Yolov3-tiny and saliency prediction to identify the patient’s selection intention intelligently. The results show that the subject can successfully complete the object delivery task with an average accuracy of 90.3%. The proposed control system can help the patients control a service robot in a more intelligent and friendly way to realize some daily tasks.

Empirical mode decomposition (EMD) has developed into a prominent tool for adaptive, scale-based signal analysis in various fields like robotics, security and biomedical engineering. Since the dramatic increase in amount of data puts forward higher requirements for the capability of real-time signal analysis, it is difficult for existing EMD and its variants to trade off the growth of data dimension and the speed of signal analysis. In order to decompose multi-dimensional signals at a faster speed, we present a novel signal-serialization method (serial-EMD), which concatenates multi-variate or multi-dimensional signals into a one-dimensional signal and uses various one-dimensional EMD algorithms to decompose it. To verify the effects of the proposed method, synthetic multi-variate time series, artificial 2D images with various textures and real-world facial images are tested. Compared with existing multi-EMD algorithms, the decomposition time becomes significantly reduced. In addition, the results of facial recognition with Intrinsic Mode Functions (IMFs) extracted using our method can achieve a higher accuracy than those obtained by existing multi-EMD algorithms, which demonstrates the superior performance of our method in terms of the quality of IMFs. Furthermore, this method can provide a new perspective to optimize the existing EMD algorithms, that is, transforming the structure of the input signal rather than being constrained by developing envelope computation techniques or signal decomposition methods. In summary, the study suggests that the serial-EMD technique is a highly competitive and fast alternative for multi-dimensional signal analysis.

Planes and edges are attractive features for simultaneous localization and mapping (SLAM) in indoor environments because they can be reliably extracted and are robust to illumination changes. However, it remains a challenging problem to seamlessly fuse two different kinds of features to avoid degeneracy and accurately estimate the camera motion. In this article, a plane-edge-SLAM system using an RGB-D sensor is developed to address the seamless fusion of planes and edges. Constraint analysis is first performed to obtain a quantitative measure of how the planes constrain the camera motion estimation. Then, using the results of the constraint analysis, an adaptive weighting algorithm is elaborately designed to achieve seamless fusion. Through the fusion of planes and edges, the solution to motion estimation is fully constrained, and the problem remains well-posed in all circumstances. In addition, a probabilistic plane fitting algorithm is proposed to fit a plane model to the noisy 3-D points. By exploiting the error model of the depth sensor, the proposed plane fitting is adaptive to various measurement noises corresponding to different depth measurements. As a result, the estimated plane parameters are more accurate and robust to the points with large uncertainties. Compared with the existing plane fitting methods, the proposed method definitely benefits the performance of motion estimation. The results of extensive experiments on public data sets and in real-world indoor scenes demonstrate that the plane-edge-SLAM system can achieve high accuracy and robustness.Note to Practitioners—This article is motivated by the robust localization and mapping for mobile robots. We suggest a novel simultaneous localization and mapping (SLAM) approach fusing the plane and edge features in indoor scenes (plane-edge-SLAM). This newly proposed approach works well in the textureless or dark scenes and is robust to the sensor noise. The experiments are carried out in various indoor scenes for mobile robots, and the results demonstrate the robustness and effectiveness of the proposed framework. In future work, we will address the fusion of other high-level features (for example, 3-D lines) and the active exploration of the environments.

Increasing command generation rate of an event-related potential-based brain-robot system is challenging, because of limited information transfer rate of a brain-computer interface system. To improve the rate, we propose a dual stimuli approach that is flashing a robot image and is scanning another robot image simultaneously. Two kinds of event-related potentials, N200 and P300 potentials, evoked in this dual stimuli condition are decoded by a convolutional neural network. Compared with the traditional approaches, this proposed approach significantly improves the online information transfer rate from 23.0 or 17.8 to 39.1 bits/min at an accuracy of 91.7%. These results suggest that combining multiple types of stimuli to evoke distinguishable ERPs might be a promising direction to improve the command generation rate in the brain-computer interface.

Brain-Machine Interface (BMI) has emerged as a powerful tool for assisting disabled people and for augmenting human performance. Up so far, no studies have succeeded in the power augmentation for the multi-DOFs robot based on EEG signals, especially for the complex shoulder joint. In this work, we propose an electromyography (EMG) estimation method based on electroencephalography (EEG) signals to realize the power assistance. The positions of the electrodes where the motion information of shoulder joint is effectively and exactly extracted are discussed, and a linear model that correlates the EMG to the EEG signal is constructed utilizing motion-related features extracted from multi-location EEG measurements. The constructed model is used to estimate the human muscular activity of shoulder joint from EEG using Principal Component Analysis (PCA) method. The proposed approach is experimentally verified, and an average correlation coefficients are as high as about 0.90 for different subjects are obtained between the estimated and the actually measured EMG signal. Our results suggest that the estimation of EMG based on EEG is feasible. This demonstrates the potential of using EEG signals to support human activities via brain-machine interface.

Brain-computer interface (BCI) systems can translate the human mind into control commands, which makes it feasible to improve the life quality of physically challenged people. However, in real-life situations, it is still difficult for users to utilize robots to provide basic services with BCI systems. We aimed to propose a BCI-based system with a visual servo module to operate a service robot. We recorded single-channel steady-state visual evoked potentials (SSVEP) as input signals for the BCI system of this study. The visual stimuli for inducing SSVEP were modulated at seven different frequencies with the sampled sinusoidal method. Correspondingly, this SSVEP-based BCI system can generate seven control commands for the operation of the service robot, which can provide three fundamental services: mobility, manipulation, and delivery. The visual servo module was established to reduce the burden of users and accelerate service procedures. To evaluate the performance of this system, subjects were recruited to participate in the experiments. All the participants succeed in operating the robot to provide the basic services. According to the experimental results, this SSVEP-based BCI system that incorporates the visual servo module can be effectively used to operate service robots with reduced number of channels and increased ability to perform multiple tasks.

BACKGROUND: Predicting disease-associated genes is helpful for understanding the molecular mechanisms during the disease progression. Since the pathological mechanisms of neurodegenerative diseases are very complex, traditional statistic-based methods are not suitable for identifying key genes related to the disease development. Recent studies have shown that the computational models with deep structure can learn automatically the features of biological data, which is useful for exploring the characteristics of gene expression during the disease progression. RESULTS: In this paper, we propose a deep learning approach based on the restricted Boltzmann machine to analyze the RNA-seq data of Huntington's disease, namely stacked restricted Boltzmann machine (SRBM). According to the SRBM, we also design a novel framework to screen the key genes during the Huntington's disease development. In this work, we assume that the effects of regulatory factors can be captured by the hierarchical structure and narrow hidden layers of the SRBM. First, we select disease-associated factors with different time period datasets according to the differentially activated neurons in hidden layers. Then, we select disease-associated genes according to the changes of the gene energy in SRBM at different time periods. CONCLUSIONS: The experimental results demonstrate that SRBM can detect the important information for differential analysis of time series gene expression datasets. The identification accuracy of the disease-associated genes is improved to some extent using the novel framework. Moreover, the prediction precision of disease-associated genes for top ranking genes using SRBM is effectively improved compared with that of the state of the art methods.

Although a cellular manufacturing system can produce diversified products flexibly, its assembly efficiency is mainly limited by the operators' assembly performance. To improve the operators' assembly performance without longtime training, an assembly-skill-transferring method was proposed to realize the assembly skill transfer process from skilled operators to novice operators. In the cellular manufacturing system, the assembly skills consist of cognition skills and motor skills. Taking a “Peg-in-Hole Task” as an example, the effect of the proposed assembly transfer method was verified for motor skills. The results show that the proposed assembly-skill-transferring system can improve the novice operators' assembly performance greatly.

A cellular manufacturing system is good at producing diversified products flexibly; however, its assembly efficiency depends mainly on its operators' abilities. As the workforce shrinks in Japan, cellular manufacturing systems are difficult to maintain. In this case, a new assembly system has been developed since 2006 that combines both the dexterities of human operators and the advantages of automatic machinery. Its characteristics consist of three aspects: collaboration between an operator and twin manipulators on a mobile base, assembly information guidance, and safe design for collaboration. To meet the rapid changing tastes of customers, operators have to assemble various products without longtime training. This requires an effective assembly skill transfer system to extract assembly skills from skilled operators, and then transfer them to novice ones. Considering the characteristics of a cellular manufacturing system, an assembly skill transfer system was proposed and used to extract and transfer assembly skills in both cognition and execution aspects. Taking a cable harness task as an example, the proposed skill transfer system was applied in a developed assembly system. The results show that working under the developed assembly system with physical support, informational support, and the assembly skill transfer system, novice operators' assembly performance was greatly improved. This verified the effect of the proposed solution to maintain the cellular manufacturing system in the aging Japanese society.

The objective of this work is to enhance man-machine collaboration in cell production by task modeling approach. Task modeling provides a method to define and study the collaboration between man and machine. Six requirements have been derived as basis of development. In this work, the task modeling has extended classical HTA method to represent collaborative operation in a hierarchical structure with the classification of operation task levels and extended operation details as task properties. A modeling tool is developed to provide a development and data management platform. The man-machine collaboration is well defined in a structural format by the ability of task analysis method to address human task in a more flexible way. The collaboration modeling and operation resources information have significantly facilitated operation planning in both assembly and control levels. From the case study, the improvement in safety aspect has proven the ability of this approach in assisting safety design and development of production system. The integration with operation control system has shown the execution potential in real-time operation. The unique development of operation data management has expanded this work into operation information support. Through the multimodal information support system, the human operator is well guided by the operation information corresponded to the task components. The operation information also can work as evaluation functions to measure the human working performance and system performance as a whole.

The challenge of this work is to study the design and development of human-robot collaboration (HRC) in cellular manufacturing. Based on the concept of human collaborative design, four main design factors are being identified and developed in an active HRC prototype production cell for cable harness assembly. Human collaborative design aims to optimize the system design for the advantage of collaboration between human and robots based on human considerations. Task modeling approach is developed to study and analyze the task in order to identify the collaboration tasks to develop the collaboration planning. In the collaboration safety development, five safety designs, cover both hardware and control design, are proposed and developed in the prototype system. Risk assessment is conducted to verify the safety design. Two main experiments were conducted as preliminary study to investigate mental workload in HRC. A multimodal information support system is developed in the study of man-machine interface in this work to provide a comprehensive human-robot interface to facilitate human operator. The system performance evaluation had proven the improvement of prototype production cell with HRC design for cellular manufacturing.

The purpose of this work is to conduct human factors studies in the development of an information support system for human-robot collaborative cellular manufacturing system. Multimedia technologies are being utilized to enhance the support system to function as operational information support and interface between human operator and robot system. In the study, five experiments were carried out in both system hardware and support information developments to investigate the human factors design of the support system. In the information display approach study, comparison of information presentation ability among paper manual, LCD TV and projector was conducted. Similar information was displayed in different visual areas by vertical and horizontal LCD TV configurations to study the effect on human visual working area. The comparison experiment on image, text and voice formats had resulted varying modality effectiveness of information. Different information formats require different human cognitive processes even though the information contents are similar. The investigation on multimedia displays in five combinations of information elements was performed in multimedia display design study. The information support system is implemented in a human-robot collaborative prototype production cell. The cable harness assembly experiment had proven the effectiveness of the information support system in guiding the operators on the assembly operation and also to collaborate well with the robot system.