Accomplishing a plug-and-play steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) remains a critical challenge, due to the unsatisfying performance of calibration-free decoding algorithms.A current method called online adaptive canonical correlation analysis (OACCA) has proved efficient in enhancing calibration-free performance by self-adaptation merely with online data.However, OACCA only concerns the adaptation of spatial filters and excludes other useful adaptive procedures like individual template estimation, hindering fully exploitable model decoding and adaptation. This study proposes a new online adaptation framework termed online adaptive extended correlation analysis (OAECA) to augment the calibration-free online adaptation loop. OAECA first recalls and cleans the online trials for reliable data learning, then tunes individual templates and spatial filters for complete model updating, and finally adopts extended feature matching to improve target recognition. The simulation results on two public SSVEP datasets revealed that OAECA significantly outperformed OACCA for almost all 105 subjects, and both offline and online experiments further confirmed the effectiveness of OAECA. Particularly, OAECA achieved the highest average information transfer rate (ITR) of 202.17 bits/min in the online experiment, significantly exceeding the state-of-the-art OACCA of 177.02 bits/min. This study enhanced the calibration-free performance through comprehensive online adaptation, hopefully advancing SSVEP-based BCIs toward practical plug-and-play real-world applications.

Objective: Because of the non-stationary nature of electroencephalogram (EEG) signals, traditional non-invasive brain-computer interfaces (BCIs) usually only produce discrete commands, limiting their ability to control external devices continuously. This study proposes a novel BCI control strategy mapping multiple discrete commands to continuous motion, enabling real-time manipulation of a drone in four degrees of freedom (DOF).Methods: Our strategy used the fast steady state visual evoked potential (SSVEP) encoding and decoding method to convert user intentions into the drone’s flight status in near real-time. Simultaneously, the drone’s live video was embedded into the SSVEP stimuli, providing users with a first-person perspective control experience.Results: In drone control experiments, participants successfully maneuvered the drone through complex path-following tasks in simulated and physical scenarios. The mean flight trajectory bias ratio was measured as 0.81, with a mean flight smoothness of -3.31 (measured by spectral arc length) and mean Fitts’s throughput of 9.18 bits/min. Notably, the brain-to-hand ratio (BHR) for all metrics approached 1, indicating that our non-invasive control system achieved comparable performance to manual control systems.Conclusion: These results suggest the effectiveness of our proposed BCI control strategy that maps discrete commands to continuous motion and extends the capabilities of non-invasive BCIs in continuous control scenarios.Significance: This study significantly advances the applications of BCI and propels human-machine interaction towards a more direct realm.

Objective: Emergency anticipation (EA) refers to the brain’s rapid perceptual, cognitive, and motor preparation in response to imminent emergencies. Timely decoding of EA can facilitate proactive responses before full behavioral execution, which is critical in real-world scenarios such as avoiding hazards or mitigating accidents. However, the cortical activation underlying the EA process has not been fully explored. This study aims to analyze the neural activity of the EA process and explore the feasibility of detecting emergency motor intention in conjunction with brain-computer interface (BCI) technology. Methods: We designed a new emergency state induction paradigm in the virtual environment, including a target task (emergency anticipation, EA) and two baseline tasks (emergency anticipation execution, EAE, visual observation, VO). A total of 31 healthy subjects were recruited for the offline experiment. The cortical responses during the EA process were quantified by analyzing event-related potential, movement-related cortical potential, and event-related spectral perturbation. Discriminative canonical pattern matching, common spatial patterns, and shrinkage linear discriminant analysis were employed to perform binary classification. Six subjects participated in the pseudo-online asynchronous experiment to valid the feasibility of identifying emergency motor intention. Results: The results showed that the cascading process associated with EA existed in both the temporal and spectral domains. Particularly, temporal domain feature demonstrated superior classification performance, with averages of 90.13% (>80% chance level). The pseudo-online evaluation showed that the system response time with an average of 257.12 ms, which was 35 ms faster than the behavioral response. Significance: Our work demonstrated the cascading process of perceptual recognition, cognitive evaluation, and motor preparation during the EA processes and provided preliminary evidence supporting the feasibility of detecting emergency motor intentions. These findings lay a theoretical foundation for extending the application of BCI technology to rapid control scenarios.

Detecting arm movement direction is significant for individuals with upper-limb motor disabilities to restore independent self-care abilities. It involves accurately decoding the fine movement patterns of the arm, which has become feasible using invasive brain-computer interfaces (BCIs). However, it is still a significant challenge for traditional electroencephalography (EEG) based BCIs to decode multi-directional arm movements effectively. This study designed an ultra-high-density (UHD) EEG system to decode multi-directional arm movements. The system contains 200 electrodes with an interval of about 4 mm. We analyzed the patterns of the UHD EEG signals induced by arm movements in different directions. To extract discriminative features from UHD EEG, we proposed a spatial filtering method combining principal component analysis (PCA) and discriminative spatial pattern (DSP). We collected EEG signals from five healthy subjects (two left-handed and three right-handed) to verify the system's feasibility. The movement-related cortical potentials (MRCPs) showed a certain degree of separability both in waveforms and spatial patterns for arm movements in different directions. This study achieved an average classification accuracy of 63.15 (8.71)% for both arms (eight-class task) with a peak accuracy of 77.24%. For the dominant arm (four-class task), we obtained an average accuracy of 75.31 (9.21)% with a peak accuracy of 85.00%. For the first time, this study simultaneously decodes multi-directional movements of both arms using UHD EEG. This study provides a promising approach for detecting information about arm movement directions, which is significant for the development of BCIs.

Currently, spatiotemporal convolutional neural networks (CNNs) for electroencephalogram (EEG) signals have emerged as promising tools for seizure prediction (SP), which explore the spatiotemporal biomarkers in an epileptic brain. Generally, these CNNs capture spatiotemporal features at single spectral resolution. However, epileptiform EEG signals contain irregular neural oscillations of different frequencies in different brain regions. Therefore, it may be underperforming and uninterpretable for the CNNs without capturing complex spectral properties sufficiently. This study proposed a novel interpretable multi-branch architecture for spatiotemporal CNNs, namely MultiSincNet. On the one hand, the MultiSincNet could directly show the frequency boundaries using the interpretable sinc-convolution layers. On the other hand, it could extract and integrate multiple spatiotemporal features across varying spectral resolutions using parallel branches. Moreover, we also constructed a post-hoc explanation technique for multi-branch CNNs, using the first- order Taylor expansion and chain rule based on the multivariate composite function, which demonstrates the crucial spatiotemporal features learned by the proposed multi-branch spatiotemporal CNN. When combined with the optimal MultiSincNet, ShallowConvNet, DeepConvNet, and EEGWaveNet had significantly improved the subject-specific performance on most metrics. Specifically, the optimal MultiSincNet significantly increased the average accuracy, sensitivity, specificity, binary F1-score, weighted F1-score, and AUC of EEGWaveNet by about 7%, 8%, 7%, 8%, 7%, and 7%, respectively. Besides, the visualization results showed that the optimal model mainly extracts the spectral energy difference from the high gamma band focalized to specific spatial areas as the dominant spatiotemporal EEG feature.

Emotion is an intrinsic property of human beings and emotion recognition in conversations (ERC) contributes to developing human-like machines. To fuse different modality information effectively is the key part for multimodal ERC. Current methodologies employ graph convolution network to fuse multimodal information. They perform well in intra-modal fusion, but poor in cross-modal fusion, resulting a weak integration of multimodal information. To address aforementioned problem, in this paper, we propose a dual graph structure network for emotion recognition in multimodal conversations (DGSNet). Specially, the multimodal fusion mechanism based on the dual graph structure network is designed. The heterogeneity features of each modal are extracted through the separated graph and the complementary features of each modal are extracted through the aggregation graph. Then, the local attention mechanism for emotional dependency is designed to constrain the scope and target of the emotion. It enhances the analysis of emotional dependency. To demonstrate the superior performance of our proposed method, we evaluate it on two benchmarks, IEMOCAP and MELD, and the experimental results show that the DGSNet model can fuse multimodal information effectively and improve the performance of emotion recognition.

Emotion recognition in conversation (ERC) aims to detect the emotion in a conversation, which has drawn increasing interests due to its widely applications. Current methodologies mainly endeavor to capture a good representation of conversation context. However, we argue that the conversation context are not always consistent with the emotion evolution. This incongruity can greatly restrict the recognition performance. To address aforementioned challenges, in this paper, we propose an emotion evolution network for emotion recognition in conversation (E2Net). Specifically, a speaker-aware modeling methodology is firstly constructed to fuse the utterance from conversations. We employ the gated recurrent unit (GRU) encodes the utterance sequentially. For encoding the interaction between speakers, a listener state is introduced to aid in analyzing conversation context. Then, a Transformer-based method is proposed to capture the emotion evolution accompanying with the emotion transformation matrix. To demonstrate the superior performance of our proposed method, extensive experiments are conducted on four REC datasets and the experimental results suggest that our method is effective and outperforms the current state-of-the-art methods on multiple datasets.

Brain-computer interfaces (BCIs) provide users with a direct communication pathway between the brain and the peripheral environment. BCI-controlled devices have the potential to assist disabled patients in regaining motor functions. However, it remains unclear what happens to the functional coupling between the brain and muscle after BCI-based long-term motor training. Therefore, we developed a neurofeedback training method for long-term motor training that combines visual scenes and electrical stimulation. During the experiment, we collected electroencephalography (EEG) and electromyography (EMG) data from 20 subjects to explore their neurophysiological responses and the EEG-EMG coupling relationship. Event-related desynchronization (ERD), root mean square (rms) analysis, transfer entropy (TE) patterns, and other techniques were used to evaluate the cortical muscle response. Compared with the initial states, the ERD and rms significantly improved after long-term motor training. However, there was no significant difference in BCI performance. Directional TE values revealed the cortical muscle mechanism. These results demonstrate that incorporating EEG and EMG patterns to evaluate and establish a BCI-based motor training method is feasible. Furthermore, this article could provide evidence for functional coupling mechanisms for cortical muscles and motor rehabilitation.

Although the inter-subject information has been demonstrated to be effective for a rapid calibration of the P300-based brain-computer interface (BCI), it has never been comprehensively tested to find if the incorporation of heterogeneous data could enhance the accuracy. This study aims to improve the subject-specific P300 classifier by adding other subject's data. A classifier calibration strategy, weighted ensemble learning generic information (WELGI), was developed, in which elementary classifiers were constructed by using both the intra- and inter-subject information and then integrated into a strong classifier with a weight assessment. 55 subjects were recruited to spell 20 characters offline using the conventional P300-based BCI, i.e. the P300-speller. Four different metrics, the P300 accuracy and precision, the round accuracy, and the character accuracy, were performed for a comprehensive investigation. The results revealed that the classifier constructed on the training dataset in combination with adding other subject's data was significantly superior to that without the inter-subject information. Therefore, the WELGI is an effective classifier calibration strategy which uses the inter-subject information to improve the accuracy of subject-specific P300 classifiers, and could also be applied to other BCI paradigms.