Most existing studies analyzed the resting-state electroencephalogram (EEG) of DOC patients, and recent research demonstrated that the passive auditory paradigm was helpful for bedside detection of DOC and better captured sensory and cognitive responses. However, further studies of classification algorithms were needed for consciousness assessment in DOC based on task-state EEG data. In this study, EEG data from minimally conscious state (MCS) patients, vegetative state (VS) patients, and a healthy control group (HC) were collected using an auditory oddball paradigm. First, compared to the fragmented features adopted by most studies, multiple effective biomarkers for consciousness assessment in the time-frequency domains, connectivity and nonlinear dynamics were identified. Event-related potentials (ERP) results showed that MCS and VS patients exhibited lower N100 and MMN amplitudes than the HC group. Spectral analysis results indicated that VS patients had higher Delta power, and lower Alpha and Beta power than the MCS and HC groups. Second, different from insufficient classifiers in previous studies, this study systematically compared the performance of multiple machine learning and deep learning (DL) classifiers, including support vector machine (SVM), linear discriminant analysis (LDA), random forest (RF), eXtreme Gradient Boosting (XGBoost), decision tree (DT), EEGNet and ShallowConvNet. For machine learning methods, SVM and RF had an advantage in binary classification, and SVM had better performance in three-class classification. Among all individual classifiers, Shallow ConvNet had the best performance for binary and three-class classification. Moreover, an ensemble model incorporating all seven classifiers was proposed using a voting strategy, and further improved classification performance that was superior to existing studies. In addition, the importance of each feature was analyzed, identifying N100, MMN, Delta, Alpha, and Beta power as significant biomarkers of consciousness assessment.

Steady-state visual evoked potential-based brain-computer interfaces (SSVEP-BCIs) hold significant promise for enabling high-speed human-computer interaction in real-world scenarios. However, existing frequency-domain decoding methods treat frequency spectrum features (the real and imaginary spectrum features) as a single feature without considering their unique spatial and spectral characteristics, resulting in insufficient generalizable features and limited classification accuracy in cross-subject scenarios. To address this issue, we propose a Dual-Branch Attention-Based Frequency Domain Network (DB-AFDNet) to independently decode real and imaginary spectral components, aiming to acquire more discriminative and generalizable features for cross-subject applications. Specifically, we construct inter-branch attention similarity constraints to encourage the two branches to have similar attention properties, promoting to learn the consensus characteristics in the dual branches. Furthermore, we propose intra-branch orthogonality constraints to explore branch-specific discriminative features to learn generalizable features. Experimental studies on two public datasets, the Benchmark and Beta datasets, demonstrate that DB-AFDNet outperforms state-of-the-art methods in cross-subject classification, achieving a relative improvement of 1.36$\%$ and 1.45$\%$, respectively.

The brain–computer interface (BCI) establishes a direct communication system between the brain and a computer or other external devices. Since the inception of BCI technology half a century ago, it has advanced rapidly and developed into an active area of frontier research in modern applied science and technology. This article provides a comprehensive survey on BCI with respect to a brain-in-the-loop communication system. In the present work, we first introduce the underlying architecture of the BCI system from the theoretical and methodological perspectives of communication systems. The key technologies are then detailed, including the construction of BCI system, brain-to-computer (B2C) communication, computer-to-brain (C2B) communication, and multiuser BCI systems. Additionally, this article discusses the various applications of BCI and the challenges they face. Finally, this article discusses BCI’s future development, with an emphasis on the convergence of human intelligence (HI) and artificial intelligence (AI), and the interaction of BCI with wireless communication and the metaverse.

Electroencephalography (EEG), known for its convenient noninvasive acquisition but moderate signal-to-noise ratio, has recently gained much attention due to the potential to decode image information. However, previous works have not delivered sufficient evidence of this task, primarily limited by performance and biological plausibility. In this work, we first introduce a self-supervised framework to demonstrate the feasibility of recognizing images from EEG signals. Contrastive learning is leveraged to align the representations of EEG responses with image stimuli. Then, language descriptions of the stimuli generated by large language models (LLMs) help guide learning core semantic information. With the framework, we attain significantly above-chance results on the THINGS-EEG2 dataset, achieving a top-1 accuracy of 19.7% and a top-5 accuracy of 51.5% in challenging 200-way zero-shot tasks. Furthermore, we conduct thorough experiments to resolve the human visual responses with EEG from temporal, spatial, spectral, and semantic perspectives. These results provide evidence of feasibility and plausibility regarding EEG-based image recognition, substantiated by comparative studies with the THINGS-Magnetoencephalography (MEG) dataset. The findings offer valuable insights for neural decoding and real-world applications of brain-computer interfaces (BCIs), such as health care and robot control. The code is available at https://github.com/eeyhsong/NICE-LLM.

Electroencephalography (EEG) signals, known for convenient non-invasive acquisition but low signal-to-noise ratio, have recently gained substantial attention due to the potential to decode natural images. This paper presents a self-supervised framework to demonstrate the feasibility of learning image representations from EEG signals, particularly for object recognition. The framework utilizes image and EEG encoders to extract features from paired image stimuli and EEG responses. Contrastive learning aligns these two modalities by constraining their similarity. Our approach achieves state-of-the-art results on a comprehensive EEG-image dataset, with a top-1 accuracy of 15.6% and a top-5 accuracy of 42.8% in 200-way zero-shot tasks. Moreover, we perform extensive experiments to explore the biological plausibility by resolving the temporal, spatial, spectral, and semantic aspects of EEG signals. Besides, we introduce attention modules to capture spatial correlations, providing implicit evidence of the brain activity perceived from EEG data. These findings yield valuable insights for neural decoding and brain-computer interfaces in real-world scenarios. Code available at https://github.com/eeyhsong/NICE-EEG.

The ultimate goal of brain-computer interfaces (BCIs) based on visual modulation paradigms is to achieve high-speed performance without the burden of extensive calibration. Code-modulated visual evoked potential-based BCIs (c-VEP-BCIs) modulated by broadband white noise (WN) offer various advantages, including increased communication speed, expanded encoding target capabilities, and enhanced coding flexibility. However, the complexity of the spatial-temporal patterns under broadband stimuli necessitates extensive calibration for effective target identification in c-VEP-BCIs. Consequently, the information transfer rate (ITR) of c-VEP-BCI under limited calibration usually stays around 100 bits per minute (bpm), significantly lagging behind state-of-the-art steady-state visual evoked potential-based BCIs (SSVEP-BCIs), which achieve rates above 200 bpm. To enhance the performance of c-VEP-BCIs with minimal calibration, we devised an efficient calibration stage involving a brief single-target flickering, lasting less than a minute, to extract generalizable spatial-temporal patterns. Leveraging the calibration data, we developed two complementary methods to construct c-VEP temporal patterns: the linear modeling method based on the stimulus sequence and the transfer learning techniques using cross-subject data. As a result, we achieved the highest ITR of 250 bpm under a minute of calibration, which has been shown to be comparable to the state-of-the-art SSVEP paradigms. In summary, our work significantly improved the c-VEP performance under few-shot learning, which is expected to expand the practicality and usability of c-VEP-BCIs.

Electroencephalography (EEG) biometrics has garnered significant attention in recent years owing to its nonintrusive nature, real-time detection capabilities, concealment, and high complexity. Despite these promising attributes, the practical deployment of EEG-based identity recognition systems remains hindered by limited cross-day recognition performance. While some studies have reported cross-day recognition, they often suffer from slow recognition speeds, failing to meet the basic requirements for practical applications. To address this issue, we propose an unsupervised domain adaptation algorithm based on spatial pattern alignment for visual-evoked potential (VEP)-based identity recognition. This method employs rotational alignment of spatial patterns to correct cross-day spatial filters and utilizes forward selection to identify optimal sub-bands. By utilizing this approach, significant improvements of speed and accuracy in cross-day recognition can be achieved. We validate the proposed algorithm on three existing VEP data sets: 1) Data Set I (25 subjects across 30 days); 2) Data Set II (21 subjects across 5 days); and 3) Data Set III (15 subjects across 200 days). The results demonstrate a significant superiority over the compared algorithms. Furthermore, we conduct online experiments with 15 individuals across over 1000 days, and the outcomes remain consistent. Analyzing the data set over nearly three years in terms of the temporal dimension, we observe evident performance differences caused by template aging effect: 30 days > 200 days > 1000 days. However, the proposed method effectively mitigates template aging, resulting in minimal performance differences among the various data sets. The introduced algorithm substantially enhances speed and accuracy in cross-day recognition, paving the way for the long-term stability and practicality of online brainwave recognition systems.

In recent years, electroencephalography (EEG) has received increasing attention in the field of biometrics because of its unique advantages such as covertness, resistance to spoofing, sensitivity to emotional and mental states, and continuous nature. Visual evoked potentials (VEPs) have been widely used in EEG-based biometrics owing to fast recognition speed and high accuracy. This study proposes a new method to combine multiple visual stimuli for VEP-based individual identification. Correct recognition rate (CRR) was estimated using steady-state VEPs (ss-VEPs), and code modulated VEPs (c-VEPs) recorded from a group of 35 subjects. c-VEPs achieved a 100% CRR using 3.1-s of VEP data (a 10.8-s duration, including 7.7-s intervals) in the cross-session condition. An online system based on the combination of stimuli optimized from the data of 35 subjects was further developed and validated with an additional group of 22 subjects. A cross-session CRR of 99.55% was achieved using the same parameters. These results indicate that the proposed VEP-based individual identification method using multiple visual stimuli shows great potential for practical applications.

While recent developments in electroencephalogram (EEG)-based brain-computer interfaces (BCIs) have enabled a bridge between the brain and external devices with relatively high communication speed, there is still room for improvement. Notably, the phenomenon of “BCI illiteracy,” which refers to the 15%–30% of people who struggle to type or control devices using BCI, remains unsolved, limiting the practical application of BCI systems. The EEG-based BCIs performance is constrained by the low-quality scalp EEG signals due to the attenuation and distortion of the skull. To address these limitations, this study proposes a hybrid BCI system combining EEG with magnetoencephalogram (MEG), a neuroimaging technology not influenced by the volume conduction effect, to boost BCI performance by enhancing signal quality. Comparative experiments involving 22 subjects showed that the steady-state visual evoked response (SSVER) from MEG has a wider range of effective bandwidth and higher signal-to-noise ratio than EEG. Moreover, differences in the spectral and spatiotemporal characteristics of MEG and EEG explain better performance. Simultaneous MEG-EEG recording experiments suggested that the hybrid MEG-EEG BCI achieved a significantly higher information transfer rate than either modality alone (hybrid: 312 ± 17 bits/min, MEG: 272 ± 17 bits/min, EEG: 240 ± 27 bits/min). Moreover, the 40-target classification accuracy of “BCI illiterate” increased from 50% to 95% with the help of MEG. These results highlight the methodological advantages of a hybrid MEG-EEG BCI, suggesting a promising paradigm for implementing high-speed BCIs.

The increasing popularity of Internet of Things (IoT) devices provides a huge data source for intelligent identification. Images captured by the unmanned aerial vehicle (UAV) are often exploited in target detection missions for search, rescue and fire prevention. However, without sufficient training samples, the machine learning method is usually difficult to meet application requirements. To solve the problem, a brain–computer interface (BCI) real-time system is applied to UAV target detection. In this study, a novel rapid serial visual presentation (RSVP) paradigm was formulated to present images, enhancing the ability for wide-area target detection. Since there is spatial correlation in captured images, joint decision for identical target can improve the recognition efficiency. To suppress interfering components and improve event-related potential (ERP) detection efficiency, an enhancing ERP component (EEC) algorithm is proposed. Both decision method and EEC algorithm are based on strictly statistical theory. RSVP task was performed by 12 subjects. The interference components and noise correlation were significantly reduced by the EEC algorithm. The target detection rate online was 86.6% while the false alarm rate was less than 5%. Besides, the joint decision strategy raised the area under curve (AUC) value from 0.876 to 0.963. The proposed BCI real-time system realizes the complementarity of human intelligence and IoT, ushering UAV target detection into the era of hybrid intelligence.

While search technologies have evolved to be robust and ubiquitous, the fundamental interaction paradigm has remained relatively stable for decades. With the maturity of the Brain-Machine Interface(BMI), we build an efficient and effective communication system between human beings and search engines based on electroencephalogram (EEG) signals, called Brain Machine Search Interface (BMSI)system. The BMSI system provides functions including query reformulation and search result interaction. In our system, users can perform search tasks without having to use the mouse and keyboard. Therefore, it is useful for application scenarios in which hand-based interactions are infeasible, e.g, for users with severe neuromuscular disorders. Besides, based on brain signals decoding, our system can provide abundant and valuable user-side context information (e.g., real-time satisfaction feedback, extensive context information, and a clearer description of information needs) to the search engine, which is hard to capture in the previous paradigm. In our implementation, the system can decode user satisfaction from brain signals in real-time during the interaction process and re-rank the search results list based on user satisfaction feedback.The demo video is available at http://www.thuir.cn/group/YQLiu/videos/BMSISystem.html

Canonical correlation analysis (CCA) is an effective spatial filtering algorithm widely used in steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs). In existing CCA methods, training data are used for constructing templates of stimulus targets and the spatial filters are created between the template signals and a single-trial testing signal. The fact that spatial filters rely on testing data, however, results in low classification performance of CCA compared to other state-of-the-art algorithms such as task-related component analysis (TRCA). In this study, we proposed a novel CCA method in which spatial filters are estimated using training data only. This is achieved by using observed EEG training data and their SSVEP components as the two inputs of CCA and the objective function is optimized by averaging multiple training trials. In this case, we proved in theory that the two spatial filters estimated by the CCA are equivalent, and that the CCA and TRCA are also equivalent under certain hypotheses. A benchmark SSVEP data set from 35 subjects was used to compare the performance of the two algorithms according to different lengths of data, numbers of channels and numbers of training trials. In addition, the CCA was also compared with power spectral density analysis (PSDA). The experimental results suggest that the CCA is equivalent to TRCA if the signal-to-noise ratio of training data is high enough; otherwise, the CCA outperforms TRCA in terms of classification accuracy. The CCA is much faster than PSDA in detecting time of targets. The robustness of the training data-driven CCA to noise gives it greater potential in practical applications.

The past decade has witnessed rapid development in the field of brain-computer interfaces (BCIs). While the performance is no longer the biggest bottleneck in the BCI application, the tedious training process and the poor ease-of-use have become the most significant challenges. In this study, a spatio-temporal equalization dynamic window (STE-DW) recognition algorithm is proposed for steady-state visual evoked potential (SSVEP)-based BCIs. The algorithm can adaptively control the stimulus time while maintaining the recognition accuracy, which significantly improves the information transfer rate (ITR) and enhances the adaptability of the system to different subjects. Specifically, a spatio-temporal equalization algorithm is used to reduce the adverse effects of spatial and temporal correlation of background noise. Based on the theory of multiple hypotheses testing, a stimulus termination criterion is used to adaptively control the dynamic window. The offline analysis which used a benchmark dataset and an offline dataset collected from 16 subjects demonstrated that the STE-DW algorithm is superior to the filter bank canonical correlation analysis (FBCCA), canonical variates with autoregressive spectral analysis (CVARS), canonical correlation analysis (CCA) and CCA reducing variation (CCA-RV) algorithms in terms of accuracy and ITR. The results show that in the benchmark dataset, the STE-DW algorithm achieved an average ITR of 134 bits/min, which exceeds the FBCCA, CVARS, CCA and CCA-RV. In off-line experiments, the STE-DW algorithm also achieved an average ITR of 116 bits/min. In addition, the online experiment also showed that the STE-DW algorithm can effectively expand the number of applicable users of the SSVEP-based BCI system. We suggest that the STE-DW algorithm can be used as a reliable identification algorithm for training-free SSVEP-based BCIs, because of the good balance between ease of use, recognition accuracy, ITR and user applicability.

Although robot technology has been successfully used to empower people who suffer from motor disabilities to increase their interaction with their physical environment, it remains a challenge for individuals with severe motor impairment, who do not have the motor control ability to move robots or prosthetic devices by manual control. In this study, to mitigate this issue, a noninvasive brain-computer interface (BCI)-based robotic arm control system using gaze based steady-state visual evoked potential (SSVEP) was designed and implemented using a portable wireless electroencephalogram (EEG) system. A 15-target SSVEP-based BCI using a filter bank canonical correlation analysis (FBCCA) method allowed users to directly control the robotic arm without system calibration. The online results from 12 healthy subjects indicated that a command for the proposed brain-controlled robot system could be selected from 15 possible choices in 4[Formula: see text]s (i.e. 2[Formula: see text]s for visual stimulation and 2[Formula: see text]s for gaze shifting) with an average accuracy of 92.78%, resulting in a 15 commands/min transfer rate. Furthermore, all subjects (even naive users) were able to successfully complete the entire move-grasp-lift task without user training. These results demonstrated an SSVEP-based BCI could provide accurate and efficient high-level control of a robotic arm, showing the feasibility of a BCI-based robotic arm control system for hand-assistance.

Recent years have witnessed brain-computer interface (BCI) as a promising technology for integrating human intelligence and machine intelligence. Currently, event-related potential (ERP)-based BCI is an important branch of noninvasive electroencephalogram (EEG)-based BCIs. Extracting ERPs from a limited number of trials remains challenging due to their low signal-to-noise ratio (SNR) and low spatial resolution caused by volume conduction. In this paper, we propose a probabilistic model for trial-by-trial concatenated EEG, in which the concatenated ERPs are expressed as a linear combination of a set of discrete sine and cosine bases. The bases are simply determined by the data length of a single trial. A sparse prior on the rank of the spatio-spectral pattern matrix is introduced into the model to allow the number of components to be automatically determined. A maximum posterior estimation algorithm based on cyclic descent is then developed to estimate the spatiospectral patterns. A spatial filter can then be obtained by maximizing the SNR of the ERP components. Experiments on both synthetic data and real N170 ERP from 13 subjects were conducted to test the efficacy and efficiency of the algorithm. The results showed that the proposed algorithm can estimate the ERPs more accurately than the several state-of-the-art algorithms.

Common spatial patterns (CSP) is a well-known spatial filtering algorithm for multichannel electroencephalogram (EEG) analysis. In this paper, we cast the CSP algorithm in a probabilistic modeling setting. Specifically, probabilistic CSP (P-CSP) is proposed as a generic EEG spatio-temporal modeling framework that subsumes the CSP and regularized CSP algorithms. The proposed framework enables us to resolve the overfitting issue of CSP in a principled manner. We derive statistical inference algorithms that can alleviate the issue of local optima. In particular, an efficient algorithm based on eigendecomposition is developed for maximum a posteriori (MAP) estimation in the case of isotropic noise. For more general cases, a variational algorithm is developed for group-wise sparse Bayesian learning for the P-CSP model and for automatically determining the model size. The two proposed algorithms are validated on a simulated data set. Their practical efficacy is also demonstrated by successful applications to single-trial classifications of three motor imagery EEG data sets and by the spatio-temporal pattern analysis of one EEG data set recorded in a Stroop color naming task.

A BCI-FES training platform has been designed for rehabilitation on chronic stroke patients to train their upper limb motor functions. The conventional functional electrical stimulation (FES) was driven by users’ intention through EEG signals to move their wrist and hand. Such active participation was expected to be important for motor rehabilitation according to motor relearning theory. The common spatial pattern (CSP) algorithm was applied as one pre-processing step in brain-computer interface (BCI) module to search for the optimal spatial projection direction after brain reorganization. The pre- and post- clinical assessment was conducted to identify the possible functional improvement after the training. Two chronic stroke subjects attended this pilot study and the error rate of the BCI control was less than 20% after training of 10 sessions. This implementation showed the feasibility for stroke patients to accomplish the BCI triggered FES rehabilitation training.

Canonical Correlation Analysis (CCA) is used to frequency recognition of multichannel signals. The unknown signals are compared against known templates and their frequencies are recognized by simply comparing the biggest coefficients of their CCA coefficient vectors. This strategy is straightforward but may not give optimal results. To boost the accuracy of recognition we reformulate the approach in views of machine learning. In this paper, we propose a new strategy based on supervised learning. We also employ feature selection within this framework to adopt efficient coefficients which may not be the largest coefficients for the features vectors. The recognition method is validated by results with real world data.

This paper presents a novel speckle suppression method for medical B-scan ultrasonic images. An original image is first separated into two parts with an adaptive filter. These two parts are then transformed into a multiscale wavelet domain and the wavelet coefficients are processed by a soft thresholding method, which is a variation of Donoho's soft thresholding method. The processed coefficients for each part are then transformed back into the space domain. Finally, the denoised image is obtained as the sum of the two processed parts. A computer-simulated image and an in vitro B-scan image of a pig heart have been used to test the performance of this new method. This technique effectively reduces the speckle noise, while preserving the resolvable details. It performs well in comparison to the multiscale thresholding technique without adaptive preprocessing and two other speckle-suppression methods.