The brain–computer interface (BCI) establishes a direct communication system between the brain and a computer or other external devices. Since the inception of BCI technology half a century ago, it has advanced rapidly and developed into an active area of frontier research in modern applied science and technology. This article provides a comprehensive survey on BCI with respect to a brain-in-the-loop communication system. In the present work, we first introduce the underlying architecture of the BCI system from the theoretical and methodological perspectives of communication systems. The key technologies are then detailed, including the construction of BCI system, brain-to-computer (B2C) communication, computer-to-brain (C2B) communication, and multiuser BCI systems. Additionally, this article discusses the various applications of BCI and the challenges they face. Finally, this article discusses BCI’s future development, with an emphasis on the convergence of human intelligence (HI) and artificial intelligence (AI), and the interaction of BCI with wireless communication and the metaverse.

Electroencephalography (EEG), known for its convenient noninvasive acquisition but moderate signal-to-noise ratio, has recently gained much attention due to the potential to decode image information. However, previous works have not delivered sufficient evidence of this task, primarily limited by performance and biological plausibility. In this work, we first introduce a self-supervised framework to demonstrate the feasibility of recognizing images from EEG signals. Contrastive learning is leveraged to align the representations of EEG responses with image stimuli. Then, language descriptions of the stimuli generated by large language models (LLMs) help guide learning core semantic information. With the framework, we attain significantly above-chance results on the THINGS-EEG2 dataset, achieving a top-1 accuracy of 19.7% and a top-5 accuracy of 51.5% in challenging 200-way zero-shot tasks. Furthermore, we conduct thorough experiments to resolve the human visual responses with EEG from temporal, spatial, spectral, and semantic perspectives. These results provide evidence of feasibility and plausibility regarding EEG-based image recognition, substantiated by comparative studies with the THINGS-Magnetoencephalography (MEG) dataset. The findings offer valuable insights for neural decoding and real-world applications of brain-computer interfaces (BCIs), such as health care and robot control. The code is available at https://github.com/eeyhsong/NICE-LLM.

The brain-computer interface (BCI) based on steady-state visual evoked potential (SSVEP) has received considerable attention for its high communication speed. While large datasets provide an important opportunity to enhance decoding accuracies, the key challenge lies in the exploration of existing data to extract valuable information based on the distinctive characteristics of brain responses. In this study, we introduce ConsenNet, a framework designed to enhance SSVEP classification performance by leveraging information from the diverse perspectives of existing subjects. First, this study exploits the diversity of existing subjects to generate new samples, which retain both task-related components and variability. This effectively enhances the network generalization capability on new subjects. Second, the structured knowledge that encapsulates the interrelationships between categories has been constructed and then transferred from the teacher network to the student network, guiding the student network to extract invariant features across subjects. Finally, our model incorporates a small amount of new subject data for model calibration in the final stage. Offline experiments conducted on three public datasets demonstrate the superiority of ConsenNet over 19 methods compared in this study, while online experiments validate its feasibility for real-world applications.

Electroencephalography (EEG) signals, known for convenient non-invasive acquisition but low signal-to-noise ratio, have recently gained substantial attention due to the potential to decode natural images. This paper presents a self-supervised framework to demonstrate the feasibility of learning image representations from EEG signals, particularly for object recognition. The framework utilizes image and EEG encoders to extract features from paired image stimuli and EEG responses. Contrastive learning aligns these two modalities by constraining their similarity. Our approach achieves state-of-the-art results on a comprehensive EEG-image dataset, with a top-1 accuracy of 15.6% and a top-5 accuracy of 42.8% in 200-way zero-shot tasks. Moreover, we perform extensive experiments to explore the biological plausibility by resolving the temporal, spatial, spectral, and semantic aspects of EEG signals. Besides, we introduce attention modules to capture spatial correlations, providing implicit evidence of the brain activity perceived from EEG data. These findings yield valuable insights for neural decoding and brain-computer interfaces in real-world scenarios. Code available at https://github.com/eeyhsong/NICE-EEG.

The ultimate goal of brain-computer interfaces (BCIs) based on visual modulation paradigms is to achieve high-speed performance without the burden of extensive calibration. Code-modulated visual evoked potential-based BCIs (c-VEP-BCIs) modulated by broadband white noise (WN) offer various advantages, including increased communication speed, expanded encoding target capabilities, and enhanced coding flexibility. However, the complexity of the spatial-temporal patterns under broadband stimuli necessitates extensive calibration for effective target identification in c-VEP-BCIs. Consequently, the information transfer rate (ITR) of c-VEP-BCI under limited calibration usually stays around 100 bits per minute (bpm), significantly lagging behind state-of-the-art steady-state visual evoked potential-based BCIs (SSVEP-BCIs), which achieve rates above 200 bpm. To enhance the performance of c-VEP-BCIs with minimal calibration, we devised an efficient calibration stage involving a brief single-target flickering, lasting less than a minute, to extract generalizable spatial-temporal patterns. Leveraging the calibration data, we developed two complementary methods to construct c-VEP temporal patterns: the linear modeling method based on the stimulus sequence and the transfer learning techniques using cross-subject data. As a result, we achieved the highest ITR of 250 bpm under a minute of calibration, which has been shown to be comparable to the state-of-the-art SSVEP paradigms. In summary, our work significantly improved the c-VEP performance under few-shot learning, which is expected to expand the practicality and usability of c-VEP-BCIs.

Electroencephalography (EEG) biometrics has garnered significant attention in recent years owing to its nonintrusive nature, real-time detection capabilities, concealment, and high complexity. Despite these promising attributes, the practical deployment of EEG-based identity recognition systems remains hindered by limited cross-day recognition performance. While some studies have reported cross-day recognition, they often suffer from slow recognition speeds, failing to meet the basic requirements for practical applications. To address this issue, we propose an unsupervised domain adaptation algorithm based on spatial pattern alignment for visual-evoked potential (VEP)-based identity recognition. This method employs rotational alignment of spatial patterns to correct cross-day spatial filters and utilizes forward selection to identify optimal sub-bands. By utilizing this approach, significant improvements of speed and accuracy in cross-day recognition can be achieved. We validate the proposed algorithm on three existing VEP data sets: 1) Data Set I (25 subjects across 30 days); 2) Data Set II (21 subjects across 5 days); and 3) Data Set III (15 subjects across 200 days). The results demonstrate a significant superiority over the compared algorithms. Furthermore, we conduct online experiments with 15 individuals across over 1000 days, and the outcomes remain consistent. Analyzing the data set over nearly three years in terms of the temporal dimension, we observe evident performance differences caused by template aging effect: 30 days > 200 days > 1000 days. However, the proposed method effectively mitigates template aging, resulting in minimal performance differences among the various data sets. The introduced algorithm substantially enhances speed and accuracy in cross-day recognition, paving the way for the long-term stability and practicality of online brainwave recognition systems.

In recent years, electroencephalography (EEG) has received increasing attention in the field of biometrics because of its unique advantages such as covertness, resistance to spoofing, sensitivity to emotional and mental states, and continuous nature. Visual evoked potentials (VEPs) have been widely used in EEG-based biometrics owing to fast recognition speed and high accuracy. This study proposes a new method to combine multiple visual stimuli for VEP-based individual identification. Correct recognition rate (CRR) was estimated using steady-state VEPs (ss-VEPs), and code modulated VEPs (c-VEPs) recorded from a group of 35 subjects. c-VEPs achieved a 100% CRR using 3.1-s of VEP data (a 10.8-s duration, including 7.7-s intervals) in the cross-session condition. An online system based on the combination of stimuli optimized from the data of 35 subjects was further developed and validated with an additional group of 22 subjects. A cross-session CRR of 99.55% was achieved using the same parameters. These results indicate that the proposed VEP-based individual identification method using multiple visual stimuli shows great potential for practical applications.

While recent developments in electroencephalogram (EEG)-based brain-computer interfaces (BCIs) have enabled a bridge between the brain and external devices with relatively high communication speed, there is still room for improvement. Notably, the phenomenon of “BCI illiteracy,” which refers to the 15%–30% of people who struggle to type or control devices using BCI, remains unsolved, limiting the practical application of BCI systems. The EEG-based BCIs performance is constrained by the low-quality scalp EEG signals due to the attenuation and distortion of the skull. To address these limitations, this study proposes a hybrid BCI system combining EEG with magnetoencephalogram (MEG), a neuroimaging technology not influenced by the volume conduction effect, to boost BCI performance by enhancing signal quality. Comparative experiments involving 22 subjects showed that the steady-state visual evoked response (SSVER) from MEG has a wider range of effective bandwidth and higher signal-to-noise ratio than EEG. Moreover, differences in the spectral and spatiotemporal characteristics of MEG and EEG explain better performance. Simultaneous MEG-EEG recording experiments suggested that the hybrid MEG-EEG BCI achieved a significantly higher information transfer rate than either modality alone (hybrid: 312 ± 17 bits/min, MEG: 272 ± 17 bits/min, EEG: 240 ± 27 bits/min). Moreover, the 40-target classification accuracy of “BCI illiterate” increased from 50% to 95% with the help of MEG. These results highlight the methodological advantages of a hybrid MEG-EEG BCI, suggesting a promising paradigm for implementing high-speed BCIs.

Canonical correlation analysis (CCA) is an effective spatial filtering algorithm widely used in steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs). In existing CCA methods, training data are used for constructing templates of stimulus targets and the spatial filters are created between the template signals and a single-trial testing signal. The fact that spatial filters rely on testing data, however, results in low classification performance of CCA compared to other state-of-the-art algorithms such as task-related component analysis (TRCA). In this study, we proposed a novel CCA method in which spatial filters are estimated using training data only. This is achieved by using observed EEG training data and their SSVEP components as the two inputs of CCA and the objective function is optimized by averaging multiple training trials. In this case, we proved in theory that the two spatial filters estimated by the CCA are equivalent, and that the CCA and TRCA are also equivalent under certain hypotheses. A benchmark SSVEP data set from 35 subjects was used to compare the performance of the two algorithms according to different lengths of data, numbers of channels and numbers of training trials. In addition, the CCA was also compared with power spectral density analysis (PSDA). The experimental results suggest that the CCA is equivalent to TRCA if the signal-to-noise ratio of training data is high enough; otherwise, the CCA outperforms TRCA in terms of classification accuracy. The CCA is much faster than PSDA in detecting time of targets. The robustness of the training data-driven CCA to noise gives it greater potential in practical applications.

The electroencephalography (EEG) method has recently attracted increasing attention in the study of brain activity-based biometric systems because of its simplicity, portability, noninvasiveness, and relatively low cost. However, due to the low signal-to-noise ratio of EEG, most of the existing EEG-based biometric systems require a long duration of signals to achieve high accuracy in individual identification. Besides, the feasibility and stability of these systems have not yet been conclusively reported, since most studies did not perform longitudinal evaluation. In this paper, we proposed a novel EEG-based individual identification method using code-modulated visualevoked potentials (c-VEPs). Specifically, this paper quantitatively compared eight code-modulated stimulation patterns, including six 63-bit (1.05 s at 60-Hz refresh rate) m-sequences (M1-M6) and two spatially combined sequence groups (M×4: M1-M4 and M× 6: M1-M6) in recording the c-VEPs from a group of 25 subjects for individual identification. To further evaluate the influence of inter-session variability, we recorded two data sessions for each individual on different days to measure intra-session and cross-session identification performance. State-of-the-art VEP detection algorithms in brain-computer interfaces (BCIs) were employed to construct a template-matching-based identification framework. For intra-session identification, we achieved a 100% correct recognition rate (CRR) using 5.25-s EEG data (average of five trials for M5). For cross-session identification, 99.43% CRR was attained using 10.5-s EEG signals (average of ten trials for M5). These results suggest that the proposed c-VEP based individual identification method is promising for real-world applications.

Gaze interaction provides an efficient way for users to communicate and control in virtual reality (VR) presented by head-mounted displays. In gaze-based text-entry systems, eye tracking and brain-computer interface (BCI) are the two most commonly used approaches. This paper presents a hybrid BCI system for text entry in VR by combining steady-state visual evoked potentials (SSVEP) and eye tracking. The user interface in VR designed a 40-target virtual keyboard using a joint frequency-phase modulation method for SSVEP. Eye position was measured by an eye-tracking accessory in the VR headset. Target-related gaze direction was detected by combining simultaneously recorded SSVEP and eye position data. Offline and online experiments indicate that the proposed system can type at a speed around 10 words per minute, leading to an information transfer rate (ITR) of 270 bits per minute. The results further demonstrate the superiority of the hybrid method over single-modality methods for VR applications.

The past decade has witnessed rapid development in the field of brain-computer interfaces (BCIs). While the performance is no longer the biggest bottleneck in the BCI application, the tedious training process and the poor ease-of-use have become the most significant challenges. In this study, a spatio-temporal equalization dynamic window (STE-DW) recognition algorithm is proposed for steady-state visual evoked potential (SSVEP)-based BCIs. The algorithm can adaptively control the stimulus time while maintaining the recognition accuracy, which significantly improves the information transfer rate (ITR) and enhances the adaptability of the system to different subjects. Specifically, a spatio-temporal equalization algorithm is used to reduce the adverse effects of spatial and temporal correlation of background noise. Based on the theory of multiple hypotheses testing, a stimulus termination criterion is used to adaptively control the dynamic window. The offline analysis which used a benchmark dataset and an offline dataset collected from 16 subjects demonstrated that the STE-DW algorithm is superior to the filter bank canonical correlation analysis (FBCCA), canonical variates with autoregressive spectral analysis (CVARS), canonical correlation analysis (CCA) and CCA reducing variation (CCA-RV) algorithms in terms of accuracy and ITR. The results show that in the benchmark dataset, the STE-DW algorithm achieved an average ITR of 134 bits/min, which exceeds the FBCCA, CVARS, CCA and CCA-RV. In off-line experiments, the STE-DW algorithm also achieved an average ITR of 116 bits/min. In addition, the online experiment also showed that the STE-DW algorithm can effectively expand the number of applicable users of the SSVEP-based BCI system. We suggest that the STE-DW algorithm can be used as a reliable identification algorithm for training-free SSVEP-based BCIs, because of the good balance between ease of use, recognition accuracy, ITR and user applicability.

Although robot technology has been successfully used to empower people who suffer from motor disabilities to increase their interaction with their physical environment, it remains a challenge for individuals with severe motor impairment, who do not have the motor control ability to move robots or prosthetic devices by manual control. In this study, to mitigate this issue, a noninvasive brain-computer interface (BCI)-based robotic arm control system using gaze based steady-state visual evoked potential (SSVEP) was designed and implemented using a portable wireless electroencephalogram (EEG) system. A 15-target SSVEP-based BCI using a filter bank canonical correlation analysis (FBCCA) method allowed users to directly control the robotic arm without system calibration. The online results from 12 healthy subjects indicated that a command for the proposed brain-controlled robot system could be selected from 15 possible choices in 4[Formula: see text]s (i.e. 2[Formula: see text]s for visual stimulation and 2[Formula: see text]s for gaze shifting) with an average accuracy of 92.78%, resulting in a 15 commands/min transfer rate. Furthermore, all subjects (even naive users) were able to successfully complete the entire move-grasp-lift task without user training. These results demonstrated an SSVEP-based BCI could provide accurate and efficient high-level control of a robotic arm, showing the feasibility of a BCI-based robotic arm control system for hand-assistance.

Implementing a complex spelling program using a steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) remains a challenge due to difficulties in stimulus presentation and target identification. This study aims to explore the feasibility of mixed frequency and phase coding in building a high-speed SSVEP speller with a computer monitor. A frequency and phase approximation approach was developed to eliminate the limitation of the number of targets caused by the monitor refresh rate, resulting in a speller comprising 32 flickers specified by eight frequencies (8-15 Hz with a 1 Hz interval) and four phases (0°, 90°, 180°, and 270°). A multi-channel approach incorporating Canonical Correlation Analysis (CCA) and SSVEP training data was proposed for target identification. In a simulated online experiment, at a spelling rate of 40 characters per minute, the system obtained an averaged information transfer rate (ITR) of 166.91 bits/min across 13 subjects with a maximum individual ITR of 192.26 bits/min, the highest ITR ever reported in electroencephalogram (EEG)-based BCIs. The results of this study demonstrate great potential of a high-speed SSVEP-based BCI in real-life applications.