Electroencephalography (EEG)-based emotion recognition is essential for medical assistance and human-computer interaction. Although deep learning-based emotion recognition methods have demonstrated high performance, several challenges remain: 1) How to effectively utilize the complex dynamic-static spatial patterns inherent in emotion-related EEG signals. 2) How to hierarchically learn the latent correlations among multi-domain features. To address these challenges, a spatio-temporal representation fusion learning network (STRFLNet) is proposed to improve both the accuracy and robustness of emotion recognition using EEG signals. Specifically, dynamic-static graph topologies are constructed to capture comprehensive brain functional connectivity states, and a continuous dynamic-static graph ordinary differential equations is introduce to reveal continuous spatial patterns within EEG signals. Additionally, a hierarchical transformer fusion module is developed to fully leverage the latent correlations among multi-domain features to obtain the fused spatio-temporal representation. The experimental results on the SEED, SEED-IV, and DREAMER public EEG emotion datasets, under both subject-independent and subject-dependent settings, demonstrate that STRFLNet outperforms state-of-the-art methods in emotion recognition tasks. We further validate the effectiveness of the proposed model through interpretability analysis, which reveals the associations between the activated brain regions and corresponding emotional states. Our work highlights the significance of continuous spatial pattern learning and spatio-temporal feature fusion in emotion recognition, providing new insights for EEG-based emotion modeling.

Estimating the extents of E/MEG source activities is crucial for exploring brain dynamics at high spatiotemporal resolution. In this study, we introduce a novel ESI method - Block-Champagne, a Bayesian framework designed to accurately estimate both the locations and extents of extended sources. Our approach leverages a block-sparsity constraint that models each voxel and its neighbors as a single block to account for local homogeneity. The blocks, inherently overlapping in the original source domain, can be adaptively combined to reconstruct sources with arbitrary spatial extents. Furthermore, prior constraints from other neuroimaging modalities with additional spatial information, such as fMRI, can be incorporated to model interactions between distinct sources to further enhance source reconstruction accuracy. The performance of Block-Champagne is quantitatively evaluated through a series of simulation experiments, which demonstrate its overall superiority under various complex scenarios (i.e., SNR, extent size, number of sources, intra-source correlation, & number of EEG channels) compared to benchmark algorithms (including LORETA, EBI-Convex, ts-Cham, L21-Sissy, & BESTIES). Validation results using deep brain stimulation EEG and epilepsy data confirm the practical feasibility of Block-Champagne. Moreover, findings from face processing multimodal data indicate that incorporating relevant and accurate priors significantly enhances source reconstruction accuracy. In conclusion, our study reveals the superiority of the proposed Block-Champagne in accurate reconstruction of extended source, positioning Block-Champagne as a highly promising tool for realistic applications where source locations and extents are of equivalent importance.

E/MEG source imaging (ESI) provides non-invasive measurements of brain activity with high spatial and temporal resolution. In particular, the wearability and portability of EEG make it an attractive area of research beyond the biomedical communities, especially given the broad application prospects including brain-computer interface (BCI), neuromarketing and neuroergonomics. Although existing reviews offer valuable insights, they often present ESI models in a relatively isolated manner and may not encompass the most recent advancements in the field. In this work, we aim to: 1) provide a timely in-depth review of the widely-explored and state-of-the-art ESI models, including their underlying neurophysiological assumptions and mathematical derivations; 2) list the primary applications of ESI and highlight crucial steps regarding its implementations; 3) discuss current challenges in ESI and propose future research prospects; 4) demonstrate practical usage and implementation details of various representative ESI models. As a rapidly expanding field, ESI is continuously developing and evolving to integrate new technologies. We believe the widespread applications of ESI is happening, and it will dramatically expand our understanding of brain dynamics.

The human brain connectome has long been recognized as a crucial component for various cognitive functions. While connectome-based predictive modeling (CPM) has been extensively explored for predicting behavior outcomes at the individual-level, its application to electroencephalogram (EEG) remains limited due to the inherent diversity and complexity of EEG frequency information. In the present work, we aim to address this issue by developing a filter bank CPM (FBCPM) framework that leverages narrowband EEG functional connectivity (FC) for individual prediction. Four independent datasets comprising 280 healthy subjects with 392 EEG recordings during the psychomotor vigilance test (PVT), were adopted here. Using the discovery dataset (i.e., Dataset 1) with 137 recordings, the feasibility of FBCPM was evaluated via predicting mean reaction time (RT) measures within a 15-min PVT task. The results showed that FBCPM framework achieved notable prediction accuracy and outperformed four benchmark approaches. Subsequent comprehensive internal and external validation analyses further affirmed its robustness across various hyper-parameters and generalizability to another three independent datasets (i.e., Dataset 2 to Dataset 4) with divergent recording or preprocessing settings. Moreover, the FBCPM framework exhibited satisfactory performance when generalized to time-on-task (TOT) effect measures (i.e., $\mathit {\Delta RT}$ and $\mathit {TOT_{slope}}$). Further investigation of contributing features to mean RT prediction indicated the remarkable predictive ability of negative features, manifesting as a pattern of low-frequency (below 8 Hz) predominance and complex topological distributions. Overall, these findings indicated that FBCPM provided a significant methodological advance in EEG-based individual prediction approaches, moving a step forward towards practical application in cognitive neuroscience.

To obtain accurate brain source activities, the highly ill-posed source imaging of electro- and magneto-encephalography (E/MEG) requires proficiency in incorporation of biophysiological constraints and signal-processing techniques. Here, we propose a spatio-temporal-constrainted E/MEG source imaging framework-STARTS that can reconstruct the source in a fully automatic way. Specifically, a block-diagonal covariance was adopted to reconstruct the source extents while maintain spatial homogeneity. Temporal basis functions (TBFs) of both sources and noise were estimated and updated in a data-driven fashion to alleviate the influence of noises and further improve source localization accuracy. The performance of the proposed STARTS was quantitatively assessed through a series of simulation experiments, wherein superior results were obtained in comparison with the benchmark ESI algorithms (including LORETA, EBI-Convex, BESTIES & SI-STBF). Additional validations on epileptic and resting-state EEG data further indicate that the STARTS can produce neurophysiologically plausible results. Moreover, a computationally efficient version of STARTS: smooth STARTS was also introduced with an elementary spatial constraint, which exhibited comparable performance and reduced execution cost. In sum, the proposed STARTS, with its advanced spatio-temporal constraints and self-adapted update operation, provides an effective and efficient approach for E/MEG source imaging.

Due to the increasing fatal traffic accidents, there are strong desire for more effective and convenient techniques for driving fatigue detection. Here, we propose a unified frameworkE-Keyto simultaneously perform personal identification (PI) and driving fatigue detection using a convolutional attention neural network (CNN-Attention). The performance was assessed using EEG data collected through a wearable dry-sensor system from 31 healthy subjects undergoing a 90-min simulated driving task. In comparison with three widely-used competitive models (including CNN, CNN-LSTM, and Attention), the proposed scheme achieved the best (p < 0.01) performance in both PI (98.5%) and fatigue detection (97.8%). Besides, the spatial-temporal structure of the proposed framework exhibits an optimal balance between classification performance and computational efficiency. Additional validation analyses were conducted to assess the reliability and practicability of the model via re-configuring the kernel size and manipulating the input data, showing that it can achieve a satisfactory performance using a subset of the input data. In sum, these findings would pave the way for further practical implementation of in-vehicle expert system, showing great potential in autonomous driving and car-sharing where currently monitoring of PI and driving fatigue are of particular interest.

A brain–computer interface (BCI) establishes a direct communication pathway between the human brain and a computer. It has been widely used in medical diagnosis, rehabilitation, education, entertainment, and so on. Most research so far focuses on making BCIs more accurate and reliable, but much less attention has been paid to their privacy. Developing a commercial BCI system usually requires close collaborations among multiple organizations, e.g., hospitals, universities, and/or companies. Input data in BCIs, e.g., electroencephalogram (EEG), contain rich privacy information, and the developed machine learning model is usually proprietary. Data and model transmission among different parties may incur significant privacy threats, and hence, privacy protection in BCIs must be considered. Unfortunately, there does not exist any contemporary and comprehensive review on privacy-preserving BCIs. This article fills this gap, by describing potential privacy threats and protection strategies in BCIs. It also points out several challenges and future research directions in developing privacy-preserving BCIs.

Intention decoding is an indispensable procedure in hands-free human–computer interaction (HCI). A conventional eye-tracker system using a single-model fixation duration may issue commands that ignore users' real expectations. Here, an eye-brain hybrid brain–computer interface (BCI) interaction system was introduced for intention detection through the fusion of multimodal eye-tracker and event-related potential (ERP) [a measurement derived from electroencephalography (EEG)] features. Eye-tracking and EEG data were recorded from 64 healthy participants as they performed a 40-min customized free search task of a fixed target icon among 25 icons. The corresponding fixation duration of eye tracking and ERP were extracted. Five previously-validated linear discriminant analysis (LDA)-based classifiers [including regularized LDA, stepwise LDA, Bayesian LDA, shrinkage linear discriminant analysis (SKLDA), and spatial-temporal discriminant analysis] and the widely-used convolutional neural network (CNN) method were adopted to verify the efficacy of feature fusion from both offline and pseudo-online analysis, and the optimal approach was evaluated by modulating the training set and system response duration. Our study demonstrated that the input of multimodal eye tracking and ERP features achieved a superior performance of intention detection in the single-trial classification of active search tasks. Compared with the single-model ERP feature, this new strategy also induced congruent accuracy across classifiers. Moreover, in comparison with other classification methods, we found that SKLDA exhibited a superior performance when fusing features in offline tests (ACC = 0.8783, AUC = 0.9004) and online simulations with various sample amounts and duration lengths. In summary, this study revealed a novel and effective approach for intention classification using an eye-brain hybrid BCI and further supported the real-life application of hands-free HCI in a more precise and stable manner.

As a common complaint in contemporary society, mental fatigue is a key element in the deterioration of the daily activities known as time-on-task (TOT) effect, making the prediction of fatigue-related performance decline exceedingly important. However, conventional group-level brain-behavioral correlation analysis has the limitation of generalizability to unseen individuals and fatigue prediction at individual-level is challenging due to the significant differences between individuals both in task performance efficiency and brain activities. Here, we introduced a cross-validated data-driven analysis framework to explore, for the first time, the feasibility of utilizing pre-task idiosyncratic resting-state functional connectivity (FC) on the prediction of fatigue-related task performance degradation at individual level. Specifically, two behavioral metrics, namely$\Delta$RT (between the most vigilant and fatigued states) and$TOT_{slope}$over the course of the 15-min sustained attention task, were estimated among three sessions from 37 healthy subjects to represent fatigue-related individual behavioral impairment. Then, a connectome-based prediction model was employed on pre-task resting-state FC features, identifying the network-related differences that contributed to the prediction of performance deterioration. As expected, prominent populational TOT-related performance declines were revealed across three sessions accompanied with substantial inter-individual differences. More importantly, we achieved significantly high accuracies for individualized prediction of both TOT-related behavioral impairment metrics using pre-task neuroimaging features. Despite the distinct patterns between both behavioral metrics, the identified top FC features contributing to the individualized predictions were mainly resided within/between frontal, temporal and parietal areas. Overall, our results of individualized prediction framework extended conventional correlation/classification analysis and may represent a promising avenue for the development of applicable techniques that allow precaution of the TOT-related performance declines in real-world scenarios.

The prediction of schizophrenia-related psychopathologic deficits is exceedingly important in the fields of psychiatry and clinical practice. However, objective association of the brain structure alterations to the illness clinical symptoms is challenging. Although, schizophrenia has been characterized as a brain dysconnectivity syndrome, evidence accounting for neuroanatomical network alterations remain scarce. Moreover, the absence of generalized connectome biomarkers for the assessment of illness progression further perplexes the prediction of long-term symptom severity. In this paper, a combination of individualized prediction models with quantitative graph theoretical analysis was adopted, providing a comprehensive appreciation of the extent to which the brain network properties are affected over time in schizophrenia. Specifically, Connectome-based Prediction Models were employed on Structural Connectivity (SC) features, efficiently capturing individual network-related differences, while identifying the anatomical connectivity disturbances contributing to the prediction of psychopathological deficits. Our results demonstrated distinctions among widespread cortical circuits responsible for different domains of symptoms, indicating the complex neural mechanisms underlying schizophrenia. Furthermore, the generated models were able to significantly predict changes of symptoms using SC features at follow-up, while the preserved SC features suggested an association with improved positive and overall symptoms. Moreover, cross-sectional significant deficits were observed in network efficiency and a progressive aberration of global integration in patients compared to healthy controls, representing a group-consensus pathological map, while supporting the dysconnectivity hypothesis.

In the nascent field of neuroergonomics, mental workload assessment is one of the most important issues and has an apparent significance in real-world applications. Although prior research has achieved efficient single-task classification, scatted studies on cross-task mental workload assessment usually result in unsatisfactory performance. Here, we introduce a data-driven analysis framework to overcome the challenges regarding task-independent workload assessment using a fusion of EEG spectral characteristics and unveil the common neural mechanisms underlying mental workload. Specifically, multi-frequency power spectrum and functional connectivity (FC) were estimated for two workload levels in two working-memory tasks performed by 40 healthy participants, subsequently being fed into a machine learning approach to obtain the importance of each feature vector and evaluate classification performance in a cross-task fashion. Our framework achieved a classification accuracy of 0.94 for task-independent mental workload discrimination. Further investigation of the designated features in terms of their spectral and localization properties revealed task-independent common patterns in the neural mechanisms governing workload. In particular, increased workload was associated with elevated frontal delta and theta power but reduced parietal alpha power, whereas FC exhibited complex frequency- and region-dependent alterations. By implication, the employment of the EEG feature fusion emphasized their utility in serving as promising indicators for different workload conditions applications.

Most brain-computer interfaces (BCIs) utilize univariate features (i.e., power spectrum or amplitude) for motor imagery (MI) pattern recognition, while less attention has been paid on multivariate analysis that considers information flow between various brain areas through brain connectivity estimations. Most recently, researches have proved that connectivity features were able to characterize different MI tasks. In this study, we investigated the performance of functional connectivity features measured by phase lag index (PLI), weighted phase lag index(wPLI) and phase-locking value(PLV) on MI classification. The widely-used filter-bank common spatial pattern (FBCSP) approach was employed here for performance assessment. A linear support vector machine was trained to classify different MI tasks using two publicly-available datasets from BCI Competition III and IV. The classification results showed that connectivity features achieved classification accuracy >85% in most cases and PLI outperformed all other methods including FBCSP. Our work suggested that functional connectivity features could be utilized as a powerful tool for recognizing different motor intention.

The classification of motor imagery (MI) task has gained lots of attention in brain-computer interface (BCI) field and numerous studies have proposed various methods for MI classification. In recent years, time-frequency analysis method and convolutional neural network (CNN) have been combined to extract more complex features and exhibit superiors performance in comparison with conventional methods. In this paper, we presented a new classification method based on Stockwell Transform of common spatial pattern and CNN. The proposed framework with two different activation functions (ReLU and ELU) was compared with CSP-SVM and CSP-CNN methods by validation on BCI competition IV dataset I calibration data to assess the performance of the proposed method. We achieved an average classification accuracy of 81.22% (ReLU) and 81.34% (ELU), which outperformed the conventional CSP-SVM (76.45%) and CSP-CNN methods (69.29%). Further interrogation showed that the higher accuracy was attributed to improved performance in the subjects with lower detection rate by CSP-SVM. In sum, our results showed that the proposed framework can be applied to MI-BCI systems with superior performance, leading new insights towards more practical MI-based BCI applications.

We consider the topic of multivariate regression on manifold-valued output, that is, for a multivariate observation, its output response lies on a manifold. Moreover, we propose a new regression model to deal with the presence of grossly corrupted manifold-valued responses, a bottleneck issue commonly encountered in practical scenarios. Our model first takes a correction step on the grossly corrupted responses via geodesic curves on the manifold, then performs multivariate linear regression on the corrected data. This results in a nonconvex and nonsmooth optimization problem on Riemannian manifolds. To this end, we propose a dedicated approach named PALMR, by utilizing and extending the proximal alternating linearized minimization techniques for optimization problems on euclidean spaces. Theoretically, we investigate its convergence property, where it is shown to converge to a critical point under mild conditions. Empirically, we test our model on both synthetic and real diffusion tensor imaging data, and show that our model outperforms other multivariate regression models when manifold-valued responses contain gross errors, and is effective in identifying gross errors.

MRCPs (movement related cortical potentials) are slow negative potentials observed in EEG preceding movement, which represents the processing of the cerebral cortex during planning and preparation. Clinical studies have shown that, rehabilitation therapy with the active participation of the central nervous system can rebuild related nerve function, restore the patient's neural plasticity and characterize the intention to move by means of electroencephalographic activity which can be used in rehabilitation protocols with patients' cortical activity taking an active role during the intervention. We describe our method framework including protocol, data processing and pattern recognition. Then the experimental results are analyzed. The tests were carried out with healthy people and achieved satisfactory performance both in trial and asynchronous detection. Our purpose is to apply to restoration of neural pathway of patients.

Workload is the amount of cognitive effort executed by a certain subject. Several attempts have been done in order to measure workload level. However, there exists a difficulty in analyzing workload: the problem of individuality, or variability among different individuals and how they respond to similar tasks. In order to have a more objective measure of workload level, the authors employed a more direct analysis upon the system that does cognitive work itself. The use of electroencephalogram (EEG) was employed to measure brain signals and process them to get an objective estimation of workload level. In this study, the authors evaluated the workload level related to complex training-based type task. Piloting simulation task was used to represent such type of task. The authors assess the EEG channel connections and found important connections for the estimation of workload level. This information can be used to build a more an EEG-based workload level estimator that is more efficient, i.e. less channels needed be used to accurately construct the estimation. The authors also found the significant brain signal frequency band that is related to the measure of workload in complex tasks. The problem of individual differences was also resolved using the proposed algorithm.