Brain decoding currently faces significant challenges in individual differences, modality alignment, and high-dimensional embeddings. To address individual differences, researchers often use source subject data, which leads to issues such as privacy leakage and heavy data storage burdens. In modality alignment, current works focus on aligning the softmax probability distribution but neglect the alignment of marginal probability distributions, resulting in modality misalignment. Additionally, images and text are aligned separately with fMRI without considering the complex interplay between images and text, leading to poor image reconstruction. Finally, the enormous dimensionality of CLIP embeddings causes significant computational costs. Although the dimensionality of CLIP embeddings can be reduced by ignoring the number of patches obtained from images and the number of tokens acquired from text, this comes at the cost of a significant drop in model performance, creating a dilemma. To overcome these limitations, we propose a source-free domain adaptation-based brain decoding framework. Firstly, we apply source-free domain adaptation, which only acquires the source model without accessing source data during target model adaptation, to brain decoding to address cross-subject variations, privacy concerns, and the heavy burden of data storage. Secondly, we employ maximum mean discrepancy (MMD) to align the marginal probability distributions between embeddings of different modalities. Moreover, to accommodate the complex interplay between image and text, we concatenate the embeddings of image and text and then use singular value decomposition (SVD) to obtain a new embedding. What’s more, to achieve better image generation quality, we employ the Wasserstein distance (WD) to align the probability distributions of new embeddings. Finally, in the target model adaptation phase of source-free domain adaptation, we employ low-rank adaptation (LoRA) to reduce the high expense of tuning the target model. Sufficient experiments demonstrate our work outperforms state-of-the-art methods for brain decoding tasks.

Electroencephalogram (EEG)-based motor imagery (MI) signals are frequently used in brain-computer interfaces (BCIs) due to their wide applications in the rehabilitation field. However, cross-subject variations often result in a model trained on one participant failing when applied to another. Additionally, privacy concerns regarding sensitive health and mental information in EEG-based MI signals further complicate the situation. Source-free domain adaptation aims to address these cross-subject variations by transferring knowledge from a source domain (i.e., a previous participant) to a target domain (i.e., a new participant) without accessing sensitive source data. However, source-free unsupervised domain adaptation models often face issues with incorrect pseudo-labels, which can lead to unstable and ineffective adaptation. To address this, we propose a source-free semi-supervised domain adaptation algorithm for EEG-based MI signal classification. This algorithm tackles noise accumulation caused by incorrect pseudo-labels while effectively handling data distribution variations and privacy concerns, similar to source-free unsupervised domain adaptation models. Specifically, we train the classifier head using only a limited amount of labeled target data to prevent noise accumulation, and generate pseudo-labels for the unlabeled target data. Furthermore, we introduce an independent self-training head that learns better representations using the generated pseudo-labels, mitigating overfitting caused by the limited labeled target data. Additionally, we design an adversarial head that plays a minimax game to extract more discriminative feature representations from the unlabeled target data. Extensive experiments on three benchmark datasets, compared with eighteen state-of-the-art SFDA methods, demonstrate the superiority of our approach.