Advanced cognition can be measured from the human brain using brain-computer interfaces (BCIs). Integrating these interfaces with computer vision techniques, which possess efficient feature extraction capabilities, can achieve more robust and accurate detection of dim targets in aerial images. However, existing target detection methods primarily concentrate on homogeneous data, lacking efficient and versatile processing capabilities for heterogeneous multimodal data. In this article, we first build a brain-eye-computer-based object detection system for aerial images under few-shot conditions. This system detects suspicious targets using region proposal networks (RPNs), evokes the event-related potential (ERP) signal in electroencephalogram (EEG) through the eye-tracking-based slow serial visual presentation (ESSVP) paradigm, and constructs the EEG-image data pairs with eye movement data. Then, an adaptive modality balanced online knowledge distillation (AMBOKD) method is proposed to recognize dim objects with the EEG-image data. AMBOKD fuses EEG and image features using a multihead attention module, establishing a new modality with comprehensive features. To enhance the performance and robust capability of the fusion modality, simultaneous training and mutual learning between modalities are enabled by end-to-end online KD (OKD). During the learning process, an adaptive modality balancing module is proposed to ensure multimodal equilibrium by dynamically adjusting the weights of the importance and the training gradients across various modalities. The effectiveness and superiority of our method are demonstrated by comparing it with existing state-of-the-art methods. Additionally, experiments conducted on public datasets and real-world scenarios demonstrate the reliability and practicality of the proposed system and the designed method. The dataset and the source code can be found at: https://github.com/lizixing23/AMBOKD.

Improving the decoding performance of steady-state visual evoked (SSVEP) signals is crucial for the practical application of SSVEP-based brain-computer interface (BCI) systems. Although numerous methods have achieved impressive results in decoding SSVEP signals, most of them focus only on the temporal or spectral domain information or concatenate them directly, which may ignore the complementary relationship between different features. To address this issue, we propose a dual-branch convolution-based Transformer network with multi-scale temporal-spectral feature fusion, termed MTSNet, to improve the decoding performance of SSVEP signals. Specifically, the temporal branch extracts temporal features from the SSVEP signals using the multi-level convolution- based Transformer (Convformer) that can adapt to the dynamic fluctuations of SSVEP signals. In parallel, the spectral branch takes the complex spectrum converted from temporal signals by the zero-padding fast Fourier transform as input and uses the Convformer to extract spectral features. These extracted temporal and spectral features are then integrated by the multi-scale feature fusion module to obtain comprehensive features with different scale information, thereby enhancing the interactions between the features and improving the effectiveness and robustness. Extensive experimental results on two widely used public SSVEP datasets, Benchmark and BETA, show that the proposed MTSNet significantly outperforms the state-of-the-art calibration-free methods in terms of accuracy and ITR. The superior performance demonstrates the effectiveness of our method in decoding SSVEP signals, which may facilitate the practical application of SSVEP-based BCI systems.

Automated object detection in aerial images is crucial in both civil and military applications. Existing computer vision-based object detection methods are not robust enough to precisely detect dim objects in aerial images due to the cluttered backgrounds, various observing angles, small object scales, and severe occlusions. Recently, electroencephalography (EEG)-based object detection methods have received increasing attention owing to the advanced cognitive capabilities of human vision. However, how to combine the human intelligence with computer intelligence to achieve robust dim object detection is still an open question. In this paper, we propose a novel approach to efficiently fuse and exploit the properties of multi-modal data for dim object detection. Specifically, we first design a brain-computer interface (BCI) paradigm called eye-tracking-based slow serial visual presentation (ESSVP) to simultaneously collect the paired EEG and image data when subjects search for the dim objects in aerial images. Then, we develop an attention-based multi-modal fusion network to selectively aggregate the learned features of EEG and image modalities. Furthermore, we propose an adaptive multi-teacher knowledge distillation method to efficiently train the multi-modal dim object detector for better performance. To evaluate the effectiveness of our method, we conduct extensive experiments on the collected dataset in subject-dependent and subject-independent tasks. The experimental results demonstrate that the proposed dim object detection method exhibits superior effectiveness and robustness compared to the baselines and the state-of-the-art methods.

Object classification in aerial images is expected to play an important role in a wide range of applications. Multi-modal methods have emerged as a promising approach in aerial image classification due to the differences and comple-mentarities between different modalities. However, most existing methods simply combine multi-modal features or directly use a single optimization strategy for joint training, which is not comprehensive and usually constrains the classification accuracy. To mitigate this problem, we propose a multi-teacher multi-modal knowledge distillation (M2KD) method for aerial view object classification tasks. Specifically, the attention-based feature fusion network is first constructed to extract and merge more discriminative features from multi-modal data, i.e., the aerial images and electroencephalography (EEG) signals. To further improve the classification performance, the multi-teacher knowledge distillation framework is designed to assist the training of the student by leveraging the complementary multi-modal knowledge. Extensive experiments on the collected multi-modal dataset demonstrate the contribution and effectiveness of our M2KD method for aerial view object classification.

Deep learning methods have shown promise in various visual tasks such as object recognition. However, achieving robust and accurate performance in dim object recognition for remote sensing images remains challenging in the field of computer vision. This challenge can be attributed to factors such as cluttered backgrounds, varying observing angles, and limited availability of labeled data. In contrast, the human brain exhibits robust and efficient recognition of sensitive targets. To leverage the strengths of both computer calculation and human cognition, we propose a multimodal mutual learning with online knowledge distillation method (MMOKD) for object recognition. Our approach enables simultaneous training and mutual learning between modalities, where each modality serves as both a teacher and a student. A series of experiments are conducted to verify the potential of multimodal learning for object recognition. The results demonstrate that our approach not only enhances the robustness of multimodal fusion model, but also improves the accuracy of visual modality.

Due to the low signal-to-noise ratio, limited training samples, and large inter-subject variabilities in electroencephalogram (EEG) signals, developing a subject-independent brain-computer interface (BCI) system used for new users without any calibration is still challenging. In this letter, we propose a novel Multi-Attention Convolutional Recurrent mOdel (MACRO) for EEG-based event-related potential (ERP) detection in the subject-independent scenario. Specifically, the convolutional recurrent network is designed to capture the spatial-temporal features, while the multi-attention mechanism is integrated to focus on the most discriminative channels and temporal periods of EEG signals. Comprehensive experiments conducted on a benchmark dataset for RSVP-based BCIs show that our method achieves the best performance compared with the five state-of-the-art baseline methods. This result indicates that our method is able to extract the underlying subject-invariant EEG features and generalize to unseen subjects. Finally, the ablation studies verify the effectiveness of the designed multi-attention mechanism in MACRO for EEG-based ERP detection.