Advanced cognition can be measured from the human brain using brain-computer interfaces (BCIs). Integrating these interfaces with computer vision techniques, which possess efficient feature extraction capabilities, can achieve more robust and accurate detection of dim targets in aerial images. However, existing target detection methods primarily concentrate on homogeneous data, lacking efficient and versatile processing capabilities for heterogeneous multimodal data. In this article, we first build a brain-eye-computer-based object detection system for aerial images under few-shot conditions. This system detects suspicious targets using region proposal networks (RPNs), evokes the event-related potential (ERP) signal in electroencephalogram (EEG) through the eye-tracking-based slow serial visual presentation (ESSVP) paradigm, and constructs the EEG-image data pairs with eye movement data. Then, an adaptive modality balanced online knowledge distillation (AMBOKD) method is proposed to recognize dim objects with the EEG-image data. AMBOKD fuses EEG and image features using a multihead attention module, establishing a new modality with comprehensive features. To enhance the performance and robust capability of the fusion modality, simultaneous training and mutual learning between modalities are enabled by end-to-end online KD (OKD). During the learning process, an adaptive modality balancing module is proposed to ensure multimodal equilibrium by dynamically adjusting the weights of the importance and the training gradients across various modalities. The effectiveness and superiority of our method are demonstrated by comparing it with existing state-of-the-art methods. Additionally, experiments conducted on public datasets and real-world scenarios demonstrate the reliability and practicality of the proposed system and the designed method. The dataset and the source code can be found at: https://github.com/lizixing23/AMBOKD.

Image-based ground-to-aerial geo-localization aims to determine the geo-location of a ground query image by matching it with a large geo-tagged aerial image database. Due to the drastic difference between ground and aerial views, achieving high-accuracy geo-localization remains a huge challenge, especially in practical scenarios where ground query images have unknown orientations and even limited field-of-views (FoV). The incomplete information significantly hampers the process of learning discriminative features for image matching. In this article, we propose a novel automatic progressive learning (AMPLE) method for the orientation unknown geo-localization task. Specifically, we design a ConvNeXt-based network to effectively extract orientation-aware features from the two views. We then present two progressive training strategies without manually predefined training stages to promote the learning process. The first adaptively mines harder negative samples that contribute more to the loss, by automatically discarding redundant samples as the current best accuracy increases. The second leverages the proposed alignment-correlation hybrid (ACH) loss to guide model optimization in a progressive manner, gradually reducing the reliance on auxiliary orientation information. Extensive experiments on two benchmark datasets demonstrate that AMPLE outperforms state-of-the-art methods in orientation unknown, FoV limited, and cross-area tasks. Finally, we propose the concept of unifying unknown orientation tasks at different FoVs and show the cross-FoV generalization capability of our method.

Improving the decoding performance of steady-state visual evoked (SSVEP) signals is crucial for the practical application of SSVEP-based brain-computer interface (BCI) systems. Although numerous methods have achieved impressive results in decoding SSVEP signals, most of them focus only on the temporal or spectral domain information or concatenate them directly, which may ignore the complementary relationship between different features. To address this issue, we propose a dual-branch convolution-based Transformer network with multi-scale temporal-spectral feature fusion, termed MTSNet, to improve the decoding performance of SSVEP signals. Specifically, the temporal branch extracts temporal features from the SSVEP signals using the multi-level convolution- based Transformer (Convformer) that can adapt to the dynamic fluctuations of SSVEP signals. In parallel, the spectral branch takes the complex spectrum converted from temporal signals by the zero-padding fast Fourier transform as input and uses the Convformer to extract spectral features. These extracted temporal and spectral features are then integrated by the multi-scale feature fusion module to obtain comprehensive features with different scale information, thereby enhancing the interactions between the features and improving the effectiveness and robustness. Extensive experimental results on two widely used public SSVEP datasets, Benchmark and BETA, show that the proposed MTSNet significantly outperforms the state-of-the-art calibration-free methods in terms of accuracy and ITR. The superior performance demonstrates the effectiveness of our method in decoding SSVEP signals, which may facilitate the practical application of SSVEP-based BCI systems.

Automated object detection in aerial images is crucial in both civil and military applications. Existing computer vision-based object detection methods are not robust enough to precisely detect dim objects in aerial images due to the cluttered backgrounds, various observing angles, small object scales, and severe occlusions. Recently, electroencephalography (EEG)-based object detection methods have received increasing attention owing to the advanced cognitive capabilities of human vision. However, how to combine the human intelligence with computer intelligence to achieve robust dim object detection is still an open question. In this paper, we propose a novel approach to efficiently fuse and exploit the properties of multi-modal data for dim object detection. Specifically, we first design a brain-computer interface (BCI) paradigm called eye-tracking-based slow serial visual presentation (ESSVP) to simultaneously collect the paired EEG and image data when subjects search for the dim objects in aerial images. Then, we develop an attention-based multi-modal fusion network to selectively aggregate the learned features of EEG and image modalities. Furthermore, we propose an adaptive multi-teacher knowledge distillation method to efficiently train the multi-modal dim object detector for better performance. To evaluate the effectiveness of our method, we conduct extensive experiments on the collected dataset in subject-dependent and subject-independent tasks. The experimental results demonstrate that the proposed dim object detection method exhibits superior effectiveness and robustness compared to the baselines and the state-of-the-art methods.

Image based 3 Degrees-of-Freedom (DoF) cross-view geo-localization aims to estimate the position and orientation of a camera on the ground by matching the captured ground image with geo-tagged aerial images. However, most existing methods do not sufficiently exploit the difference between positive and negative samples for feature extraction, resulting in low localization accuracy. In this paper, we propose a novel method called HADGEO for accurate 3-DoF cross-view geo-localization. Specifically, we design a double-siamese structure with All Learnable Fully Convolutional Networks (ALFCN) to separately extract features from the aerial and ground images. To tap full potential of our network, we define a new weighted soft-margin triplet loss by integrating the Hard Sample Mining (HSM) strategy. This loss increases the training difficulty, forcing the network to be more discriminative for orientation-aware features. A series of experiments demonstrate that our method outperforms existing methods and achieves state-of-the-art performance on orientation unknown and Field-of-View (FoV) limited conditions, further improving the accuracy of 3-DoF geo-localization.

Object classification in aerial images is expected to play an important role in a wide range of applications. Multi-modal methods have emerged as a promising approach in aerial image classification due to the differences and comple-mentarities between different modalities. However, most existing methods simply combine multi-modal features or directly use a single optimization strategy for joint training, which is not comprehensive and usually constrains the classification accuracy. To mitigate this problem, we propose a multi-teacher multi-modal knowledge distillation (M2KD) method for aerial view object classification tasks. Specifically, the attention-based feature fusion network is first constructed to extract and merge more discriminative features from multi-modal data, i.e., the aerial images and electroencephalography (EEG) signals. To further improve the classification performance, the multi-teacher knowledge distillation framework is designed to assist the training of the student by leveraging the complementary multi-modal knowledge. Extensive experiments on the collected multi-modal dataset demonstrate the contribution and effectiveness of our M2KD method for aerial view object classification.

Deep learning methods have shown promise in various visual tasks such as object recognition. However, achieving robust and accurate performance in dim object recognition for remote sensing images remains challenging in the field of computer vision. This challenge can be attributed to factors such as cluttered backgrounds, varying observing angles, and limited availability of labeled data. In contrast, the human brain exhibits robust and efficient recognition of sensitive targets. To leverage the strengths of both computer calculation and human cognition, we propose a multimodal mutual learning with online knowledge distillation method (MMOKD) for object recognition. Our approach enables simultaneous training and mutual learning between modalities, where each modality serves as both a teacher and a student. A series of experiments are conducted to verify the potential of multimodal learning for object recognition. The results demonstrate that our approach not only enhances the robustness of multimodal fusion model, but also improves the accuracy of visual modality.

Due to the low signal-to-noise ratio, limited training samples, and large inter-subject variabilities in electroencephalogram (EEG) signals, developing a subject-independent brain-computer interface (BCI) system used for new users without any calibration is still challenging. In this letter, we propose a novel Multi-Attention Convolutional Recurrent mOdel (MACRO) for EEG-based event-related potential (ERP) detection in the subject-independent scenario. Specifically, the convolutional recurrent network is designed to capture the spatial-temporal features, while the multi-attention mechanism is integrated to focus on the most discriminative channels and temporal periods of EEG signals. Comprehensive experiments conducted on a benchmark dataset for RSVP-based BCIs show that our method achieves the best performance compared with the five state-of-the-art baseline methods. This result indicates that our method is able to extract the underlying subject-invariant EEG features and generalize to unseen subjects. Finally, the ablation studies verify the effectiveness of the designed multi-attention mechanism in MACRO for EEG-based ERP detection.

This paper presents an onboard micro aerial vehicle (MAV) localization algorithm VLO using onboard multi-sensor system consisting of a camera, a laser scanner and an inertial measurement unit. On the basis of onboard processor, the VLO can operate in real time without any prior information and ground assistance. Besides, it shows a strong robustness since it can work in both small and large, indoor and outdoor environment. As the main sensing devices of this system, the camera and laser scanner generate different characteristic data. The VLO fuses these two kinds of data for a more sufficient information about environment. A filter and an optimizer are then designed to estimate the MAV poses with extra onboard sensors data. Finally, an incremental dense map is updated. Different with vision-based or laser-based odometry, this system has no requirements for environments such as strong texture or structured surroundings. The Gazebo-based simulated and real MAV systems are built together for algorithm validation. The simulated and real results show that our onboard odometry VLO performs strong robustness without any prior information and basic assumptions.