Advanced cognition can be measured from the human brain using brain-computer interfaces (BCIs). Integrating these interfaces with computer vision techniques, which possess efficient feature extraction capabilities, can achieve more robust and accurate detection of dim targets in aerial images. However, existing target detection methods primarily concentrate on homogeneous data, lacking efficient and versatile processing capabilities for heterogeneous multimodal data. In this article, we first build a brain-eye-computer-based object detection system for aerial images under few-shot conditions. This system detects suspicious targets using region proposal networks (RPNs), evokes the event-related potential (ERP) signal in electroencephalogram (EEG) through the eye-tracking-based slow serial visual presentation (ESSVP) paradigm, and constructs the EEG-image data pairs with eye movement data. Then, an adaptive modality balanced online knowledge distillation (AMBOKD) method is proposed to recognize dim objects with the EEG-image data. AMBOKD fuses EEG and image features using a multihead attention module, establishing a new modality with comprehensive features. To enhance the performance and robust capability of the fusion modality, simultaneous training and mutual learning between modalities are enabled by end-to-end online KD (OKD). During the learning process, an adaptive modality balancing module is proposed to ensure multimodal equilibrium by dynamically adjusting the weights of the importance and the training gradients across various modalities. The effectiveness and superiority of our method are demonstrated by comparing it with existing state-of-the-art methods. Additionally, experiments conducted on public datasets and real-world scenarios demonstrate the reliability and practicality of the proposed system and the designed method. The dataset and the source code can be found at: https://github.com/lizixing23/AMBOKD.

Deep reinforcement learning (DRL) has recently emerged as a promising tool for tackling pursuit-evasion tasks. However, most existing DRL-based pursuit approaches still rely on individual rewards and struggle with complex scenarios. To address these challenges, we propose a knowledge-enhanced DRL approach for multi-agent pursuit-evasion in complex environments. Specifically, the cooperative pursuit problem is modeled as a decentralized partially observable Markov decision process from each pursuers perspective, where the team reward function is elaborately designed to encourage collaborative behavior and enhance team coordination. Then, a novel knowledge enhanced multi-agent twin delayed deep deterministic policy gradient (KE-MATD3) algorithm is presented to efficiently learn the cooperative pursuit policy. By integrating a knowledge enhancement mechanism that extracts effective information from an improved artificial potential field method, the cooperative pursuit policy achieves more robust convergence, mitigating the local optima that typically arise from individual reward-based learning. Finally, extensive numerical simulations and real-world experiments validate the efficiency and superiority of the proposed approach, demonstrating emergent cooperative behaviors among the pursuers.

Image-based ground-to-aerial geo-localization aims to determine the geo-location of a ground query image by matching it with a large geo-tagged aerial image database. Due to the drastic difference between ground and aerial views, achieving high-accuracy geo-localization remains a huge challenge, especially in practical scenarios where ground query images have unknown orientations and even limited field-of-views (FoV). The incomplete information significantly hampers the process of learning discriminative features for image matching. In this article, we propose a novel automatic progressive learning (AMPLE) method for the orientation unknown geo-localization task. Specifically, we design a ConvNeXt-based network to effectively extract orientation-aware features from the two views. We then present two progressive training strategies without manually predefined training stages to promote the learning process. The first adaptively mines harder negative samples that contribute more to the loss, by automatically discarding redundant samples as the current best accuracy increases. The second leverages the proposed alignment-correlation hybrid (ACH) loss to guide model optimization in a progressive manner, gradually reducing the reliance on auxiliary orientation information. Extensive experiments on two benchmark datasets demonstrate that AMPLE outperforms state-of-the-art methods in orientation unknown, FoV limited, and cross-area tasks. Finally, we propose the concept of unifying unknown orientation tasks at different FoVs and show the cross-FoV generalization capability of our method.

Improving the decoding performance of steady-state visual evoked (SSVEP) signals is crucial for the practical application of SSVEP-based brain-computer interface (BCI) systems. Although numerous methods have achieved impressive results in decoding SSVEP signals, most of them focus only on the temporal or spectral domain information or concatenate them directly, which may ignore the complementary relationship between different features. To address this issue, we propose a dual-branch convolution-based Transformer network with multi-scale temporal-spectral feature fusion, termed MTSNet, to improve the decoding performance of SSVEP signals. Specifically, the temporal branch extracts temporal features from the SSVEP signals using the multi-level convolution- based Transformer (Convformer) that can adapt to the dynamic fluctuations of SSVEP signals. In parallel, the spectral branch takes the complex spectrum converted from temporal signals by the zero-padding fast Fourier transform as input and uses the Convformer to extract spectral features. These extracted temporal and spectral features are then integrated by the multi-scale feature fusion module to obtain comprehensive features with different scale information, thereby enhancing the interactions between the features and improving the effectiveness and robustness. Extensive experimental results on two widely used public SSVEP datasets, Benchmark and BETA, show that the proposed MTSNet significantly outperforms the state-of-the-art calibration-free methods in terms of accuracy and ITR. The superior performance demonstrates the effectiveness of our method in decoding SSVEP signals, which may facilitate the practical application of SSVEP-based BCI systems.

Deep reinforcement learning (DRL) has emerged as a promising solution for autonomous operations of autonomous aerial vehicles (AAVs) in unknown environments. However, learning to navigate and avoid obstacles under sparse reward settings remains challenging. In this article, we propose an end-to-end learning approach that synthesizes imitation learning with DRL for AAV navigation and obstacle avoidance. Specifically, we formulate this problem as a partially observable Markov decision process with sparse rewards and learn an end-to-end policy that maps imperfect sensor data to control signals. To efficiently optimize the policy under the sparse reward setting, we propose the selective behavior cloning enhanced actor-critic (SBCAC) algorithm. By integrating an experience filter and a Q-value based action selector to selectively mimic an artificial potential field based non-expert policy, our approach significantly improves the learning performance and sample efficiency. Extensive simulations with fixed-wing and multi-rotor AAVs in different scenarios demonstrate that SBCAC achieves an average improvement of up to 16.07% in success rate, a 72.46% reduction in crash rate, and a 94.62% reduction in stray rate compared to the state-of-the-art selective imitation baseline. Furthermore, hardware-in-the-loop and physical experiments validate the effectiveness of our approach, showing its potential for practical applications in complex environments.

Flocking with multiple unmanned aerial vehicles (UAVs) offers significant potential for diverse applications due to its enhanced maneuverability, improved efficiency, and increased robustness. Collision avoidance is a critical and challenging issue for distributed flocking control with a UAV fleet, especially in dynamic environments with varying numbers of non-cooperative intruders. However, existing reinforcement learning based methods mainly focus on flocking with collision avoidance tasks with static obstacles and a fixed number of UAVs. In this article, we propose a scalable multi-agent reinforcement learning based method to solve the distributed flocking with collision avoidance problem for a scalable fleet of fixed-wing UAVs in dynamic environments. Specifically, we cast this problem in a decentralized partially observable Markov decision process framework and propose a scalable multi-agent reinforcement learning algorithm called spatial-temporal attention multi-agent actor-critic (STAAC). In this algorithm, we design a spatial-temporal attention based population-invariant network architecture to facilitate the representation learning of dynamic dimensional observations. By integrating the local spatial attention and global temporal attention mechanisms, STAAC is able to adapt to the changes in the scale of UAV fleets and the number of intruders. Finally, we empirically demonstrate the effectiveness, scalability, and adaptability of the proposed approach in numerical simulations and hardware-in-the-loop experiments.

Automated object detection in aerial images is crucial in both civil and military applications. Existing computer vision-based object detection methods are not robust enough to precisely detect dim objects in aerial images due to the cluttered backgrounds, various observing angles, small object scales, and severe occlusions. Recently, electroencephalography (EEG)-based object detection methods have received increasing attention owing to the advanced cognitive capabilities of human vision. However, how to combine the human intelligence with computer intelligence to achieve robust dim object detection is still an open question. In this paper, we propose a novel approach to efficiently fuse and exploit the properties of multi-modal data for dim object detection. Specifically, we first design a brain-computer interface (BCI) paradigm called eye-tracking-based slow serial visual presentation (ESSVP) to simultaneously collect the paired EEG and image data when subjects search for the dim objects in aerial images. Then, we develop an attention-based multi-modal fusion network to selectively aggregate the learned features of EEG and image modalities. Furthermore, we propose an adaptive multi-teacher knowledge distillation method to efficiently train the multi-modal dim object detector for better performance. To evaluate the effectiveness of our method, we conduct extensive experiments on the collected dataset in subject-dependent and subject-independent tasks. The experimental results demonstrate that the proposed dim object detection method exhibits superior effectiveness and robustness compared to the baselines and the state-of-the-art methods.

Image based 3 Degrees-of-Freedom (DoF) cross-view geo-localization aims to estimate the position and orientation of a camera on the ground by matching the captured ground image with geo-tagged aerial images. However, most existing methods do not sufficiently exploit the difference between positive and negative samples for feature extraction, resulting in low localization accuracy. In this paper, we propose a novel method called HADGEO for accurate 3-DoF cross-view geo-localization. Specifically, we design a double-siamese structure with All Learnable Fully Convolutional Networks (ALFCN) to separately extract features from the aerial and ground images. To tap full potential of our network, we define a new weighted soft-margin triplet loss by integrating the Hard Sample Mining (HSM) strategy. This loss increases the training difficulty, forcing the network to be more discriminative for orientation-aware features. A series of experiments demonstrate that our method outperforms existing methods and achieves state-of-the-art performance on orientation unknown and Field-of-View (FoV) limited conditions, further improving the accuracy of 3-DoF geo-localization.

Object classification in aerial images is expected to play an important role in a wide range of applications. Multi-modal methods have emerged as a promising approach in aerial image classification due to the differences and comple-mentarities between different modalities. However, most existing methods simply combine multi-modal features or directly use a single optimization strategy for joint training, which is not comprehensive and usually constrains the classification accuracy. To mitigate this problem, we propose a multi-teacher multi-modal knowledge distillation (M2KD) method for aerial view object classification tasks. Specifically, the attention-based feature fusion network is first constructed to extract and merge more discriminative features from multi-modal data, i.e., the aerial images and electroencephalography (EEG) signals. To further improve the classification performance, the multi-teacher knowledge distillation framework is designed to assist the training of the student by leveraging the complementary multi-modal knowledge. Extensive experiments on the collected multi-modal dataset demonstrate the contribution and effectiveness of our M2KD method for aerial view object classification.

Deep learning methods have shown promise in various visual tasks such as object recognition. However, achieving robust and accurate performance in dim object recognition for remote sensing images remains challenging in the field of computer vision. This challenge can be attributed to factors such as cluttered backgrounds, varying observing angles, and limited availability of labeled data. In contrast, the human brain exhibits robust and efficient recognition of sensitive targets. To leverage the strengths of both computer calculation and human cognition, we propose a multimodal mutual learning with online knowledge distillation method (MMOKD) for object recognition. Our approach enables simultaneous training and mutual learning between modalities, where each modality serves as both a teacher and a student. A series of experiments are conducted to verify the potential of multimodal learning for object recognition. The results demonstrate that our approach not only enhances the robustness of multimodal fusion model, but also improves the accuracy of visual modality.

Multiple unmanned aerial vehicles (UAVs) are able to efficiently accomplish a variety of tasks in complex scenarios. However, developing a collision-avoiding flocking policy for multiple fixed-wing UAVs is still challenging, especially in obstacle-cluttered environments. In this article, we propose a novel curriculum-based multiagent deep reinforcement learning (MADRL) approach called task-specific curriculum-based MADRL (TSCAL) to learn the decentralized flocking with obstacle avoidance policy for multiple fixed-wing UAVs. The core idea is to decompose the collision-avoiding flocking task into multiple subtasks and progressively increase the number of subtasks to be solved in a staged manner. Meanwhile, TSCAL iteratively alternates between the procedures of online learning and offline transfer. For online learning, we propose a hierarchical recurrent attention multiagent actor-critic (HRAMA) algorithm to learn the policies for the corresponding subtask(s) in each learning stage. For offline transfer, we develop two transfer mechanisms, i.e., model reload and buffer reuse, to transfer knowledge between two neighboring stages. A series of numerical simulations demonstrate the significant advantages of TSCAL in terms of policy optimality, sample efficiency, and learning stability. Finally, the high-fidelity hardware-in-the-loop (HITL) simulation is conducted to verify the adaptability of TSCAL. A video about the numerical and HITL simulations is available at https://youtu.be/R9yLJNYRIqY.

A swarm of fixed-wing unmanned aerial vehicles (UAVs) is expected to efficiently accomplish various tasks in complex scenarios. This paper proposes an attention-based population-invariant multi-agent deep reinforcement learning (MADRL) approach to deal with the decentralized collision-free flocking problem for a scalable fixed-wing UAV swarm. First, this problem is modeled as a decentralized partially observable Markov decision process from the perspective of each follower. Then, an improved multi-agent deep deterministic policy gradient (MADDPG) algorithm is presented to efficiently learn the population-invariant flocking policy. In this algorithm, the parameter sharing with ego-centric representation mechanism is incorporated to improve learning efficiency. Besides, the attention-based population-invariant network structure (APINet) is designed by leveraging the self-attention mechanism. With this structure, the learned flocking policy is invariant to the population of the swarm. Finally, both numerical and hardware-in-the-loop simulation results verify the efficiency and scalability of the proposed approach.

The evolution of artificial intelligence and Internet of Things (IoT) envision a highly integrated artificial IoT (AIoT) network. Flocking and cooperation with multiple unmanned aerial vehicles (UAVs) are expected to play a vital role in industrial AIoT networks. In this article, we formulate the collision-free flocking problem of fixed-wing UAVs as a Markov decision process and solve it in the deep reinforcement learning (DRL) framework. Our method can deal with a variable number of followers by encoding the dynamic environmental state into a fixed-length embedding tensor. Specifically, each follower constructs a fixed-size local situation map that describes the collision risks with other followers nearby. The local situation maps are used by a proposed DRL algorithm to learn the collision-free flocking behavior. To further improve the learning efficiency, we design a reference-point-based action selection strategy and an adaptive mechanism. We compare the proposed MA2D3QN algorithm with several benchmark DRL algorithms through numerical simulation, and we verify its advantages in learning efficiency and performance. Finally, we demonstrate the scalability and adaptability of MA2D3QN in a semiphysical simulation experiment.

In this article, a multilayered and distributed architecture for mission-oriented miniature fixed-wing UAV swarms is presented. Based on the concept of modularity, the proposed architecture divides the overall system into five layers: 1) low-level control layer; 2) high-level control layer; 3) coordination layer; 4) communication layer; and 5) human interaction layer, and many modules that can be viewed as black boxes with interfaces of inputs and outputs. In this way, not only the complexity of developing a large system can be reduced but also the versatility of supporting diversified missions can be ensured. Furthermore, the proposed architecture is fully distributed that each UAV performs the decision-making procedure autonomously so as to achieve better scalability. Moreover, different kinds of aerial platforms can be feasibly extended by using the control allocation matrices and the integrated hardware box. A prototype swarm system based on the proposed architecture is built and the proposed architecture is evaluated through field experiments with a scale of 21 fixed-wing UAVs. Particularly, to the best of our knowledge, this article is the first work which successfully demonstrates formation flight, target recognition, and tracking missions within an integrated architecture for fixed-wing UAV swarms through field experiments.

Developing the flocking behavior for a dynamic squad of fixed-wing UAVs is still a challenge due to kinematic complexity and environmental uncertainty. In this paper, we deal with the decentralized flocking and collision avoidance problem through deep reinforcement learning (DRL). Specifically, we formulate a decentralized DRL-based decision making framework from the perspective of every follower, where a collision avoidance mechanism is integrated into the flocking controller. Then, we propose a novel reinforcement learning algorithm PS-CACER for training a shared control policy for all the followers. Besides, we design a plug-n-play embedding module based on convolutional neural networks and the attention mechanism. As a result, the variable-length system state can be encoded into a fixed-length embedding vector, which makes the learned DRL policy independent with the number and the order of followers. Finally, numerical simulation results demonstrate the effectiveness of the proposed method, and the learned policies can be directly transferred to semi-physical simulation without any parameter finetuning.

Due to the low signal-to-noise ratio, limited training samples, and large inter-subject variabilities in electroencephalogram (EEG) signals, developing a subject-independent brain-computer interface (BCI) system used for new users without any calibration is still challenging. In this letter, we propose a novel Multi-Attention Convolutional Recurrent mOdel (MACRO) for EEG-based event-related potential (ERP) detection in the subject-independent scenario. Specifically, the convolutional recurrent network is designed to capture the spatial-temporal features, while the multi-attention mechanism is integrated to focus on the most discriminative channels and temporal periods of EEG signals. Comprehensive experiments conducted on a benchmark dataset for RSVP-based BCIs show that our method achieves the best performance compared with the five state-of-the-art baseline methods. This result indicates that our method is able to extract the underlying subject-invariant EEG features and generalize to unseen subjects. Finally, the ablation studies verify the effectiveness of the designed multi-attention mechanism in MACRO for EEG-based ERP detection.

Controlling a squad of fixed-wing UAVs is challenging due to the kinematics complexity and the environmental dynamics. In this paper, we develop a novel actor-critic reinforcement learning approach to solve the leader-follower flocking problem in continuous state and action spaces. Specifically, we propose a CACER algorithm that uses multilayer perceptron to represent both the actor and the critic, which has a deeper structure and provides a better function approximator than the original continuous actor-critic learning automation (CACLA) algorithm. Besides, we propose a double prioritized experience replay (DPER) mechanism to further improve the training efficiency. Specifically, the state transition samples are saved into two different experience replay buffers for updating the actor and the critic separately, based on the calculation of sample priority using the temporal difference errors. We have not only compared CACER with CACLA and a benchmark deep reinforcement learning algorithm DDPG in numerical simulation, but also demonstrated the performance of CACER in semi-physical simulation by transferring the learned policy in the numerical simulation without parameter tuning.

Images obtained by unmanned aerial vehicle (UAV) are superior in many aspects such as convenience, low cost and so on. However, the obtained attitude of the UAV is usually not accurate, which further leads to the decrease of efficiency and accuracy of the algorithm based on the position and attitude information. Despite the algorithm based on image feature can get a more accurate result, it usually requires a longer time to process the images and even can bring a large cumulative error when continuous image is stitched. This paper presents a new method for stitching panoramas from UAV's images which is mainly composed of three steps. Firstly, once the images have been obtained, global motion model for UAV aerial photography can be used to establish a predicted region. Second, features in the onboard images are matched by using SIFT algorithm in this region. Finally, the accuracy of image mosaic is predicted through the Grey Relational Analysis. Then, according to the result, the algorithm process can adjust itself automatically to meet the needs of different types of sensor image mosaic. We demonstrate through flight experiments that compared with conventional approach our method can stitch images accurately and rapidly and produce high-quality panoramas.

Parallel processing system has been extensively developed and used in numerous commercial servers for large-scale data analysis. However, the issues of scalability, reliability and efficiency cannot be achieved simultaneously. Motivated by this observation, a Multi-level Parallel Database System based on B-tree structure (MPDBS) is designed for large-scale structured data and semi-structured data. Correspondingly, a multi-level index scheme (MLIS) is proposed in this paper. Based on MPDBS framework and MLIS scheme, the system can parallel execute analyzing task and full-text query efficiently, meanwhile reducing the network I/O and disk I/O greatly. The optimal architecture of MPDBS is also derived by mathematical approach. Experimental results show that, given the same hardware configuration and TPC-H benchmark, comparing with Hive using Hadoop Distributed File System (HDFS), the query (i.e., statistical query, keyword query and point query) latency on 200GB commercial data for the proposed MPDBS is declined by 95%.

Switching behavior among different swimming modes of fish is a normal phenomenon in nature. We find that there is the joints' vibration fact for robotic fish in the process of switching. It is believed that the difference might be caused by discontinuous driven signal. This paper analyzes the discontinuous signal's the effect on the robotic fish joints and fin surface. Furthermore, this paper proposes an effective phase compensation method based on sinusoidal model to solve the discontinuous problem that enables robotic fish to mimic this spontaneous mode switching behavior of live fish. Finally, experimental results illustrate that the phase compensation method shows the superiority in saving energy and reducing the complexity of the control tracking algorithm.

We propose Birds: a bare-metal recovery system for instant restoration of data services, focusing on a general-purpose automatic backup-and-recovery approach to protect data and resume data services from scratch instantly after disasters. We design BIRDS to possess two appealing features: full automation in the backup-and-recovery process, and instant data service resumption after disasters. BIRDS achieves the former one with automatic whole system replication and restoration, by taking the backup process outside of the protected system with the help of a novel non-intrusive light-weight physical to virtual conversion method. The latter one is enabled by a novel pipelined parallel recovery mechanism, which allows data services being instantly resumed while data recovery between the backup data center and the production site is still in progress. We implemented a BIRDS prototype and evaluated it using standard benchmarks. We show that BIRDS outperforms existing disaster recovery techniques by the means of recovery efficiency while introducing relatively small runtime overhead. Furthermore, BIRDS can be directly applied to any existing system in a plug-and-protect fashion without requiring re-installation or any modification of the existing system.