Vision-based target motion estimation is a fundamental problem in many robotic tasks. The existing methods have the limitation of low observability and, hence, face challenges in tracking highly maneuverable targets. Motivated by the aerial target pursuit task where a target may maneuver in 3D space, this paper studies how to further enhance observability by incorporating the bearing rate information that has not been well explored in the literature. The main contribution of this paper is to propose a new cooperative estimator called STT-R (Spatial-Temporal Triangulation with bearing Rate), which is designed under the framework of distributed recursive least squares. This theoretical result is further verified by numerical simulation and real-world experiments. It is shown that the proposed STT-R algorithm can effectively generate more accurate estimations and effectively reduce the lag in velocity estimation, enabling tracking of more maneuverable targets.

Although acrobatic flight control has been studied extensively, one key limitation of the existing methods is that they are usually restricted to specific maneuver tasks and cannot change flight pattern parameters online. In this work, we propose a target-and-command-oriented reinforcement learning (TACO) framework, which can handle different maneuver tasks in a unified way and allows online parameter changes. We also propose a spectral normalization method with input-output rescaling to enhance the policy’s temporal and spatial smoothness, independence, and symmetry, thereby overcoming the sim-to-real gap. We validate the TACO approach through extensive simulation and real-world experiments, demonstrating its ability to achieve high-speed, high-accuracy circular flights and continuous multi-flips. The code is available at https://github.com/yinzikang/TACO.

MAV-capturing-MAV (MCM) is one of the few effective methods for physically countering misused or malicious MAVs. This paper presents a vision-based cooperative MCM system, where multiple pursuer MAVs equipped with onboard vision systems detect, localize, and pursue a target MAV. To enhance robustness, a distributed state estimation and control framework enables the pursuer MAVs to autonomously coordinate their actions. Pursuer trajectories are optimized using Model Predictive Control (MPC) and executed via a low-level SO(3) controller, ensuring smooth and stable pursuit. Once the capture conditions are satisfied, the pursuer MAVs automatically deploy a flying net to intercept the target. These capture conditions are determined based on the predicted motion of the net. To enable real-time decision-making, we propose a lightweight computational method to approximate the net’s motion, avoiding the prohibitive cost of solving the full net dynamics. The effectiveness of the proposed system is validated through simulations and real-world experiments. In real-world tests, our approach successfully captures a moving target traveling at 4 m/s with an acceleration of 1 m/s2, achieving a success rate of 64.7%.

In recent years, there has been a growing interest in the visual detection of micro aerial vehicles (MAVs) due to its importance in numerous applications. However, the existing methods based on either appearance or motion features encounter difficulties when the background is complex or the MAV is too small. In this paper, we propose a novel motion-guided MAV detector that can accurately identify small MAVs in complex and non-planar scenes. This detector first exploits a motion feature enhancement module to capture the motion features of small MAVs. Then it uses multi-object tracking and trajectory filtering to eliminate false positives caused by motion parallax . Finally, an appearance-based classifier and an appearance-based detector that operates on the cropped regions are used to achieve precise detection results. Our proposed method can effectively and efficiently detect extremely small MAVs from dynamic and complex backgrounds because it aggregates pixel-level motion features and eliminates false positives based on the motion and appearance features of MAVs. Experiments on the ARD-MAV dataset demonstrate that the proposed method could achieve high performance in small MAV detection under challenging conditions and outperform other state-of-the-art methods across various metrics.

Visual detection of micro aerial vehicles (MAVs) is an important problem in many tasks such as vision-based swarming of MAVs. This paper studies vision-based 6D pose estimation to detect a 3D bounding box of a target MAV and then estimate its 3D position and 3D attitude. The 3D attitude information is critical to better estimate the target's velocity since the attitude and motion are dynamically coupled. In this paper, we propose a novel 6D pose estimation method, whose novelties are threefold. First, we propose a novel centroid point-guided keypoint localization network that outperforms the state-of-the-art methods in terms of both accuracy and efficiency. Second, while there are no publicly available real-world datasets for 6D pose estimation for MAVs up to now, we propose a high-quality dataset based on an automatic dataset collection method. Third, since the dataset is collected in an indoor environment but detection tasks are usually in outdoor environments, we propose a self-training-based unsupervised domain adaption method to transfer the method from indoor to outdoor. Finally, we show that the estimated 6D pose especially the 3D attitude can significantly help improve the target's velocity estimation.

Malicious use of micro aerial vehicles (MAVs) has become a serious threat to public safety and personal privacy in recent years. Motivated by this problem, we propose a systematic approach to monitor the intrusion of malicious MAVs based on a novel type of panoramic stereo camera networks. Each sensing node of such a network consists of 16 lenses that can form a 360-degree panoramic vision system. The 16 lenses further form 8 pairs of stereo cameras that can directly localize aerial targets. The effective range for a sensing node localizing a MAV like DJI M300 could reach 80 meters, which is much farther than existing commercial stereo cameras. In terms of algorithms, we propose i) a novel visual MAV detection algorithm based primarily on motion features of MAVs, ii) an efficient stereo localization algorithm based on sparse feature points, and iii) robust multi-target tracking and trajectory fusion algorithm to fuse the observations of different sensing nodes. The effectiveness, robustness, and accuracy of the proposed algorithms together with the overall system have been verified by extensive experimental tests. To the best of our knowledge, this is the first systematic approach to detect, localize, and track unknown MAVs in the literature. Our approach provides a scalable solution to securely cover large areas of interest against malicious MAV intrusion. Note to Practitioners—Micro aerial vehicles (MAVs) have been widely used in many domains nowadays. However, they have also brought many safety problems. To monitor the intrusion of malicious MAVs, this paper proposes a novel type of panoramic stereo camera networks that can detect, localize, and track multiple MAVs simultaneously. Such a network consists of a number of sensing nodes and a central node. Each sensing node is able to detect, localize, and track multiple MAV targets. The role of the central node is to fuse the observations from multiple sensing nodes to generate more accurate trajectories of the MAV targets and in the meantime secure a large area in a coordinated way. This paper presents the details of the prototype of the system and the key algorithms therein.