In recent years, technical advancement and robotic technologies highlight the way for Unmanned Aerial Vehicles (UAVs) researchers to enlarge and increase their capabilities so that many can act as autonomous systems on their own. For this reason, UAV swarming has caught the attention of several researchers. This paper proposes a hybrid swarm control architecture focused on a centralized node to deal with an optimal allocation task problem within firefighting application scenarios. Furthermore, we will apply Jonker-Volgenant auction-based algorithm and present a variation re-enhanced: Iterative Jonker-Volgenant method for our specific new algorithm. The results show robustness in a controlled simulation and computational efficiency for task allocation problems even for many UAVs (approximately 100 UAVs) and target tasks (about 50 targets).

Execution control is a critical task of robot architectures which has a deep impact on the quality of the final system. In this study, we describe a general method for execution control, which is a part of the Aerostack software framework for aerial robotics, and present technical challenges for execution control and design decisions to develop the method. The proposed method has an original design combining a distributed approach for execution control of behaviors (such as situation checking and performance monitoring) and centralizes coordination to ensure consistency of the concurrent execution. We conduct experiments to evaluate the method. The experimental results show that the method is general and usable with acceptable development efforts to efficiently work on different types of aerial missions. The method is supported by standards based on a robot operating system (ROS) contributing to its general use, and an open-source project is integrated in the Aerostack framework. Therefore, its technical details are fully accessible to developers and freely available to be used in the development of new aerial robotic systems.

In this paper we propose a particle filter localization approach, based on stereo visual odometry (VO) and semantic information from indoor environments, for mini-aerial robots. The prediction stage of the particle filter is performed using the 3D pose of the aerial robot estimated by the stereo VO algorithm. This predicted 3D pose is updated using inertial as well as semantic measurements. The algorithm processes semantic measurements in two phases; firstly, a pre-trained deep learning (DL) based object detector is used for real time object detections in the RGB spectrum. Secondly, from the corresponding 3D point clouds of the detected objects, we segment their dominant horizontal plane and estimate their relative position, also augmenting a prior map with new detections. The augmented map is then used in order to obtain a drift free pose estimate of the aerial robot. We validate our approach in several real flight experiments where we compare it against ground truth and a state of the art visual SLAM approach.

Obstacle avoidance is a key feature for safe Unmanned Aerial Vehicle (UAV) navigation. While solutions have been proposed for static obstacle avoidance, systems enabling avoidance of dynamic objects, such as drones, are hard to implement due to the detection range and field-of-view (FOV) requirements, as well as the constraints for integrating such systems on-board small UAVs. In this work, a dataset of 6k synthetic depth maps of drones has been generated and used to train a state-of-the-art deep learning-based drone detection model. While many sensing technologies can only provide relative altitude and azimuth of an obstacle, our depth map-based approach enables full 3D localization of the obstacle. This is extremely useful for collision avoidance, as 3D localization of detected drones is key to perform efficient collision-free path planning. The proposed detection technique has been validated in several real depth map sequences, with multiple types of drones flying at up to 2 m/s, achieving an average precision of 98.7 %, an average recall of 74.7 % and a record detection range of 9.5 meters.

Deep learning techniques for motion control have recently been qualitatively improved, since the successful application of Deep Q- Learning to the continuous action domain in Atari-like games. Based on these ideas, Deep Deterministic Policy Gradients (DDPG) algorithm was able to provide impressive results in continuous state and action domains, which are closely linked to most of the robotics-related tasks. In this paper, a vision-based autonomous multirotor landing maneuver on top of a moving platform is presented. The behaviour has been completely learned in simulation without prior human knowledge and by means of deep reinforcement learning techniques. Since the multirotor is controlled in attitude, no high level state estimation is required. The complete behaviour has been trained with continuous action and state spaces, and has provided proper results (landing at a maximum velocity of 2 m/s), Furthermore, it has been validated in a wide variety of conditions, for both simulated and real-flight scenarios, using a low-cost, lightweight and out-of-the-box consumer multirotor.

Navigation in unknown indoor environments with fast collision avoidance capabilities is an ongoing research topic. Traditional motion planning algorithms rely on precise maps of the environment, where re-adapting a generated path can be highly demanding in terms of computational cost. In this paper, we present a fast reactive navigation algorithm using Deep Reinforcement Learning applied to multi rotor aerial robots. Taking as input the 2D-laser range measurements and the relative position of the aerial robot with respect to the desired goal, the proposed algorithm is successfully trained in a Gazebo-based simulation scenario by adopting an artificial potential field formulation. A thorough evaluation of the trained agent has been carried out both in simulated and real indoor scenarios, showing the appropriate reactive navigation behavior of the agent in the presence of static and dynamic obstacles.

In this paper, we propose a novel Image-Based Visual Servoing (IBVS) controller for multirotor aerial robots based on a recent deep reinforcement learning algorithm named Deep Deterministic Policy Gradients (DDPG). The proposed RL-IBVS controller is successfully trained in a Gazebo-based simulation scenario in order to learn the appropriate IBVS policy for directly mapping a state, based on errors in the image, to the linear velocity commands of the aerial robot. A thorough validation of the proposed controller has been conducted in simulated and real flight scenarios, demonstrating outstanding capabilities in object following applications. Moreover, we conduct a detailed comparison of the RL-IBVS controller with respect to classic and partitioned IBVS approaches.

Urban-oriented autonomous vehicles require a reliable perception technology to tackle the high amount of uncertainties. The recently introduced compact 3D LIDAR sensor offers a surround spatial information that can be exploited to enhance the vehicle perception. We present a real-time integrated framework of multi-target object detection and tracking using 3D LIDAR geared toward urban use. Our approach combines sensor occlusion-aware detection method with computationally efficient heuristics rule-based filtering and adaptive probabilistic tracking to handle uncertainties arising from sensing limitation of 3D LIDAR and complexity of the target object movement. The evaluation results using real-world pre-recorded 3D LIDAR data and comparison with state-of-the-art works shows that our framework is capable of achieving promising tracking performance in the urban situation.

Aircraft tracking plays a key and important role in the Sense-and-Avoid system of Unmanned Aerial Vehicles (UAVs). This paper presents a novel robust visual tracking algorithm for UAVs in the midair to track an arbitrary aircraft at real-time frame rates, together with a unique evaluation system. This visual algorithm mainly consists of adaptive discriminative visual tracking method, Multiple-Instance (MI) learning approach, Multiple-Classifier (MC) voting mechanism and Multiple-Resolution (MR) representation strategy, that is called Adaptive M3tracker, i.e. AM3. In this tracker, the importance of test sample has been integrated to improve the tracking stability, accuracy and real-time performances. The experimental results show that this algorithm is more robust, efficient and accurate against the existing state-of-art trackers, overcoming the problems generated by the challenging situations such as obvious appearance change, variant surrounding illumination, partial aircraft occlusion, blur motion, rapid pose variation and onboard mechanical vibration, low computation capacity and delayed information communication between UAVs and Ground Station (GS). To our best knowledge, this is the first work to present this tracker for solving online learning and tracking freewill aircraft/intruder in the UAVs.

Inspection of power line infrastructures must be periodically conducted by electric companies in order to ensure reliable electric power distribution. Research efforts are focused on automating the power line inspection process by looking for strategies that satisfy the different requirements of the inspection: simultaneously detect transmission towers, check for defects, and analyze security distances. Following this direction, this paper proposes a supervised learning approach for solving the tower detection and classification problem, where HOG (Histograms of Oriented Gradients) features are used to train two MLP (multi-layer perceptron) neural networks. The first classifier is used for background-foreground segmentation, and the second multi-class MLP is used for classifying within 4 different types of electric towers. A thorough evaluation of the tower detection and classification approach has been carried out on image data from real inspections tasks with different types of towers and backgrounds. In the different evaluations, highly encouraging results were obtained. This shows that a learning-based approach is a promising technique for power line inspection.

For the purposes of achieving Autonomous Air-to-Air Refuelling (AAAR) in Unmanned Aerial Vehicles (UAVs), this paper presents the use of a visual tracking algorithm based on direct methods and image registration techniques, with the aim of solving the drogue tracking problem in order to obtain vision-based relative position estimations of the aircrafts for the probe and drogue refuelling method. Proposed vision-based AAAR approaches to date have explored the use of features (such as corners, painted marks, or LEDs) to detect and estimate the relative motion of either the receiver or the tanker aircraft, with the drawback that sometimes this requires the installation of specific hardware on-board. Conversely, the strategy we propose to use does not require the installation of additional hardware on-board. The strategy is based on a hierarchical implementation of an image registration technique: the Inverse Compositional Image Alignment ICIA. Real images and real flight hardware (probe and drogue) are used to test the algorithm using a robotic testbed that simulates the motion of the aircrafts (the tanker and the receiver) during the refuelling task. Results show that the tracking algorithm is robust to fast motion, changes in appearance, and situations where part of the drogue is occluded or is outside of the field of view of the camera. Additionally, results show that robust position estimations at real-time frame rates are obtained, proving that the technique is fast enough to form the basis for automated aerial refuelling sensing capabilities.

In this work we present an optimized fuzzy visual servoing system for obstacle avoidance using an unmanned aerial vehicle. The cross-entropy theory is used to optimise the gains of our controllers. The optimization process was made using the ROS-Gazebo 3D simulation with purposeful extensions developed for our experiments. Visual servoing is achieved through an image processing front-end that uses the Camshift algorithm to detect and track objects in the scene. Experimental flight trials using a small quadrotor were performed to validate the parameters estimated from simulation. The integration of cross-entropy methods is a straightforward way to estimate optimal gains achieving excellent results when tested in real flights.

This paper presents a Fuzzy Control application for a landing task of an Unmanned Aerial Vehicle, using the 3D-position estimation based on visual tracking of piecewise planar objects. This application allows the UAV to land on scenarios in which it is only possible to use visual information to obtain the position of the vehicle. The use of the homography permits a realtime estimation of the UAV's pose with respect to a helipad using a monocular camera. Fuzzy Logic allows the definition of a model-free control system of the UAV. The Fuzzy controller analyzes the visual information to generate altitude commands for the UAV to develop the landing task.

This article presents a real time Unmanned Aerial Vehicles UAVs 3D pose estimation method using planar object tracking, in order to be used on the control system of a UAV. The method explodes the rich information obtained by a projective transformation of planar objects on a calibrated camera. The algorithm obtains the metric and projective components of a reference object (landmark or helipad) with respect to the UAV camera coordinate system, using a robust real time object tracking based on homographies. The algorithm is validated on real flights that compare the estimated data against that obtained by the inertial measurement unit IMU, showing that the proposed method robustly estimates the helicopter's 3D position with respect to a reference landmark, with a high quality on the position and orientation estimation when the aircraft is flying at low altitudes, a situation in which the GPS information is often inaccurate. The obtained results indicate that the proposed algorithm is suitable for complex control tasks, such as autonomous landing, accurate low altitude positioning and dropping of payloads.

A neural behavior initiating agent (BIA) is proposed to integrate relevant compressed image information coming from others cooperating and specialized neural agents. Using this arrangement the problem of tracking and recognizing a moving icon has been solved by partitioning it into three simpler and separated tasks. Neural modules associated to those tasks proved to be easier to train and show a good general performance. The obtained neural controller can handle spurious images and solve an acute image related task in a dynamical environment. Under prolonged dead-lock conditions the controller shows traces of genuine spontaneity. The overall performance has been tested using a pan and tilt camera platform and real images taken from several objects, showing the good tracking results discussed in the paper.

Dimensionality Reduction is a key issue in many scientific problems, in which data is originally given by high dimensional vectors, all of which lie however over a fewer dimensional manifold. Therefore, they can be represented by a reduced number of values that parametrize their position over the mentioned non-linear manifold. This dimensionality reduction is essential not only for representing and managing data, but also for its understanding at a high interpretation level, similar to the way it is performed by the mammal cortex. This paper presents an algorithm for representing the data that lie on a non-linear manifold by the reduced number of their coordinates along a grid or map of neurons extended over this manifold. This map is generated by a Self-organization learning process whose key feature is the fact that the winning neuron is selected in order to preserve distances of input data when they are represented by their coordinates in the output map. Unlike other methods, the proposed algorithm has important features, that namely the intrinsic dimensionality is obtained simultaneously in the learning process itself, it doesn't require a long course positioning phase, and it seeks to maintain the data structure from the beginning, not leaving it as an ulterior fact to be proven. The algorithm has proven to efficiently solve classical dimensionality reduction problems, and has also showed that it can be useful for realistic problems, such as face images classification or document indexing.

In this paper we introduce a real-time trinocular system to control rotary wing Unmanned Aerial Vehicles based on the 3D information extracted by cameras located on the ground. The algorithm is based on key features onboard the UAV to estimate the vehicle's position and orientation. The algorithm is validated against onboard sensors and known 3D positions, showing that the proposed camera configuration robustly estimates the helicopter's position with an adequate resolution, improving the position estimation, especially the height estimation. The obtained results show that the proposed algorithm is suitable to complement or replace the GPS-based position estimation in situations where GPS information is unavailable or where its information is inaccurate, allowing the vehicle to develop tasks at low heights, such as autonomous landing, take-off, and positioning, using the extracted 3D information as a visual feedback to the flight controller.

This paper presents an implementation of two Fuzzy Logic controllers working in parallel for a pan-tilt camera platform on an UAV. This implementation uses a basic Lucas-Kanade tracker algorithm, which sends information about the error between the center of the object to track and the center of the image, to the Fuzzy controller. This information is enough for the controller to follow the object by moving a two axis servo-platform, regardless the UAV vibrations and movements. The two Fuzzy controllers for each axis, work with a rules-base of 49 rules, two inputs and one output with a more significant sector defined to improve the behavior of those controllers. The controllers have shown very good performances in real flights for statics objects, tested on the Colibri prototypes.

UAVs (unmanned aerial vehicles) are becoming more popular for civil task due to their unique flight capabilities. The video shows several capabilities of a platform used as main research testbed of computer vision for unmanned aerial vehicles. Four major research areas are explored in the present work as visual servoing, trajectory planning, power line inspection and stereo-based visual navigation.

The use of unmanned aerial vehicles (UAVs) in civilian and domestic applications is highly demanding, requiring a high-level of capability from the vehicles. This work addresses the design and implementation of a vision-based feature tracker for an autonomous helicopter. Using vision in the control loop allows estimating the position and velocity of a set of features with respect to the helicopter. The helicopter is then autonomously guided to track these features (in this case windows in an urban environment) in real time. The results obtained from flight trials in a real world scenario demonstrate that the algorithm for tracking features in an urban environment, used for visual servoing of an autonomous helicopter is reliable and robust

The use of UAVs in civilian and domestic applications is highly demanding, requiring a high-level of capability from the vehicles. This work addresses the task in which a UAV is performing an inspection on a set of power lines and an emergency situation occurs requiring the UAV to avoid the lines and then find a safe landing area (a forced landing). This problem is approached using vision, where the vision system acts as the overall controller sending velocity commands to a low-level controller. The use of vision here allows the 2D position of the UAV to be updated by an image-based signal where the error to minimize is the location of a feature or set of features in the image. The system has been tested in a air vehicle simulator (AVS) - a cable array robot which allows to simulate and control three DOF (translation) of a UAV. Results obtained from tests in a scale scenario show the feasibility of this approach

We present the design and implementation of a real-time vision-based approach to detect and track features in a structured environment using an autonomous helicopter. Using vision as a sensor enables the helicopter to track features in an urban environment. We use vision for feature detection and a combination of vision and GPS for navigation and tracking. The vision algorithm sends high level velocity commands to the helicopter controller which is then able to command the helicopter to track them. We present results obtained from flight trials that demonstrate our algorithms for detection and tracking are applicable in real world scenarios by applying them to the task of tracking rectangular features in structured environments.