Enabling the capability of assessing risk and making risk-aware decisions is essential to applying reinforcement learning to safety-critical robots like drones. In this paper, we investigate a specific case where a nano quadcopter robot learns to navigate an apriori-unknown cluttered environment under partial observability. We present a distributional reinforcement learning framework to generate adaptive risk-tendency policies. Specifically, we propose to use lower tail conditional variance of the learnt return distribution as intrinsic uncertainty estimation, and use exponentially weighted average forecasting (EWAF) to adapt the risk-tendency in accordance with the estimated uncertainty. In simulation and real-world empirical results, we show that (1) the most effective risk-tendency varies across states, (2) the agent with adaptive risk-tendency achieves superior performance compared to risk-neutral policy or risk-averse policy baselines. Code and video can be found in this repository: https://github.com/tudelft/risk-sensitive-rl.git

This paper develops an event-triggered optimal control method that can deal with asymmetric input constraints for nonlinear discrete-time systems. The implementation is based on an explainable global dual heuristic programming (XGDHP) technique. Different from traditional GDHP, the required derivatives of cost function in the proposed method are computed by explicit analytical calculations, which makes XGDHP more explainable. Besides, the challenge caused by the input constraints is overcome by the combination of a piece-wise utility function and a bounding layer of the actor network. Furthermore, an event-triggered mechanism is introduced to decrease the amount of computation, and the stability analysis is provided with fewer assumptions compared to most existing studies that investigate event-triggered discrete-time control using adaptive dynamic programming. Two simulation studies are carried out to demonstrate the applicability of the constructed approach. The results present that the developed event-triggered XGDHP algorithm can substantially save the computational load, while maintain comparable performance with the time-based approach.

This paper presents a safety assessment method for unmanned aerial vehicle (UAV) operations, including the effect of a distributed conflict detection and resolution (CD&R) system, in a high density airspace environment. Here, the expected conflicts occurrence and chances for each CD&R system to perform are sufficiently high to extract two parameters of safety, i.e., the frequency of near mid-air collision (NMAC) and the frequency of mid-air collision (MAC), by series of Monte Carlo simulations. The results are then used to derive the safety parameters in a more realistic, less dense airspace. Two cases of distributed CD&R protocols are assessed and compared, i.e., 1) uncoordinated protocol, where each vehicle has its own avoidance preferences, and 2) implicitly coordinated protocol where each vehicle, while still independent from each other, applies simple common rules. Using those CD&R protocols, the result shows a reduction of more than 94% of possible NMAC. Moreover, while maintaining the target level of safety in the airspace, the maximum number of UAVs under an implicitly coordinated CD&R protocol can be at least ten times more than cases when no CD&R is applied.

Self-learning approaches, such as reinforcement learning, offer new possibilities for autonomous control of uncertain or time-varying systems. However, exploring an unknown environment under limited prediction capabilities is a challenge for a learning agent. If the environment is dangerous, free exploration can result in physical damage or in an otherwise unacceptable behavior. With respect to existing methods, the main contribution of this paper is the definition of a new approach that does not require global safety functions, nor specific formulations of the dynamics or of the environment, but relies on interval estimation of the dynamics of the agent during the exploration phase, assuming a limited capability of the agent to perceive the presence of incoming fatal states. Two algorithms are presented with this approach. The first is the Safety Handling Exploration with Risk Perception Algorithm (SHERPA), which provides safety by individuating temporary safety functions, called backups. SHERPA is shown in a simulated, simplified quadrotor task, for which dangerous states are avoided. The second algorithm, denominated OptiSHERPA, can safely handle more dynamically complex systems for which SHERPA is not sufficient through the use of safety metrics. An application of OptiSHERPA is simulated on an aircraft altitude control task.

A control strategy is proposed to deal with the fundamental gain selection problem of optical flow landings. It involves detecting the height by means of an oscillating movement and setting the control gains accordingly at the start of a landing. Then, during descent, the gains are reduced exponentially, with mechanisms in place to ensure high-performance landings. Real-world experiments with a quadrotor demonstrate successful landings in both indoor and outdoor environments.

This paper proposes a taxonomy of conflict detection and resolution (CD&R) approaches for operating unmanned aerial vehicles (UAVs) in an integrated airspace. Possible approaches for UAVs are surveyed and broken down based on their types of surveillance, coordination, maneuvering, and autonomy. The factors are combined back selectively, with regard to their feasibility for operation in an integrated airspace, into several “generic approaches” that form the CD&R taxonomy. These generic approaches are then attributed to a number of available methods in the literature to determine their position in the overall CD&R scheme. The attribution shows that many proposed methods are actually unsuitable for operation in an integrated airspace. Furthermore, some part of the taxonomy does not have an adequate representative in the literature, suggesting the need to concentrate UAV CD&R research more in those particular parts. Finally, a multilayered CD&R architecture is built from the taxonomy, implementing the concept of defense in depth to ensure safe operation of UAVs in an integrated civil airspace.

This paper deals with the active fault-tolerant control for quadrotors which are subjected to a total rotor failure. Previous studies assume that the fault has been detected and isolated and then design a fault-tolerant controller. The present paper proposes a complete active fault-tolerant control system which copes with not only fault detection and isolation but also fault-tolerant control. A novel and efficient fault detection and isolation approach is proposed for the total rotor failure case. An incremental nonlinear dynamic inversion approach is introduced to design the fault-tolerant controller for the quadrotor in the presence of the fault. The complete active fault-tolerant control system enables the quadrotor to achieve any position even after the complete loss of one rotor.