Due to the probabilistic and uncertain nature of learning-enabled autonomous systems, new assurance technologies appropriate for such systems are becoming critical for their acceptance. Runtime monitors are among the key assurance technologies to address these challenges. In the DARPA Assured Autonomy program, our Boeing team is performing autonomous platform integration and assurance technology evaluation. In this paper, we present our preliminary evaluation results for runtime monitor technologies, which were developed by our three partner teams during Phase I for learning-enabled autonomous systems. The evaluation was completed using a flight simulator and real platforms. In the evaluation, the demonstrated assurance technologies showed significant promise in addressing the assurance shortfall currently facing learning enabled cyber physical systems, and the evaluation approach defined here provides a solid starting point for evaluations of the maturing assurance technologies in the future phases of the DARPA Assured Autonomy project.

Many real-world tasks involve multiple agents with partial observability and limited communication. Learning is challenging in these settings due to local viewpoints of agents, which perceive the world as non-stationary due to concurrently-exploring teammates. Approaches that learn specialized policies for individual tasks face problems when applied to the real world: not only do agents have to learn and store distinct policies for each task, but in practice identities of tasks are often non-observable, making these approaches inapplicable. This paper formalizes and addresses the problem of multi-task multi-agent reinforcement learning under partial observability. We introduce a decentralized single-task learning approach that is robust to concurrent interactions of teammates, and present an approach for distilling single-task policies into a unified policy that performs well across multiple related tasks, without explicit provision of task identity.

This paper presents the first ever approach for solving continuous-observation Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) and their semi-Markovian counterparts, Dec-POSMDPs. This contribution is especially important in robotics, where a vast number of sensors provide continuous observation data. A continuous-observation policy representation is introduced using Stochastic Kernel-based Finite State Automata (SK-FSAs). An SK-FSA search algorithm titled Entropy-based Policy Search using Continuous Kernel Observations (EPSCKO) is introduced and applied to the first ever continuous-observation Dec-POMDP/Dec-POSMDP domain, where it significantly outperforms state-of-the-art discrete approaches. This methodology is equally applicable to Dec-POMDPs and Dec-POSMDPs, though the empirical analysis presented focuses on Dec-POSMDPs due to their higher scalability. To improve convergence, an entropy injection policy search acceleration approach for both continuous and discrete observation cases is also developed and shown to improve convergence rates without degrading policy quality.

Robust environment perception is essential for decision-making on robots operating in complex domains. Intelligent task execution requires principled treatment of uncertainty sources in a robot's observation model. This is important not only for low-level observations (e.g., accelerom-eter data), but also for high-level observations such as semantic object labels. This paper formalizes the concept of macro-observations in Decentralized Partially Observable Semi-Markov Decision Processes (Dec-POSMDPs), allowing scalable semantic-level multi-robot decision making. A hierarchical Bayesian approach is used to model noise statistics of low-level classifier outputs, while simultaneously allowing sharing of domain noise characteristics between classes. Classification accuracy of the proposed macro-observation scheme, called Hierarchical Bayesian Noise Inference (HBNI), is shown to exceed existing methods. The macro-observation scheme is then integrated into a Dec-POSMDP planner, with hardware experiments running onboard a team of dynamic quadrotors in a challenging domain where noise-agnostic filtering fails. To the best of our knowledge, this is the first demonstration of a real-time, convolutional neural net-based classification framework running fully onboard a team of quadrotors in a multi-robot decision-making domain.

This paper introduces a probabilistic algorithm for multi-robot decision-making under uncertainty, which can be posed as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP). Dec-POMDPs are inherently synchronous decision-making frameworks which require significant computational resources to be solved, making them infeasible for many real-world robotics applications. The Decentralized Partially Observable Semi-Markov Decision Process (Dec-POSMDP) was recently introduced as an extension of the Dec-POMDP that uses high-level macro-actions to allow large-scale, asynchronous decision-making. However, existing Dec-POSMDP solution methods have limited scalability or perform poorly as the problem size grows. This paper proposes a cross-entropy based Dec-POSMDP algorithm motivated by the combinatorial optimization literature. The algorithm is applied to a constrained package delivery domain, where it significantly outperforms existing Dec-POSMDP solution methods.

Many robotic missions require online estimation of the unknown state transition models associated with uncertainty that stems from mission dynamics. The learning problem is usually distributed among agents in multiagent scenarios, either due to the absence of a centralized processing unit or because of the large size of the joint learning problem. This paper addresses the problem of multiagent learning in the likely scenario that agents estimate different models from their measured data, but they can share information by communicating model parameters. Previous approaches either consider homogeneous scenarios or perform model transfer in an open-loop manner, which hinders the convergence rate. We develop a closed-loop multiagent learning algorithm, Collaborative Filtering-Decentralized Incremental Feature Dependency Discovery (CF-Dec-iFDD), which enables agents to learn and share models in heterogeneous scenarios. Each agent learns a linear function approximation of the actual model, and the number of features is increased incrementally to adjust model complexity based on the observed data. The agents obtain feedback from other agents on the model error reduction associated with the communicated features. Although this increases the communication cost of exchanging features, it improves the quality/utility of what is being exchanged, leading to improved convergence rate. The approach is demonstrated in indoor hardware flight tests on a forest fire management scenario for which agents must learn the transition model of the fire spread depending on external factors such as wind and vegetation. It is shown that CF-Dec-iFDD has superior convergence rate compared to the alternative approaches.

In persistent missions, taking system's health and capability degradation into account is an essential factor to predict and avoid failures. The state space in health-aware planning problems is often a mixture of continuous vehicle-level and discrete mission-level states. This in particular poses a challenge when the mission domain is partially observable and restricts the use of computationally expensive forward search methods. This paper presents a method that exploits a structure that exists in many health-aware planning problems and performs a two-layer planning scheme. The lower layer exploits the local linearization and Gaussian distribution assumption over vehicle-level states while the higher layer maintains a non-Gaussian distribution over discrete mission-level variables. This two-layer planning scheme allows us to limit the expensive online forward search to the mission-level states, and thus predict system's behavior over longer horizons in the future. We demonstrate the performance of the method on a long duration package delivery mission using a quadrotor in a partially-observable domain in the presence of constraints and health/capability degradation.

The objective of this paper is to present the application of a methodology for designing cooperative control laws that guarantees safe coordination of unmanned aerial vehicles (UAVs). In parallel, an optimal control law and an avoidance control law are designed for each UAV of the multi-vehicle system. These two control laws are combined, guaranteeing that each UAV arrives at their desired state and that their path remains collision free.

This paper and video present the components and flight tests of an indoor, multi-vehicle testbed that was developed to study long duration UAV missions in a controlled environment. This testbed is designed to use real hardware to examine research questions related to single- and multi-vehicle health management, such as vehicle failures, refueling, and maintenance. The testbed has both aerial and ground vehicles that operate autonomously in a large, indoor flight test area and can be used to execute many different mission scenarios. The success of this testbed is largely related to our choice of vehicles, sensors, and the system's command and control architecture. The video presents flight test results from single- and multi-vehicle experiments over the past year.