After-action reviews (AARs) are professional discussions that help operators and teams enhance their task performance by analyzing completed missions with peers and professionals. Previous studies comparing different formats of AARs have focused mainly on human teams. However, the inclusion of robotic teammates brings along new challenges in understanding teammate intent and communication. Traditional AAR between human teammates may not be satisfactory for human-robot teams. To address this limitation, we propose a new training review (TR) tool, called the Virtual Spectator Interface (VSI), to enhance human-robot team performance and situational awareness (SA) in a simulated search mission. The proposed VSI primarily utilizes visual feedback to review subjects' behavior. To examine the effectiveness of VSI, we took elements from AAR to conduct our own TR, and designed a 1$\times 3$between-subjects experiment with experimental conditions: TR with (1) VSI, (2) screen recording, and (3) non-technology (only verbal descriptions). The results of our experiments demonstrated that the VSI did not result in significantly better team performance than other conditions. However, the TR with VSI led to more improvement in the subjects' SA over the other conditions.

This study presents an Actor-Critic Cooperative Compensated Model Predictive Controller$(\text{AC}^3 \text{MPC})$designed to address unknown system dynamics. To avoid the difficulty of modeling highly complex dynamics and ensuring real-time control feasibility and performance, this work uses deep reinforcement learning with a model predictive controller in a cooperative framework to handle unknown dynamics. The model-based controller takes on the primary role as both controllers are provided with predictive information about the other. This improves tracking performance and retention of inherent robustness of the model predictive controller. We evaluate this framework for off-road autonomous driving on unknown deformable terrains that represent sandy deformable soil, sandy and rocky soil, and cohesive clay-like deformable soil. Our findings demonstrate that our controller statistically outperforms standalone model-based and learning-based controllers by upto 29.2% and 10.2%. This framework generalized well over varied and previously unseen terrain characteristics to track longitudinal reference speeds with lower errors. Furthermore, this required significantly less training data compared to purely learning-based controller, while delivering better performance even when under-trained.

Skid-steered wheel mobile robots (SSWMRs) operate in a variety of outdoor environments exhibiting motion behaviors dominated by the effects of complex wheel-ground interactions. Characterizing these interactions is crucial from both the immediate robot autonomy perspective (for motion prediction and control) and a long-term predictive maintenance and diagnostics perspective. An ideal solution entails capturing precise state measurements for decisions and controls, which is considerably difficult, especially in increasingly unstructured outdoor regimes of operations for these robots. In this milieu, a framework to identify pre-determined discrete modes of operation can considerably simplify the motion model identification process. To this end, we propose an interactive multiple model (IMM) based filtering framework to probabilistically identify predefined robot operation modes that could arise due to traversal in different terrains or loss of wheel traction.

Autonomous vehicles (AVs) can operate in complex environments that require multiple types of sensors in their perception approaches. However, a single sensing configuration is not tenable in all environments, and adapting the perception approach to different domains is a challenging task. Both sensor fusion and ensemble learning methods utilize the diversity of multimodal sensor data (e.g., cameras, radar, lidar) to increase perception performance under challenging sensing conditions. In this paper, we conduct the first analysis examining how sensor diversity can impact performance across different AV perception tasks. We propose ensembles of multimodal models that leverage diversity and assess their ability to address the challenge of adaptation in AV perception. We introduce a novel, model-agnostic framework,DivFusE, that identifies the level of diversity that individual sensor modalities contribute to perception ensembles and employs adaptation based on this diversity. We benchmark our approach on five public datasets containing semantic segmentation and object detection tasks, highlighting scenarios where our proposed framework can improve perception performance by 70% (fuse it) and scenarios where fusion can reduce perception performance (lose it). Our findings indicate that the performance improvements of using multimodal ensembles is greater on objection detection compared to semantic segmentation, and that sensing conditions with higher (lower) levels of diversity further increase (decrease) performance. Ultimately, the analysis in this work highlights scenarios where these multimodal ensembles should and should not be deployed and can be used to develop more adaptive multimodal sensor fusion systems.

This paper presents a data-driven framework to discover underlying dynamics on a scaled F1TENTH vehicle using the Koopman operator linear predictor. Traditionally, a range of white, gray, or black-box models are used to develop controllers for vehicle path tracking. However, these models are constrained to either linearized operational domains, unable to handle significant variability or lose explainability through end-2-end operational settings. The Koopman Extended Dynamic Mode Decomposition (EDMD) linear predictor seeks to utilize data-driven model learning whilst providing benefits like explainability, model analysis and the ability to utilize linear model-based control techniques. Consider a trajectory-tracking problem for our scaled vehicle platform. We collect pose measurements of our F1TENTH car undergoing standard vehicle dynamics benchmark maneuvers with an OptiTrack indoor localization system. Utilizing these uniformly spaced temporal snapshots of the states and control inputs, a data-driven Koopman EDMD model is identified. This model serves as a linear predictor for state propagation, upon which an MPC feedback law is designed to enable trajectory tracking. The prediction and control capabilities of our framework are highlighted through real-time deployment on our scaled vehicle.

In operations of multi-agent teams ranging from homogeneous robot swarms to heterogeneous human-autonomy teams, unexpected events might occur. While efficiency of operation for multi-agent task allocation problems is the primary objective, it is essential that the decision-making framework is intelligent enough to manage unexpected task load with limited resources. Otherwise, operation effectiveness would drastically plummet with overloaded agents facing unforeseen risks. In this work, we present a decision-making framework for multiagent teams to learn task allocation with the consideration of load management through decentralized reinforcement learning, where idling is encouraged and unnecessary resource usage is avoided. We illustrate the effect of load management on team performance and explore agent behaviors in example scenarios. Furthermore, a measure of agent importance in collaboration is developed to infer team resilience when facing handling potential overload situations.

Safe operations of autonomous mobile robots in close proximity to humans, creates a need for enhanced trajectory tracking (with low tracking errors). Linear optimal control techniques such as Linear Quadratic Regulator (LQR) and Model Predictive Control (MPC) have been used successfully for low-speed applications while leveraging their model-based methodology with manageable computational demands. However, model and parameter uncertainties or other unmodeled nonlinearities may cause poor control actions and constraint violations. Nonlinear MPC has emerged as an alternate optimal-control approach but needs to overcome real-time deployment challenges (including fast sampling time, design complexity, and limited computational resources). In recent years, the optimal control-based deployments have benefitted enormously from the ability of Deep Neural Networks (DNNs) to serve as universal function approximators. This has led to deployments in a plethora of previously inaccessible applications – but many aspects of generalizability, benchmarking, and systematic verification and validation coupled with benchmarking have emerged. This paper presents a novel approach to fusing Deep Reinforcement Learning-based (DRL) longitudinal control with a traditional PID lateral controller for autonomous navigation. Our approach follows (i) Generation of an adequate fidelity simulation scenario via a Real2Sim approach; (ii) training a DRL agent within this framework; (iii) Testing the performance and generalizability on alternate scenarios. We use an initial tuned set of the lateral PID controller gains for observing the vehicle response over a range of velocities. Then we use a DRL framework to generate policies for an optimal longitudinal controller that successfully complements the lateral PID to give the best tracking performance for the vehicle.

We introduce and study the problem of planning a trajectory for an agent to carry out a reconnaissance mission while avoiding being detected by an adversarial guard. This introduces a multi-objective version of classical visibility-based target search and pursuit-evasion problem. In our formulation, the agent receives a positive reward for increasing its visibility (by exploring new regions) and a negative penalty every time it is detected by the guard. The objective is to find a finite-horizon path for the agent that balances the trade off between maximizing visibility and minimizing detectability.We model this problem as a discrete, sequential, two-player, zero-sum game. We use two types of game tree search algorithms to solve this problem: minimax search tree and Monte-Carlo search tree. Both search trees can yield the optimal policy but may require possibly exponential computational time and space. We propose several pruning techniques to reduce the computational cost while still preserving optimality guarantees. Simulation results show that the proposed strategy prunes approximately three orders of magnitude nodes as compared to the brute-force strategy. We also find that the Monte-Carlo search tree saves approximately one order of computational time as compared to the minimax search tree.

Correlation filters (CFs) are a well-known pattern classification approach used in biometrics. A CF is a spatial-frequency array that is specifically synthesized from a set of training patterns to produce a sharp correlation output peak at the location of the best match for an authentic image comparison and no such peak for an impostor image comparison. The underlying premise when using CFs is that this correlation output peak behavior on training data ideally extends to testing data. Yet in 1:1 verification scenarios, where there is limited training data available to represent pattern distortions, the correlation output from an authentic comparison can be difficult to discern from the correlation output from an impostor. In this paper we introduce Stacked Correlation Filters (SCFs), a simple and powerful approach to address this problem by training an additional set of classifiers which learn to differentiate correlation outputs from authentic and impostor match pairs. This is done by training a series of stacked modular CFs with each layer refining the output of the previous layer. Our basic premise is that since correlation outputs have an expected shape, an additional CF can be trained to recognize such shape and refine the final output. As previous works with CFs have only focused on individual filter design or application, which assumes the CF to provide a sharp peak, this is a new CF paradigm that can benefit many existing CF designs and applications.

The periocular region as a biometric trait has recently gained considerable traction, especially under challenging scenarios where reliable iris information is not available for human authentication. In this paper, we consider the problem of one-to-one (1 : 1) matching of highly nonideal periocular images captured in-the-wild under unconstrained imaging conditions. Such images exhibit considerable appearance variations, including nonuniform illumination variations, motion and defocus blur, off-axis gaze, and nonstationary pattern deformations. To address these challenges, we propose periocular probabilistic deformation models (PPDMs) that: 1) reduce the image matching problem to matching local image regions and 2) approximate the periocular distortions by local patch level spatial translations whose relationships are modeled by a Gaussian Markov random field. Given a periocular image pair, we determine the distortion-tolerant similarity metric by regularizing local match scores by the maximum aposteriori probability estimate of the relative local deformations between them. Unlike the existing global periocular image matching techniques, by accounting for local image deformations in the periocular matching process, PPDM exhibits greater tolerance to pattern variations. We demonstrate the effectiveness of our model via extensive evaluation on a large number of in-the-wild periocular images. We find that PPDMs outperform many benchmark 1 : 1 image matching techniques (improving verification rates at 0.1% false accept rate by ~30% over previous work and ~40% when compared with the best baseline) in challenging scenarios leading to state-of-the-art verification performance on multiple real-world periocular data sets.

Iris recognition is believed to offer excellent recognition rates for iris images acquired under controlled conditions. However, recognition rates degrade considerably when images exhibit impairments such as off-axis gaze, partial occlusions, specular reflections and out-of-focus and motion-induced blur. In this paper, we use the recently-available face and ocular challenge set (FOCS) to investigate the comparative recognition performance gains of using ocular images (i.e., iris regions as well as the surrounding peri-ocular regions) instead of just the iris regions. A new method for ocular recognition is presented and it is shown that use of ocular regions leads to better recognition rates than iris recognition on FOCS dataset. Another advantage of using ocular images for recognition is that it avoids the need for segmenting the iris images from their surrounding regions.