Human-Autonomy Teaming (HAT) is a multi-disciplinary domain with a diverse set of research needs and goals stemming from fields such as computer science, robotics, and human factors. This melting pot of fields generates a unique challenge in that there exist many disjoint research methods (measures and tasks) that cause issues with knowledge transfer and comparison between researchers. One way to address this issue is by providing researchers with a testbed containing a standardized suite of analysis tools and tasks that allow direct comparison between different approaches. Therefore, this study attempts to bring the HAT community together in a collaborative discussion to collect and organize their research needs for the future development of these testbeds. Specifically, through thematic analysis, our work reveals three emergent prongs that underpin testbed needs of HAT experts: task, AI, and technical requirements. Also, we organize our thematic analysis by priority to suggest possible paths for HAT testbed development to maximize its immediate and continued utility. Our research indicates that the HAT community places significant importance on both the pre-established, standardized functions available within the testbed and the freedom to tailor and develop their unique tasks or AI solutions.

In this extended abstract we present the design, development, and evaluation of a Minecraft-based simulated task environment to conduct human and AI teaming research. With the deluge of AI-driven applications and their infiltration into many activities of daily living, it is becoming necessary to look at ways that humans and AI can work together. There is a tremendous research burden associated with accurately evaluating the best practices and trade-offs when humans and AI have to collaborate together in completing critical tasks. Minecraft offers a low-cost alternative as an early investigating tool for researchers to build answers to emerging research questions before significantly investing in human-AI teaming activities in the real world. We demonstrate successfully via a simple rule-based AI, insights that could highly influence human-AI teaming activities can be derived to improve practical and viable development of protocols and procedures. Our findings indicate that simulated task environments play a critical role in furthering human AI teaming activities.

Predicting separation errors in the daily tasks of air traffic controllers (ATCOs) is essential for the timely implementation of mitigation strategies before performance declines and the prevention of loss of separation and aircraft collisions. However, three challenges impede accurate separation errors forecasting: 1) compounding relationships between many human factors and control processes require sufficient operation process data to capture how separation errors occur and propagate within controller-in-the-loop processes; 2) previous human factor measurement approaches are disruptive to controllers’ daily operations because they use invasive sensors, such as electroencephalography (EEG) and electrocardiography (ECG), 3) errors accumulated in using the tasks and human behaviors for estimating system dynamics challenge accurate separation error predictions with sufficient leading time for proactive control actions. This study proposed a separation error prediction framework with a long leading time (>50 s) to address the above challenges, including 1) a multi-factorial model that characterizes the inter-relationships between task complexity, behavioral activity, cognitive load, and operational performance; 2) a multimodal data analytics approach to non-intrusively extract the task features (i.e., traffic density) from high-fidelity simulation systems and visual behavioral features (i.e., head pose, eyelid movements, and facial expressions) from ATCOs’ facial videos; 3) an encoder-decoder Long Short-Term Memory (LSTM) network to predict long-time-ahead separation errors by integrating multimodal features for reducing accumulated errors. A user study with six experienced ATCOs tested the proposed framework using the Phoenix Terminal Radar Approach Control (TRACON) simulator. The authors evaluated the model performance through two types of metrics: 1) point-level metrics, including precision, recall, and F1-score, and 2) sequence-level metrics, including alignment accuracy and sequence similarity. The results showed that 1) the model using the task and visual behavioral features significantly improved the prediction performance compared to the model using one single feature (eyelid movements), with an improvement of up to 26.95% in alignment accuracy for 10s-ahead prediction; 2) the model that combined task and visual behavioral features had a higher or comparable performance to models with different hybrid features, achieving an alignment accuracy of 82.38% for 50s-ahead error prediction; and (3) the proposed method outperformed three baseline models – Convolutional Neural Network (CNN), Gated Recurrent Unit (GRU), and classic LSTM – by 8.21%, 3.47%, and 3.14% in alignment accuracy, respectively, for predicting 50s-ahead separation errors. These results suggest that the proposed model can effectively predict separation errors in air traffic control.

Teams composed of human and machine members operating in complex task environments must effectively interact in response to information flow while adapting to environmental changes. This study investigates how interpersonal coordination dynamics between team members are associated with team performance and shared situation awareness in a simulated urban search and rescue (USAR) task. More specifically, this study investigates (1) how communication recurrence affected and reflected coordination dynamics between a USAR robot and human operator when they used different communication strategies, and (2) how these dynamic characteristics of the human–robot interpersonal coordination were associated with the team performance and shared situation awareness. The USAR interpersonal coordination dynamics were systematically characterized using discrete recurrence quantification analysis. Results from this study indicate that (1) teams demonstrating more flexibility in their coordination dynamics were more adaptive to changes in the task environment, and (2) while robot explanations help to improve shared situation awareness, revisiting the same communication pattern (i.e., routine coordination) was associated with better team performance, but did not improve shared situation awareness.

This article considers human-autonomy teams (HATs) in which two human team members interact and collaborate with an autonomous teammate to achieve a common task while dealing with unexpected technological failures that were imposed either in automation or autonomy. A Wizard of Oz methodology is used to simulate the autonomous teammate. One of the critical aspects of HAT performance is the trust that develops over time as team members interact with each other in a dynamic task environment. For this reason, it is important to examine the dynamic nature of teammate trust through real-time measures of team interactions. This article examines team interaction and trust to understand better how they change under automation and autonomy failures. Thus, we address two research questions: 1) How does trust in HATs evolve over time?; and 2) How is the relationship between team interaction and trust impacted by the failures? We hypothesize that trust in HATs will decrease as autonomy failures increase. We also hypothesize that team interaction would be related to the development of trust and recovery from the failures. The results implicate three general trends: 1) team interaction dynamics are linked to the development of trust in HATs; 2) trust in the autonomous teammate is only associated with recovery from autonomy failures; 3) team interaction dynamics are related to both automation and autonomy failure recovery.

In the past, team coordination dynamics have been explored using nonlinear dynamical systems (NDS) methods, but the relationship between team coordination dynamics and team performance for all-human teams was assumed to be linear. The current study examines team coordination dynamics with an extended version of the NDS methods and assumes that its relationship with team performance for human-autonomy teams (HAT) is nonlinear. In this study, three team conditions are compared with the goals of better understanding how team coordination dynamics differ between all-human teams and HAT and how these dynamics relate to team performance and team situation awareness. Each condition was determined based on manipulation of the .pilot role: in the first condition (synthetic) the pilot role was played by a synthetic agent, in the second condition (control) it was a randomly assigned participant, and in the third condition (experimenter) it was an expert who used a role specific coordination script. NDS indices revealed that synthetic teams were rigid, followed by experimenter teams, who were metastable, and control teams, who were unstable. Experimenter teams demonstrated better team effectiveness (i.e., better team performance and team situation awareness) than control and synthetic teams. Team coordination stability is related to team performance and team situation awareness in anonlinear manner with optimal performance and situation awareness associated with metastability coupled with flexibility. This result means that future development of synthetic teams should address these coordination dynamics, specifically, rigidity in coordination.

Sequential data collected for usability testing, knowledge engineering, or cognitive task analysis are rich with information-so rich that interpretation can often be overwhelming. This dilemma can be viewed as a data reduction problem. PRO-NET (PROcedural NETworks), a method for reducing sequential data in terms of procedural networks, is introduced and then applied and evaluated in two case studies-one involving human-computer interaction (HCI) in a simulated mission control operation at the National Aeronautics and Space Administration and the other involving avionics troubleshooting behavior for an intelligent tutor application. The method involves five steps-collecting data, encoding data, generating transition matrices, conducting Pathfinder analysis, and interpreting procedural networks. The method employs the Pathfinder network scaling algorithm, which is particularly suited for asymmetric data. Evidence is presented to support the descriptive and predictive utility of this form of data reduction. In addition, lessons learned in applying PRONET to the two cases are discussed, applications of PRONET to HCI are described, and guidelines are offered for using PRONET in exploratory sequential data analysis.