Designing tools to support teams is challenging, but advancements in artificial intelligence (AI) have enabled systems that adapt in real time to evolving team processes. Most existing systems rely on speech data to monitor and assist teams by selecting actions that promote productive collaboration. However, relying solely on speech can limit team support, rewarding only predefined behaviors or failing to assist effective teams when they are silent. Prior work has also shown that unnecessary interventions can harm collaboration. To address these issues, we developed a real-time adaptive system that uses both speech and neurological data to monitor team dynamics. The system employs Q-learning to adapt over time, learn effective configurations, and deliver appropriate responses. We describe the system’s design and implementation, present results from an empirical study demonstrating its advantages over a speech-only system, and discuss design considerations, limitations, and directions for future improvement.

We investigate the utility of functional near-infrared spectroscopy (fNIRS) for workload-based adaptive automation through the lens of multiple resource theory. We focus on the criteria of unobtrusiveness, responsiveness, load sensitivity (low vs high load), and load diagnosticity (differentiating types of load). We report a large meta-review, in which we conclude that only a few studies were suitable for evaluating sensitivity and diagnosticity in complex real-world tasks. While these reveal that the fNIRS signal is adequately sensitive to gradations of load level changes (sensitivity), the diagnosticity of fNIRS to different sources of cognitive load remained uncertain. We manipulated mental load of a complex shape sorting task via working memory load (WM) and visual perceptual load (VL), while a secondary auditory task was present throughout. We measured the effect of these manipulations at the group-level using conventional secondary and eyetracking workload measures, as well as hemodynamic response in specific functional regions in the brain, including regions involved in multi-tasking (MT), VL, WM, and auditory load (AL). Our findings revealed that fNIRS is both sensitive and diagnostic to load in complex tasks, with greater sensitivity revealed by deoxyhemoglobin than oxyhemoglobin and the brain regions associated with diagnosticity align with neuroscience literature on perceptual load, WM, and goal-directed multitasking.

As collaborative technologies evolve from supportive tools to interactive teammates, there is a growing need to understand how trust and team processes develop in human-agent teams. To contribute effectively, these systems must be able to support human teammates in a task without disrupting the delicate interpersonal states and team processes that govern successful collaboration. In order to break down the complexity of monitoring multiple actors in human-agent collaborations, there is a need to identify interpretable, generalizable measures that can monitor the emergence of interpersonal and team-level processes that underlie effective teaming. We address this gap by using functional Near-Infrared Spectroscopy to concurrently measure brain activity of two individuals in a human-human-agent team during a complex, ecologically valid collaborative task, with a goal of identifying quantitative markers of cognition- and affect-based trust alongside team processes of coordination, strategy formulation, and affect management. Two multidimensional extensions of recurrence quantification analysis, a nonlinear method based in dynamical systems theory, are presented to quantify interpersonal coupling and team-level regularity as reflected in the hemodynamics of three cortical regions across multiple time-scales. Mixed-effects regressions reveal that neural recurrence between individuals uniquely reflects changes in self-reported trust, while team-level neural regularity inversely predicts self-reported team processes. Additionally, we show that recurrence metrics capture temporal dynamics of affect-based trust consistent with existing theory, showcasing the interpretability and specificity of these metrics for disentangling complex team states and processes. This paper presents a novel, interpretable, and computationally efficient model-free method capable of differentiating between latent trust and team processes a complex, naturalistic task setting. We discuss the potential applications of this technique for continuous monitoring of team states, providing clear targets for the future development of adaptive human-agent teaming systems.