Designing tools to support teams is challenging, but advancements in artificial intelligence (AI) have enabled systems that adapt in real time to evolving team processes. Most existing systems rely on speech data to monitor and assist teams by selecting actions that promote productive collaboration. However, relying solely on speech can limit team support, rewarding only predefined behaviors or failing to assist effective teams when they are silent. Prior work has also shown that unnecessary interventions can harm collaboration. To address these issues, we developed a real-time adaptive system that uses both speech and neurological data to monitor team dynamics. The system employs Q-learning to adapt over time, learn effective configurations, and deliver appropriate responses. We describe the system’s design and implementation, present results from an empirical study demonstrating its advantages over a speech-only system, and discuss design considerations, limitations, and directions for future improvement.

Gaze-assisted interaction techniques enable intuitive selections without requiring manual pointing but can result in unintended selections, known as Midas touch. A confirmation trigger eliminates this issue but requires additional physical and conscious user effort. Brain-computer interfaces (BCIs), particularly passive BCIs harnessing anticipatory potentials such as the Stimulus-Preceding Negativity (SPN) - evoked when users anticipate a forthcoming stimulus - present an effortless implicit solution for selection confirmation. Within a VR context, our research uniquely demonstrates that SPN has the potential to decode intent towards the visually focused target. We reinforce the scientific understanding of its mechanism by addressing a confounding factor - we demonstrate that the SPN is driven by the user’s intent to select the target, not by the stimulus feedback itself. Furthermore, we examine the effect of familiarly placed targets, finding that SPN may be evoked quicker as users acclimatize to target locations; a key insight for everyday BCIs.

Eye movements have long been used to reflect ongoing cognitive processing to develop explanatory and predictive models of mental states and processes. This relationship, deemed the eye-mind link, contains underlying assumptions of the mental processes occurring in the brain, which have rarely been explicitly investigated. We propose a multimodal approach to investigate alignment of eye movements and brain activations (eye-brain alignment) and how it might be predicted by unfolding cognitive processes. We applied this method to a dataset of 76 participants who read long, connected texts while their eye movements and the hemodynamic responses in their brains were tracked using functional near-infrared spectroscopy (fNIRS). We found that reliable eye-brain alignment signals varied based on the participants’ cognitive state during reading. Implications for multimodal modeling of cognitive processes are discussed.

We investigate the utility of functional near-infrared spectroscopy (fNIRS) for workload-based adaptive automation through the lens of multiple resource theory. We focus on the criteria of unobtrusiveness, responsiveness, load sensitivity (low vs high load), and load diagnosticity (differentiating types of load). We report a large meta-review, in which we conclude that only a few studies were suitable for evaluating sensitivity and diagnosticity in complex real-world tasks. While these reveal that the fNIRS signal is adequately sensitive to gradations of load level changes (sensitivity), the diagnosticity of fNIRS to different sources of cognitive load remained uncertain. We manipulated mental load of a complex shape sorting task via working memory load (WM) and visual perceptual load (VL), while a secondary auditory task was present throughout. We measured the effect of these manipulations at the group-level using conventional secondary and eyetracking workload measures, as well as hemodynamic response in specific functional regions in the brain, including regions involved in multi-tasking (MT), VL, WM, and auditory load (AL). Our findings revealed that fNIRS is both sensitive and diagnostic to load in complex tasks, with greater sensitivity revealed by deoxyhemoglobin than oxyhemoglobin and the brain regions associated with diagnosticity align with neuroscience literature on perceptual load, WM, and goal-directed multitasking.

As collaborative technologies evolve from supportive tools to interactive teammates, there is a growing need to understand how trust and team processes develop in human-agent teams. To contribute effectively, these systems must be able to support human teammates in a task without disrupting the delicate interpersonal states and team processes that govern successful collaboration. In order to break down the complexity of monitoring multiple actors in human-agent collaborations, there is a need to identify interpretable, generalizable measures that can monitor the emergence of interpersonal and team-level processes that underlie effective teaming. We address this gap by using functional Near-Infrared Spectroscopy to concurrently measure brain activity of two individuals in a human-human-agent team during a complex, ecologically valid collaborative task, with a goal of identifying quantitative markers of cognition- and affect-based trust alongside team processes of coordination, strategy formulation, and affect management. Two multidimensional extensions of recurrence quantification analysis, a nonlinear method based in dynamical systems theory, are presented to quantify interpersonal coupling and team-level regularity as reflected in the hemodynamics of three cortical regions across multiple time-scales. Mixed-effects regressions reveal that neural recurrence between individuals uniquely reflects changes in self-reported trust, while team-level neural regularity inversely predicts self-reported team processes. Additionally, we show that recurrence metrics capture temporal dynamics of affect-based trust consistent with existing theory, showcasing the interpretability and specificity of these metrics for disentangling complex team states and processes. This paper presents a novel, interpretable, and computationally efficient model-free method capable of differentiating between latent trust and team processes a complex, naturalistic task setting. We discuss the potential applications of this technique for continuous monitoring of team states, providing clear targets for the future development of adaptive human-agent teaming systems.

Predicting workload using physiological sensors has taken on a diffuse set of methods in recent years. However, the majority of these methods train models on small datasets, with small numbers of channel locations on the brain, limiting a model's ability to transfer across participants, tasks, or experimental sessions. In this paper, we introduce a new method of modeling a large, cross-participant and cross-session set of high density functional near infrared spectroscopy (fNIRS) data by using an approach grounded in cognitive load theory and employing a Bi-Directional Gated Recurrent Unit (BiGRU) incorporating attention mechanism and self-supervised label augmentation (SLA). We show that our proposed CNN-BiGRU-SLA model can learn and classify different levels of working memory load (WML) and visual processing load (VPL) across participants. Importantly, we leverage a multi-label classification scheme, where our models are trained to predict simultaneously occurring levels of WML and VPL. We evaluate our model using leave-one-participant-out (LOOCV) as well as 10-fold cross validation. Using LOOCV, for binary classification (off/on), we reached an F1-score of 0.9179 for WML and 0.8907 for VPL across 22 participants (each participant did 2 sessions). For multi-level (off, low, high) classification, we reached an F1-score of 0.7972 for WML and 0.7968 for VPL. Using 10-fold cross validation, for multi-level classification, we reached an F1-score of 0.7742 for WML and 0.7741 for VPL.

In this paper, we study the neural underpinnings relevant to user-centered web security through the lens of functional near-infrared spectroscopy (fNIRS). Specifically, we design and conduct an fNIRS study to pursue a thorough investigation of users' processing of legitimate vs. illegitimate and familiar vs. unfamiliar websites. We pinpoint the neural activity in these tasks as well as the brain areas that control such activity. We show that, at the neurological level, users process the legitimate websites differently from the illegitimate websites when subject to phishing attacks. Similarly, we show that users exhibit marked differences in the way their brains process the previously familiar websites from unfamiliar websites. These findings have several defensive and offensive implications. In particular, we discuss how these differences may be used by the system designers in the future to differentiate between legitimate and illegitimate websites automatically based on neural signals. Similarly, we discuss the potential for future malicious attackers, with access to neural signals, in compromising the privacy of users by detecting whether a website is previously familiar or unfamiliar to the user.

Detecting phishing attacks (identifying fake vs. real websites) and heeding security warnings represent classical user-centered security tasks subjected to a series of prior investigations. However, our understanding of user behavior underlying these tasks is still not fully mature, motivating further work concentrating at the neuro-physiological level governing the human processing of such tasks.

This project represents a first step towards bridging the gap between HCI and cognition research. Using functional near-infrared spectroscopy (fNIRS), we introduce tech-niques to non-invasively measure a range of cognitive workload states that have implications to HCI research, most directly usability testing. We present a set of usability experiments that illustrates how fNIRS brain measurement provides information about the cognitive demands placed on computer users by different interface designs.

A well designed user interface (UI) should be transparent, allowing users to focus their mental workload on the task at hand. We hypothesize that the overall mental workload required to perform a task using a computer system is composed of a portion attributable to the difficulty of the underlying task plus a portion attributable to the complexity of operating the user interface. In this regard, we follow Shneiderman's theory of syntactic and semantic components of a UI. We present an experiment protocol that can be used to measure the workload experienced by users in their various cognitive resources while working with a computer. We then describe an experiment where we used the protocol to quantify the syntactic workload of two user interfaces. We use functional near infrared spectroscopy, a new brain imaging technology that is beginning to be used in HCI. We also discuss extensions of our techniques to adaptive interfaces.

Because functional near-infrared spectroscopy (fNIRS) eases many of the restrictions of other brain sensors, it has potential to open up new possibilities for HCI research. From our experience using fNIRS technology for HCI, we identify several considerations and provide guidelines for using fNIRS in realistic HCI laboratory settings. We empirically examine whether typical human behavior (e.g. head and facial movement) or computer interaction (e.g. keyboard and mouse usage) interfere with brain measurement using fNIRS. Based on the results of our study, we establish which physical behaviors inherent in computer usage interfere with accurate fNIRS sensing of cognitive state information, which can be corrected in data analysis, and which are acceptable. With these findings, we hope to facilitate further adoption of fNIRS brain sensing technology in HCI research.

We are in the midst of an explosion of emerging human-computer interaction techniques that redefine our understanding of both computers and interaction. We propose the notion of Reality-Based Interaction (RBI) as a unifying concept that ties together a large subset of these emerging interaction styles. Based on this concept of RBI, we provide a framework that can be used to understand, compare, and relate current paths of recent HCI research as well as to analyze specific interaction designs. We believe that viewing interaction through the lens of RBI provides insights for design and uncovers gaps or opportunities for future research.

This work analyzes the advantages and disadvantages of using the novice programming environment Alice in the CS0 classroom. We consider both general aspects as well as specifics drawn from the authors' experiences using Alice in the classroom over the course of the last academic year.

We created Smart Blocks, an augmented mathematical manipulative that allows users to explore the concepts of volume and surface area of 3-dimensional (3D) objects. This interface supports physical manipulation for exploring spatial relationships and it provides continuous feedback for reinforcing learning. By leveraging the benefits of physicality with the advantages of digital information, this tangible interface provides an engaging environment for learning about surface area and volume of 3D objects.