Situation awareness (SA)—comprising the ability to 1) perceive critical elements in the environment, 2) comprehend their meanings, and 3) project their future states—is critical for human operator performance. Due to the disruptive nature of gold-standard SA measures, researchers have sought physiological indicators to provide real-time information about SA. We extend prior work by using a multimodal suite of neurophysiological, psychophysiological, and behavioral signals, predicting all three levels of SA along a continuum, and predicting a comprehensive measure of SA in a complex multitasking simulation. We present a lab study in which 31 participants controlled an aircraft simulator task battery while wearing physiological sensors and responding to SA “freeze-probe” assessments. We demonstrate the validity of task and assessment for measuring SA. Multimodal physiological models predict SA with greater predictive performance (Q2for levels 1–3 and total, respectively: 0.14, 0.00, 0.26, and 0.36) than models built with shuffled labels, demonstrating that multimodal physiological signals provide useful, albeit imperfect, information in predicting all SA levels. Level 3 SA (projection) was best predicted, and level 2 SA (comprehension) was the most challenging to predict. Ablation analysis and single-sensor models found EEG and eye-tracking signals to be particularly useful to predictions of level 3 and total SA. A reduced sensor fusion model showed that predictive performance can be maintained with a subset of sensors. This rigorous cross-validation assessment of predictive performance demonstrates the utility of multimodal physiological signals for inferring complex, holistic, objective measures of SA at all levels, nondisruptively, and along a continuum.

Galvanic vestibular stimulation (GVS) is an emergent tool for stimulating the vestibular system, offering the potential to manipulate or enhance processes relying on vestibular-mediated central pathways. However, the extent of GVS's influence on the perception of self-orientation pathways is not understood, particularly in the presence of physical motions. Here, we quantify roll tilt perception impacted by GVS during passive whole-body roll tilts in humans (N = 11). We find that GVS systematically amplifies and attenuates perceptions of roll tilt during physical tilt, dependent on the GVS waveform. Subsequently, we develop a novel computational model that predicts 6DoF self-motion and self-orientation perceptions for any GVS waveform and motion by modeling the vestibular afferent neuron dynamics modulated by GVS in conjunction with an observer central processing model. This effort provides a means to systematically alter spatial orientation perceptions using GVS during concurrent physical motion, and we find that irregular afferent dynamics alone best describe resultant perceptions.

Crowdsourcing and weak supervision offer methods to efficiently label large datasets. Our work builds on existing weak supervision models to accommodate ordinal target classes, in an effort to recover ground truth from weak, external labels. We define a parameterized factor function and show that our approach improves over other baselines.

Autonomous systems show promise as teammates for human operators in safety- and performance-critical environments. Humans' cognitive states (like trust, mental workload, and situation awareness) change as they work in demanding environments with autonomous systems. Providing adaptive autonomous systems with information about human teammates' cognitive states is a current gap in human-autonomy teaming research. In this paper we present results from a human-autonomy teaming experiment in which participants completed a spaceflight-relevant task in a supervisory, “on-the-loop” role. To complete the task, participants worked with an autonomous system that had five distinct modes of autonomy. Our experiment results show that unobtrusive measures (based on actions participants take and eye tracking data) could inform statistical models that accurately predicted three different subjectively reported cognitive states at the same time (RMSE from 11% to 14% of questionnaires' ranges). Our work builds upon previous research and demonstrates that our model-building approach can extend to different human-autonomy teaming scenarios. This also represents the first time that three cognitive states were predicted using unobtrusive measures from the same supervisory human-autonomy teaming task. Our research enables future experiments investigating an autonomous system which adapts according to cognitive state predictions.