Crowdsourcing and weak supervision offer methods to efficiently label large datasets. Our work builds on existing weak supervision models to accommodate ordinal target classes, in an effort to recover ground truth from weak, external labels. We define a parameterized factor function and show that our approach improves over other baselines.

Autonomous systems show promise as teammates for human operators in safety- and performance-critical environments. Humans' cognitive states (like trust, mental workload, and situation awareness) change as they work in demanding environments with autonomous systems. Providing adaptive autonomous systems with information about human teammates' cognitive states is a current gap in human-autonomy teaming research. In this paper we present results from a human-autonomy teaming experiment in which participants completed a spaceflight-relevant task in a supervisory, “on-the-loop” role. To complete the task, participants worked with an autonomous system that had five distinct modes of autonomy. Our experiment results show that unobtrusive measures (based on actions participants take and eye tracking data) could inform statistical models that accurately predicted three different subjectively reported cognitive states at the same time (RMSE from 11% to 14% of questionnaires' ranges). Our work builds upon previous research and demonstrates that our model-building approach can extend to different human-autonomy teaming scenarios. This also represents the first time that three cognitive states were predicted using unobtrusive measures from the same supervisory human-autonomy teaming task. Our research enables future experiments investigating an autonomous system which adapts according to cognitive state predictions.