Humans are often modeled as rational actors by interactive agents when they are in fact frequently observed to make biased decisions. This erroneous assumption may cause an agent’s model of the human to fail, especially when interaction occurs in bias-inducing settings that prompt risky decisions. To address this, this paper formulates a risk-sensitive multi-agent coordination problem and presents the novel Risk-Sensitive Theory of Mind (RS-ToM) framework that allows an autonomous agent to reason about and adapt to a partner of unknown risk-sensitivity. In simulated studies, we show that an agent with an RS-ToM is able to better coordinate with such a partner when compared to an agent that assumes their partner is rational. Thus, we observe significant improvements to team performance, coordination fluency, compliance with partner risk-preferences, and predictability. The presented results suggest that an RS-ToM will be able to model and plan with partners that exhibit these risk-sensitive biases in the real world.

To achieve effective coordination in human–robot teams, robots must have an accurate model of human decision-making (i.e., theory of mind) to predict human actions and plan appropriate supplemental strategies. However, humans are often assumed to be approximately rational decision-makers which is not always the case, especially when faced with risky or uncertain decisions. Recent works in human–robot interaction have begun to address this by implementing risk-sensitive models of human behavior to characterize an individual’s risk-sensitivity. However, little attention is given to the following question: what happens when the robot makes the incorrect inference about human risk-sensitivity? Failure to consider this may lead to ineffective coordination and degradation of perceived trustworthiness in the robot. In this article, we adopt a popular risk-sensitive model based on Cumulative Prospect Theory, where model accuracy is varied in the robot’s theory of mind when interacting with either a risk-averse (pessimistic) or risk-seeking (optimistic) human. We designed a joint-pursuit game where the human and robot are conditioned with different (assumptions of) human risk-sensitivity in a 2 \( \times \) 2, between-subject study. Results from both simulated and human experiments showed that team performance was decreased and perceived trustworthiness in the robot was negatively impacted when the robot made the incorrect assumption of human risk-sensitivity. Overall, this work shows that risk-sensitive models can be used to great effect but only if we remain diligent to the fact that misspecification can lead to negative outcomes.

Human-Autonomy Teaming (HAT) is a multi-disciplinary domain with a diverse set of research needs and goals stemming from fields such as computer science, robotics, and human factors. This melting pot of fields generates a unique challenge in that there exist many disjoint research methods (measures and tasks) that cause issues with knowledge transfer and comparison between researchers. One way to address this issue is by providing researchers with a testbed containing a standardized suite of analysis tools and tasks that allow direct comparison between different approaches. Therefore, this study attempts to bring the HAT community together in a collaborative discussion to collect and organize their research needs for the future development of these testbeds. Specifically, through thematic analysis, our work reveals three emergent prongs that underpin testbed needs of HAT experts: task, AI, and technical requirements. Also, we organize our thematic analysis by priority to suggest possible paths for HAT testbed development to maximize its immediate and continued utility. Our research indicates that the HAT community places significant importance on both the pre-established, standardized functions available within the testbed and the freedom to tailor and develop their unique tasks or AI solutions.