In human–robot teams, human situational awareness is the operator’s conscious knowledge of the team’s states, actions, plans and their environment. Appropriate human situational awareness is critical to successful human–robot collaboration. In human–robot teaming, it is often assumed that the best and required level of situational awareness is knowing everything at all times. This view is problematic, because what a human needs to know for optimal team performance varies given the dynamic environmental conditions, task context, and roles and capabilities of team members. We explore this topic by interviewing 16 participants with active and repeated experience in diverse human–robot teaming applications. Based on analysis of these interviews, we derive a framework explaining the dynamic nature of required situational awareness in human–robot teaming. In addition, we identify a range of factors affecting the dynamic nature of required and actual levels of situational awareness (i.e., dynamic situational awareness), types of situational awareness inefficiencies resulting from gaps between actual and required situational awareness, and their main consequences. We also reveal various strategies, initiated by humans and robots, that assist in maintaining the required situational awareness. Our findings inform the implementation of accurate estimates of dynamic situational awareness and the design of user-adaptive human–robot interfaces. Therefore, this work contributes to the future design of more collaborative and effective human–robot teams.

This tutorial aims to equip researchers with the knowledge and skills to leverage multimodal data in human-robot interaction (HRI) studies. It covers the HRI study cycle, from sensor selection to data analysis, introducing commonly used sensors, pre-processing data, feature extraction techniques, fusion techniques and analysis techniques: both frequentist and Bayesian. Hands-on exercises using public datasets are designed to provide practical experience. The concluding panel discussion on ethics and bias in HRI is focused on fostering broader ethical considerations of HRI studies. Website tutorial is found online at https://sites.google.com/monash.edu/multimodal-hri-study-tutorial.

There is a growing availability of computational and electronic toolkits designed for learning and enrichment activities, however, these toolkits are often inaccessible for blind and low vision (BLV) users. We co-designed with BLV participants, several types of adaption and augmentations that can increase the accessibility of a previously developed electronic toolkit. We explored NFC-enabled 3D-printed circuit templates, braided connectors, and other tactile adaptions developed from co-design sessions with BLV users. We evaluated the resulting toolkit with nine blind and low-vision participants and found that they experimentally and tactually learned to compose circuits of increasing complexity. A key design aspect was incorporating redundant methods that enabled participants to exercise their personal modality preferences when identifying components and making connections. Through our work, we highlight how digital fabrication can be applied to adapt modular electronic toolkits to increase the availability of existing electronics learning platforms for the BLV population.

We present the results of one year spent engaging people living with intellectual disabilities with an electronics and programming package. The program was run in collaboration with a disability support organization and delivered by support workers. We evaluate key qualities of the package at three sites via ongoing communication and reflective interviews with five support workers, along with observation of sessions and contextual inquiry with eleven people with a range of disabilities. Our findings demonstrate the importance of physicality in enabling experiences by creating real-world analogues and supporting diverse group interactions; how groups support members’ attention, motivating each other, and allow space for coping mechanisms; and participants’ growing confidence and creativity in problem solving, and the emergence of self-directed activities. We discuss the importance of diverse repetition for skill development, how skills develop over the year, and pragmatic lessons for conducting a long-term research program with a disability support organization.

A human supervisor’s Situational Awareness (SA) is a critical aspect for successful Human-Robot Teaming (HRT). SA has been estimated using different techniques; however, many of those are associated with various biases, including recall and overgeneralisation biases. A key SA metric is latency, the delay between the time the robotic system requires supervisor assistance and the time the supervisor identifies that need in HRT experiments. Eye movements are increasingly used to assess SA across a range of domains, enabling objective and continuous SA assessment. However, to date, only a small number of features have been evaluated for estimating different types of SA latencies. In this paper, we investigated how two types of SA latencies (perceptual and comprehending) correlate with eye movement data collected during a remote field experiment, where a human supervisor directed a team of robots in a smart farming context. We identified 39 instances of SA latencies (13 perceptual and 26 comprehending). These instances were used to identify how a human supervisor’s SA is affected by task context, and to evaluate correlations between five eye movement features and SA latencies. Two eye movement features related to fixation duration and saccade duration demonstrated very strong correlations ($r \approx - 0.8$ and $r \approx 0.85$). Our findings can be extended to estimate the real-time likelihood of the human experiencing SA latency.

Engagement with electronic toolkits enhances people’s creative abilities, self-esteem, problem-solving skills and enables the creation of personally meaningful artifacts. A variety of simplified electronics toolkits are increasingly available to help different user groups engage with technology. However, they are often inaccessible for people with intellectual disabilities (IDs), who experience a range of cognitive and physical impairments. We designed and developed TronicBoards, a curated set of accessible electronic modules, to address this gap. We evaluated it one-on-one with 10 participants using a guided exploration approach. Our analysis revealed that participants were able to create simple sensor-based interactive circuits with varying levels of assistance. We report the strengths and weaknesses of TronicBoards, considering participants’ successes and challenges in manipulating and comprehending toolkit components, circuit building activities, and troubleshooting processes. We discuss implications for designing inclusive electronics toolkits for people with IDs, particularly in considering design elements that improve functionality, comprehensibility and agency.

Recently, interest has grown in the assessment of anxiety that leverages human physiological and behavioral data to address the drawbacks of current subjective clinical assessments. Complex experiences of anxiety vary on multiple characteristics, including triggers, responses, duration and severity, and impact differently on the risk of anxiety disorders. This article reviews the past decade of studies that objectively analyzed various anxiety characteristics related to five common anxiety disorders in adults utilizing features of cardiac, electrodermal, blood pressure, respiratory, vocal, posture, movement, and eye metrics. Its originality lies in the synthesis and interpretation of consistently discovered heterogeneous predictors of anxiety and multimodal-multisensor analytics based on them. We reveal that few anxiety characteristics have been evaluated using multimodal-multisensor metrics, and many of the identified predictive features are confounded. As such, objective anxiety assessments are not yet complete or precise. That said, few multimodal-multisensor systems evaluated indicate an approximately 11.73% performance gain compared to unimodal systems, highlighting a promising powerful tool. We suggest six high-priority future directions to address the current gaps and limitations in infrastructure, basic knowledge, and application areas. Action in these directions will expedite the discovery of rich, accurate, continuous, and objective assessments and their use in impactful end-user applications.

Vicious cycles of anxiety responses underlie the onset of increasingly prevalent and highly impairing anxiety disorders and also contribute to their maintenance. Our goal is to evaluate whether different anxiety responses are evident in temporal patterns of physiological and behavioral features. Consequently, we established a rich multimodal-multisensor dataset of cardiac, electrodermal, movement, posture, and speech measures from 95 young adults during two anxiety experiments that induce social anxiety and bug-phobic anxiety. A subset of this dataset is publicly available at “Anxiety Phases Dataset” Figshare repository. We adopted a generalized mixed model approach and found that 10 out of 14 feature trajectories modeled for high- and low-anxiety groups differ significantly at 0.001 level in magnitude, creating at least two temporal phases in both groups. Further differences in magnitude, duration and the number of phases were observed for responses of confrontation, safety behaviors, escape, and avoidance in the high-anxiety group. Our findings contribute to the long-term aim of designing multimodal systems that have great potential to reduce the impacts of anxiety disorders and improve therapy.

Anxiety disorders are becoming more prevalent; therefore, the demand for mobile anxiety self-regulation technologies is rising. However, the existing regulation technologies have not yet reached the ability to guide suitable interventions to a user in a timely manner. This is mainly due to the lack of maturity in the anxiety detection area. Hence, this research aims to (1) identify potential temporal phases of anxiety which could become effective personalization parameters for regulation technologies, (2) detect such phases through collecting and analyzing multimodal indicators of anxiety, and (3) design self-regulation technologies that can guide suitable interventions for the detected anxiety phase. Based on an exploratory study that was conducted with therapists treating anxiety disorders, potential temporal phases and common indicators of anxiety were identified. The design of anxiety detection and regulation technologies is currently in progress. The proposed research methodology and expected contributions are further discussed in this paper.