Current multimodal instruction-recognition algorithms in human-robot interaction, developed largely from a purely technical perspective, remain rigid and incomplete in their use of human communicative cues. Therefore, a full understanding of how humans naturally refer to targets in interaction is central to enabling robots to interpret and act on user instructions. To investigate this, we collected multimodal behavior data from 30 participants who naturally instructed a robot for household tasks while we systematically varied target distance, direction, and local referent complexity. Our results show that speech instructions were often vague and lacked explicit target-position information. To resolve this ambiguity, multimodal cues are essential: gaze direction provides an order-of-magnitude improvement in target-localization accuracy, while hand pointing, head turns, and speech onset offer reliable temporal anchors for identifying target-directed gaze. We also found that speech patterns varied with distance and local referent complexity, whereas multimodal behaviors shifted with target direction, underscoring the need for context-adaptive recognition and interface design.

Integrating domestic robots into everyday life requires not only reliable execution but also prior alignment of task goals between humans and robots. While prior research has examined input interfaces and feedback strategies, it has largely focused on objective performance metrics and often overlooked user variability. To address this gap, we conducted a survey study with 113 participants across four groups: adolescents, younger adults, older adults, and wheelchair users. The survey captured participants’ expectations of future robots (roles, embodiments, and concerns) and their preferences for instruction delivery and robot feedback before execution. Our results reveal both shared and group-specific patterns. Across groups, participants prioritized efficiency in instruction delivery and reliability in robot feedback. Regarding group differences: adolescents emphasized efficiency, wheelchair users valued transparency, and older adults may benefit from additional explanations of novel interaction technologies. Based on these findings, we derive stage-aware, context-sensitive, and group-adaptive design principles and recommendations to guide future robot interfaces.

In conditionally automated driving, drivers out of the control loop may lack situational awareness (SA), leading to inappropriate takeovers. Monitoring a driver’s SA and providing alerts for overlooked objects is critical to enhancing the takeover safety and efficiency. This study aimed to construct predictive models for drivers’ SA of objects during takeover transitions. The model features include drivers’ physiological data before and after takeover requests as well as the environment and object attributes. The ground truth was obtained through a scene reconstruction task, yielding binary SA labels. The Support Vector Machine delivered the best model performance, achieving a macro F1 score of 0.75 and an accuracy of 0.77, when applied with a time window of 2-second pre-takeover request and 4-second post-takeover request. Our model predicts drivers’ SA of specific objects across diverse traffic conditions using short time windows, supporting timely and generalizable driver monitoring and takeover assistance.

While voice-controlled automated Wheelchair-mounted robotic arms (WMRAs) could potentially offer more natural and simplified interactions than manual control, they introduce new challenges related to human-robot collaboration. To explore user expectations in terms of functionality and interface, we conducted semi-structured interviews with 13 powered wheelchair users who have upper limb impairments. A prototype with a robotic arm, a camera, and a Unity-based simulated interface was developed to help users understand the concept and functionality of the voice-controlled automated WMRA system during interviews. With safety as a priority, we found that users prioritized the WMRA's ability to grasp objects in challenging positions, such as on the ground or at high locations, and emphasized the need for a versatile gripper. Users also expected the WMRA to assist in performing complex daily tasks, with varying expectations for its performance based on task difficulty. Regarding the interface, users sought more information about the system's awareness and task execution, while emphasizing the importance of avoiding information overload. The demand for detailed safety information, such as temperature and gripping force, pointed to the need for enhanced sensor capabilities in the WMRA system. Additionally, concerns about privacy underscored the need for clear communication on privacy policies. Our results provide user-centered insights for automated WMRA, offering design implications and future research directions in areas such as user modeling, hardware, algorithms, and interface development.

Trust and situational awareness (SA) are critical for the acceptance and safety of automated vehicles (AVs). While AV explanations with different information types have been studied to enhance drivers' trust and SA, their effectiveness remains unclear when AVs make errors that do not trigger takeover requests. This study investigated the effects of information type, error type, and their interaction on drivers' trust in AVs, SA, and their relationships. We recruited 300 participants in an online video study with a 3 (information type:why,how,why + how) × 3 (error type: false alarm, miss, correct [no error]) mixed design.Howinformation describes the vehicle's action, whilewhyinformation refers to the reason for the vehicle's action. Linear mixed models showed that false alarms and misses were associated with lower SA compared with correct scenarios, but possibly due to different reasons. Compared with correct scenarios, both false alarms and misses were associated with lower trust, with misses even lower than false alarms, possibly due to the varying severity of potential consequences. Compared withwhyandwhy + howinformation,howinformation was generally associated with lower SA and a higher potential of overtrust in false alarms. Trust and SA had a negative linear relationship in misses and false alarms, while no correlations were found in correct scenarios. To mitigate potential overtrust and misinterpretation of situations when AVs make errors, it is crucial to maintain higher SA. We recommend includingwhyinformation in AV explanations and deploying AV decision systems that are less miss-prone.

To promote drivers’ overall experiences in highly automated vehicles, we designed three objective criticality-adaptive displays: IO display highlighting Influential Objects, CO display highlighting Critical Objects, and ICO display highlighting Influential and Critical Objects differently. We conducted an online video-based survey study with 295 participants to evaluate them in varying traffic conditions. Results showed that low-trust propensity participants found ICO display more useful while high-trust propensity participants found CO displays more useful. When interacting with vulnerable road users (VRUs), participants had higher situational awareness (SA) but worse non-driving related task (NDRT) performance. Aging and CO displays also led to slower NDRT reactions. Nonetheless, older participants found displays more useful. We recommend providing different criticality-adaptive displays based on drivers’ trust propensity, age, and NDRT choice to enhance driving and NDRT performance and suggest carefully treating objects of different categories in traffic.