Ding Xia, Xinyue Gui, Mark Colley, Fan Gao, Zhongyi Zhou, Dongyuan Li, Renhe Jiang, Takeo Igarashi. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Pedestrian–automated vehicle (AV) encounters in shared spaces often involve hesitation and ambiguity. Vehicle-mounted external human–machine interfaces (eHMIs) can help, but obscured or poorly timed communications create significant challenges. To address this, we present a mobile smart pole interaction unit (SPIU) with integrated cameras and LED displays, designed as a pedestrian-side system to deliver explicit cues (“WALK,” “STOP”). An in-the-wild evaluation of the SPIU (N = 21) using a four-factor analysis (CarBehavior, Mobility, eHMI, SPIU) showed that the SPIU improved understandability, trust, and perceived safety, and reduced workload compared with the baseline, with a combination (eHMI+SPIU) yielding the strongest results. Beyond these quantitative benefits, participants appreciated the mobility of the SPIU for its “clear” and “easy to decide” mediation. This work contributes to (1) a design and deployment framework for a mobile SPIU and (2) an in-the-wild evaluation protocol for pedestrian–AV interactions in nonsignalized spaces. Our work sparks discussions on real world evaluations involving detailed vehicle kinematics and accessible multimodality (e.g., audio), focusing on the role of personal robots as user-side eHMIs.

With automated vehicles (AVs), the absence of a human operator could necessitate external Human-Machine Interfaces (eHMIs) to communicate with other road users. Existing research primarily focuses on pedestrian-AV interactions, with limited attention given to other road users, such as cyclists and drivers of manually driven vehicles. So far, no studies have compared the effects of eHMIs across these three road user roles. Therefore, we conducted a within-subjects virtual reality experiment (N=40), evaluating the subjective and objective impact of an eHMI communicating the AV’s intention to pedestrians, cyclists, and drivers under various levels of distraction (no distraction, visual noise, interference). eHMIs positively influenced safety perceptions, trust, perceived usefulness, and mental demand across all roles. While distraction and road user roles showed significant main effects, interaction effects were only observed in perceived usability. Thus, a unified eHMI design is effective, facilitating the standardization and broader adoption of eHMIs in diverse traffic.

Field studies are irreplaceable but costly, time-consuming, and error-prone, which need careful preparation. Inspired by rapid-prototyping in manufacturing, we propose a fast, low-cost evaluation method using Vision-Language Model (VLM) personas to simulate outcomes comparable to field results. While LLMs show human-like reasoning and language capabilities, autonomous vehicle (AV)-pedestrian interaction requires spatial awareness, emotional empathy, and behavioral generation. This raises our research question: To what extent can VLM personas mimic human responses in field studies? We conducted parallel studies: 1) one real-world study with 20 participants, and 2) one video-study using 20 VLM personas, both on a street-crossing task. We compared their responses and interviewed five HCI researchers on potential applications. Results show that VLM personas mimic human response patterns (e.g., average crossing times of 5.25 s vs. 5.07 s) lack the behavioral variability and depth. They show promise for formative studies, field study preparation, and human data augmentation.

Traffic is inherently dangerous, with around 1.19 million fatalities annually. Automotive Mediated Reality (AMR) can enhance driving safety by overlaying critical information (e.g., outlines, icons, text) on key objects to improve awareness, altering objects’ appearance to simplify traffic situations, and diminishing their appearance to minimize distractions. However, real-world AMR evaluation remains limited due to technical challenges. To fill this sim-to-real gap, we present MIRAGE, an open-source tool that enables real-time AMR in real vehicles. MIRAGE implements 15 effects across the AMR spectrum of augmented, diminished, and modified reality using state-of-the-art computational models for object detection and segmentation, depth estimation, and inpainting. In an on-road expert user study (N=9) of MIRAGE, participants enjoyed the AMR experience while pointing out technical limitations and identifying use cases for AMR. We discuss these results in relation to prior work and outline implications for AMR ethics and interaction design.

People’s transportation choices reflect complex trade-offs shaped by personal preferences, social norms, and technology acceptance. Predicting such behavior at scale is a critical challenge with major implications for urban planning and sustainable transport. Traditional methods use handcrafted assumptions and costly data collection, making them impractical for early-stage evaluations of new technologies or policies. We introduce Generative Traffic Agents (GTA) for simulating large-scale, context-sensitive transportation choices using LLM-powered, persona-based agents. GTA generates artificial populations from census-based sociodemographic data. It simulates activity schedules and mode choices, enabling scalable, human-like simulations without handcrafted rules. We evaluate GTA in Berlin-scale experiments, comparing simulation results against empirical data. While agents replicate patterns, such as modal split by socioeconomic status, they show systematic biases in trip length and mode preference. GTA offers new opportunities for modeling how future innovations, from bike lanes to transit apps, shape mobility decisions.

Despite widespread use, charts remain largely inaccessible for Low-Vision Individuals (LVI). Reading charts requires viewing data points within a global context, which is difficult for LVI who may rely on magnification or experience a partial field of vision. We aim to improve exploration by providing visual access to critical context. To inform this, we conducted a formative study with five LVI. We identified four fundamental contextual elements common across chart types: axes, legend, grid lines, and the overview. We propose two pointer-based interaction methods to provide this context: Dynamic Context, a novel focus+context interaction, and Mini-map, which adapts overview+detail principles for LVI. In a study with N=22 LVI, we compared both methods and evaluated their integration to current tools. Our results show that Dynamic Context had significant positive impact on access, usability, and effort reduction; however, worsened visual load. Mini-map strengthened spatial understanding, but was less preferred for this task. We offer design insights to guide the development of future systems that support LVI with visual context while balancing visual load.

The way users hold a smartphone depends on the interaction task, yet little is known about the fingers’ engagement with the device’s surfaces beyond the touchscreen. Such an understanding not only opens up opportunities for novel on- and off-screen interactions, but also the device’s possible physical affordances. We present a study (N=23) that examines the hands’ physical engagement with the smartphone beyond the touchscreen across nine mobile interactions. Grasps were annotated from photographs, and contact regions were captured using residual heat traces from grasping the device. Our findings show that fingers and palms adopt a variety of support roles and postures when engaging with the smartphone’s back and side edges. The hand-contact maps reveal distinct patterns, differing in contact frequency and placement. This work contributes an empirical characterisation of hands’ back and edge engagement, highlighting design opportunities for future smartphone usage extending beyond the touchscreen.

The next step for In-vehicle Conversational Assistants (IVCAs) will be their capability to initiate and automate proactive system interactions throughout journeys. However, diverse drivers make it challenging to design voice interventions tailored towards individual on-road expectations. This paper evaluates the effectiveness of Human-in-the-Loop (HITL) Multi-Objective Bayesian Optimization (MOBO) in design by implementing ProVoice: a Virtual Reality (VR) driving simulator integrating MOBO to investigate the effects of IVCA design variants on perceived mental demand, predictability, and usefulness. By reporting the Pareto Front from a within-subjects VR study (N=19), this paper proposes optimal design trade-offs. Follow-up analysis demonstrates MOBO’s success in discovering effective intervention strategies, with reduced participant mental demand, alongside enhanced predictability and usefulness while engaging with the proactive IVCA. Implications for computational techniques in future research on proactive intervention strategies are discussed. ProVoice can extend to include alternative design parameters and driving scenarios, encouraging intervention design on a broad scale.

External Human-Machine Interfaces (eHMIs) have been proposed to enhance communication between automated vehicles (AVs) and pedestrians, with growing interest in multi-modal designs such as audio-visual eHMIs. Just as poor lighting can impair visual cues, a loud background noise may mask the auditory stimuli. However, its effects within these systems have not been examined, and little is known about how pedestrians — particularly Deaf and Hard-of-Hearing (DHH) people — perceive different types of auditory stimuli. We conducted a virtual reality study (Hearing N=25, DHH N=11) to examine the effects of background noise (quiet and loud) on auditory stimuli (baseline, bell, speech) within an audio-visual eHMI. Results revealed that: (1) Crossing experiences of DHH pedestrians significantly differ from Hearing pedestrians. (2) Loud background noise adversely affects pedestrians’ crossing experiences. (3) Providing an additional auditory eHMI (bell/speech) improves crossing experiences. We outlined four practical implications for future eHMI design and research.

External Human-Machine Interfaces (eHMIs) have been proposed to facilitate communication between Automated Vehicles (AVs) and pedestrians. However, no attention was given to Deaf and Hard-of-Hearing (DHH) people. We conducted a formative study through focus groups with 6 DHH people and 6 key stakeholders (including researchers, assistive technologists, and automotive interface designers) to compare proposed eHMIs and extract key design requirements. Subsequently, we investigated the effects of visual and auditory eHMI in a virtual reality user study with 32 participants (16 DHH). Results from our scenario suggesting that (1) DHH participants spent more time looking at the AV; (2) both visual and auditory eHMIs enhanced trust, usefulness, and perceived safety; and (3) only visual eHMIs reduced the time to step into the road, time looking at the AV, gaze time, and percentage looking at active visual eHMI components. Lastly, we provided five practical implications for making eHMI inclusive of DHH people.

Video content creation offers vital opportunities for expression and participation, yet remains largely inaccessible to creators with sensory impairments, especially in low-resource settings.We conducted interviews with 20 video creators with visual and hearing impairments in Kenya to examine their tools, challenges, and collaborative practices.Our findings show that accessibility barriers and infrastructural limitations shape video creation as a staged, collaborative process involving trusted human partners and emerging AI tools.Across workflows, creators actively negotiated agency and trust, maintaining creative control while bridging sensory gaps.We discuss the need for flexible, interdependent collaboration models, inclusive human-AI workflows, and diverse storytelling practices.This work broadens accessibility research in HCI by examining how technology and social factors intersect in low-resource contexts, suggesting ways to better support disabled creators globally.

Automated vehicles (AVs) reached technological maturity and will soon arrive on streets as traffic participants.Human traffic participants such as drivers, pedestrians, or cyclists will be increasingly confronted with the presence of AVs within their environment, not necessarily knowing or understanding what to expect and how to interact with them.Although AVs are designed to act safely, effective interaction in mixed traffic scenarios will depend on successful communication, interaction, or even negotiation beyond static rules and regulations.Prosocial behavior, such as yielding one's right of way, will be needed to resolve unclear traffic situations or foster traffic flow.However, what are the characteristics of such prosocial behavior, and how to measure this not only for automated vehicles

Micromobility vehicles, such as e-scooters, provide ecological and financial advantages over automotive transportation. However, as with car drivers, micromobility users often perform non-driving related tasks (NDRTs), interacting with stereo controls or navigation tasks, which can lead to accidents. It remains unclear what control interfaces are appropriate and safe for micromobility. We evaluated six interface modalities for NDRTs and conducted a within-subjects study with 35 participants (yielding N=210 observations) in an e-scooter simulator to compare modality safety and preferences. Our results align with existing work on gaze and tactility in the automotive NDRTs context. However, unique to e-scooters, interfaces that required users to alter their grip on the handlebars were less preferred as they compromised stability. Social comfort also emerged as a critical factor due to concerns about public visibility. This work aims to encourage the design of safer, more socially acceptable interfaces for e-scooters and other emerging micromobility vehicles.

The absence of a human operator in automated vehicles (AVs) may require external Human-Machine Interfaces (eHMIs) to facilitate communication with other road users in uncertain scenarios, for example, regarding the right of way. Given the plethora of adjustable parameters, balancing visual and auditory elements is crucial for effective communication with other road users. With N=37 participants, this study employed multi-objective Bayesian optimization to enhance eHMI designs and improve trust, safety perception, and mental demand. By reporting the Pareto front, we identify optimal design trade-offs. This research contributes to the ongoing standardization efforts of eHMIs, supporting broader adoption.

As vehicle automation technology continues to mature, there is a necessity for robust remote monitoring and intervention features. These are essential for intervening during vehicle malfunctions, challenging road conditions, or in areas that are difficult to navigate. This evolution in the role of the human operator - from a constant driver to an intermittent teleoperator - necessitates the development of suitable interaction interfaces. While some interfaces were suggested, a comparative study is missing. We designed, implemented, and evaluated three interaction concepts (path planning, trajectory guidance, and waypoint guidance) with up to four concurrent requests of automated vehicles in a within-subjects study with N=23 participants. The results showed a clear preference for the path planning concept. It also led to the highest usability but lower satisfaction. With trajectory guidance, the fewest requests were resolved. The study's findings contribute to the ongoing development of HMIs focused on the remote assistance of automated vehicles.

Automated vehicle (AV) acceptance relies on their understanding via feedback. While visualizations aim to enhance user understanding of AV's detection, prediction, and planning functionalities, establishing an optimal design is challenging. Traditional "one-size-fits-all" designs might be unsuitable, stemming from resource-intensive empirical evaluations. This paper introduces OptiCarVis, a set of Human-in-the-Loop (HITL) approaches using Multi-Objective Bayesian Optimization (MOBO) to optimize AV feedback visualizations. We compare conditions using eight expert and user-customized designs for a Warm-Start HITL MOBO. An online study (N=117) demonstrates OptiCarVis's efficacy in significantly improving trust, acceptance, perceived safety, and predictability without increasing cognitive load. OptiCarVis facilitates a comprehensive design space exploration, enhancing in-vehicle interfaces for optimal passenger experiences and broader applicability.

Infinite scrolling on social media platforms is designed to encourage prolonged engagement, leading users to spend more time than desired, which can provoke negative emotions. Interventions to mitigate infinite scrolling have shown initial success, yet users become desensitized due to the lack of contextual relevance. Understanding how contextual factors influence intervention effectiveness remains underexplored. We conducted a 7-day user study (N=72) investigating how these contextual factors affect users' reactance and responsiveness to interventions during infinite scrolling. Our study revealed an interplay, with contextual factors such as being at home, sleepiness, and valence playing significant roles in the intervention's effectiveness. Low valence coupled with being at home slows down the responsiveness to interventions, and sleepiness lowers reactance towards interventions, increasing user acceptance of the intervention. Overall, our work contributes to a deeper understanding of user responses toward interventions and paves the way for developing more effective interventions during infinite scrolling.

Automated Urban Air Mobility (UAM) can improve passenger transportation and reduce congestion, but its success depends on passenger trust. While initial research addresses passengers' information needs, questions remain about how to simulate air taxi flights and how these simulations impact users and interface requirements. We conducted a between-subjects study (N=40), examining the influence of motion fidelity in Virtual-Reality-simulated air taxi flights on user effects and interface design. Our study compared simulations with and without motion cues using a 3-Degrees-of-Freedom motion chair. Optimizing the interface design across six objectives, such as trust and mental demand, we used multi-objective Bayesian optimization to determine the most effective design trade-offs. Our results indicate that motion fidelity decreases users' trust, understanding, and acceptance, highlighting the need to consider motion fidelity in future UAM studies to approach realism. However, minimal evidence was found for differences or equality in the optimized interface designs, suggesting personalized interface designs.

The introduction of Highly Automated Vehicles (HAVs) has the potential to increase the independence of blind and visually impaired people (BVIPs). However, ensuring safety and situation awareness when exiting these vehicles in unfamiliar environments remains challenging. To address this, we conducted an interactive workshop with N=5 BVIPs to identify their information needs when exiting an HAV and evaluated three prior-developed low-fidelity prototypes. The insights from this workshop guided the development of PathFinder, a multimodal interface combining visual, auditory, and tactile modalities tailored to BVIP's unique needs. In a three-factorial within-between-subject study with N=16 BVIPs, we evaluated PathFinder against an auditory-only baseline in urban and rural scenarios. PathFinder significantly reduced mental demand and maintained high perceived safety in both scenarios, while the auditory baseline led to lower perceived safety in the urban scenario compared to the rural one. Qualitative feedback further supported PathFinder's effectiveness in providing spatial orientation during exiting.

As the use of Head-Mounted Displays in moving vehicles increases, passengers can immerse themselves in visual experiences independent of their physical environment. However, interaction methods are susceptible to physical motion, leading to input errors and reduced task performance. This work investigates the impact of G-forces, vibrations, and unpredictable maneuvers on 3D interaction methods. We conducted a field study with 24 participants in both stationary and moving vehicles to examine the effects of vehicle motion on four interaction methods: (1) Gaze&Pinch, (2) DirectTouch, (3) Handray, and (4) HeadGaze. Participants performed selections in a Fitts' Law task. Our findings reveal a significant effect of vehicle motion on interaction accuracy and duration across the tested combinations of Interaction Method x Road Type x Curve Type. We found a significant impact of movement on throughput, error rate, and perceived workload. Finally, we propose future research considerations and recommendations on interaction methods during vehicle movement.

Automated vehicles (AVs) are gradually becoming part of our daily lives. However, effective communication between road users and AVs remains a significant challenge. Although various external human-machine interfaces (eHMIs) have been developed to facilitate interactions, psychological factors, such as a lack of trust and inadequate emotional signaling, may still deter users from confidently engaging with AVs in certain contexts. To address this gap, we propose TailCue, an exploration of how tail-based eHMIs affect user interaction with AVs. We first investigated mappings between tail movements and emotional expressions from robotics and zoology, and accordingly developed a motion-emotion mapping scheme. A physical robotic tail was implemented, and specific tail motions were designed based on our scheme. An online, video-based user study with 21 participants was conducted. Our findings suggest that, although the intended emotions conveyed by the tail were not consistently recognized, open-ended feedback indicated that the tail motion needs to align with the scenarios and cues. Our result highlights the necessity of scenario-specific optimization to enhance tail-based eHMIs. Future work will refine tail movement strategies to maximize their effectiveness across diverse interaction contexts.

Urban Air Mobility (UAM) emerges as a potential solution to urban congestion. However, as it lacks integration with existing transportation systems, methods to study its impact are necessary. Traditional empirical approaches are insufficient to study large-scale effects in this not-yet-real context. We developed UAM-SUMO, an extension of the SUMO simulation platform, to simulate the impact of UAM on public transportation, particularly how air taxis affect traffic flow and mode choices. We detail the modifications to SUMO and the UAM operation parameters. We open-source our code at https://github.com/M-Colley/uam-sumo and present a proof-of-concept data collection and analysis for Ingolstadt, Germany.

Large-scale effects of head-up displays (HUDs) are currently unknown, as experiments focus primarily on empirical data from one participant. Therefore, we simulate the impact of augmented reality (AR) windshield HUDs on driving performance. Using the open-source simulators SUMO and CARLA, we model various AR HUD settings, such as fatigue, and assess their effects on key driving factors such as reaction time, speed adherence, lane-changing behavior, and acceleration. The HUD settings include brightness, information frequency, field of view, and relevance of displayed information. The simulation provides insight into how different HUD configurations may influence driving behavior, contributing to future vehicle design and safety guidelines for AR HUDs.

As living and working spaces become scarce and costly, interiors transitioning between living, working, and sleeping configurations while enabling customized setups are in demand. Traditional furniture consumes space and is cumbersome to rearrange. Shape-changing furniture could solve this, yet existing options lack resolution, stability, or adaptability. We present AirClick, which facilitates on-demand room transformation using modular interactive inflatables. Our fabrication process supports personally fabricated and retrofitted retail inflatables. The touch-actuated modules connect to a floor-based air connector grid, facilitating interaction while integrating with traditional furniture. In a lab study (N=20) across four scenarios (office, meeting room, apartment room, multipurpose hall), participants rapidly transformed rooms and perceived AirClick as significantly more usable with higher intention to use in the everyday scenarios than in the hall, indicating suitability for routine activities with low to medium requirements for robustness. User feedback highlights AirClick ’s usefulness and scalability in diverse settings, hence showing AirClick’s space-saving and customizable design can enhance the functionality and adaptability of living and working spaces.

Mixed Reality (MR) integrates virtual objects with the real world, offering potential but raising concerns about misuse through dark patterns. This study explored the effects of four dark patterns, adapted from prior research, and applied to MR across three targets: places, products, and people. In a two-factorial within-subject study with 74 participants, we analyzed 13 videos simulating MR experiences during a city walk. Results show that all dark patterns significantly reduced user comfort, increased reactance, and decreased the intention to use MR glasses, with the most disruptive effects linked to personal or monetary manipulation. Additionally, the dark patterns of Emotional and Sensory Manipulation and Hiding Information produced similar impacts on the user in MR, suggesting a re-evaluation of current classifications to go beyond deceptive design techniques. Our findings highlight the importance of developing ethical design guidelines and tools to detect and prevent dark patterns as immersive technologies continue to evolve.

We review the state of open science and the perspectives on open data sharing within the automotive user research community. Openness and transparency are critical not only for judging the quality of empirical research, but also for accelerating scientific progress and promoting an inclusive scientific community. However, there is little documentation of these aspects within the automotive user research community. To address this, we report two studies that identify (1) community perspectives on motivators and barriers to data sharing, and (2) how openness and transparency have changed in papers published at AutomotiveUI over the past 5 years. We show that while open science is valued by the community and openness and transparency have improved, overall compliance is low. The most common barriers are legal constraints and confidentiality concerns. Although research published at AutomotiveUI relies more on quantitative methods than research published at CHI, openness and transparency are not as well established. Based on our findings, we provide suggestions for improving openness and transparency, arguing that the motivators for open science must outweigh the barriers. All supporting materials are freely available at: https://osf.io/zdpek/

In the emerging automated vehicle (AV)—passenger interaction domain, there is no agreed-upon set of methods to design early concepts. Non-designers may find it challenging to brainstorm interfaces for unfamiliar technology like AVs. Therefore, we explore using an immersive virtual environment to enable expert and non-expert designers to actively participate in the design phases. We built AV-Sketch, an in-situ (on-site) simulator that allows the creation of automotive interfaces while being immersed in VR depicting diverse AV-passenger interactions. At first, we conducted a participatory design study (N=15) by utilizing PICTIVE (Plastic Interface for Collaborative Technology) to conceptualize human-machine interfaces for AV passengers. The findings led to the design of AV-Sketch, which we tested in a design session (N=10), assessing users’ design experiences. Overall, participants felt more engaged and confident with the in-situ experience, enabling better contextualization of design ideas in real-world scenarios, with improved spatial considerations and dynamic aspects of in-vehicle interfaces.

For people who are blind and low vision (BLV), ridesharing provides an important means of independence and mobility. However, a common challenge relates to finding the vehicle when it arrives to an unanticipated location. Although coordinating with the driver for assistance is serviceable in the near term, new solutions are necessary when a human is no longer available in future automated vehicles. Therefore, this paper presents and evaluates a multisensory smartphone-based map system designed to enable nonvisual tracking of summoned vehicles. Results from a user study with (N=12) BLV users suggest that vibro-audio maps (VAMs) promote superior spatial confidence and reasoning compared to current nonvisual audio interfaces in ridesharing apps, while also being desirable and easy to use. A subsequent expert evaluation based on improvements suggested during the user study indicate the practical utility of VAMs to address both current and future wayfinding challenges for BLV travelers.

As vehicle automation advances to SAE Levels 3 to 5, transitioning driving control from human to system, ensuring automated vehicles (AVs) align with user preferences becomes a challenge. Natural interaction emerges as a common goal, offering ways to convey user interests in a user-friendly manner. However, technical, legal, or design constraints may prevent fulfilling these preferences, leading to potential conflicts. Through an online survey (N=50), potential driver-passenger conflicts and their handling strategies were explored. Subsequently, in a Virtual Reality study (N=14), we applied identified strategies (ranging from distracting to motivating and adhering to social norms) to user-AV interactions using a state-of-the-art language model (GPT-4 Turbo) primed with the strategies to simulate realistic dialogues. Additionally, adaptive communication was compared to non-adaptive communication. Our findings reveal a preference for adaptive communication. Yet, despite using advanced modeling, accurately predicting user interactions remained challenging, with users often trying to outsmart the AI.

In automotive user interface design, testing often starts with lab-based driving simulators and migrates toward on-road studies to mitigate risks. Mixed reality (XR) helps translate virtual study designs to the real road to increase ecological validity. However, researchers rarely run the same study in both in-lab and on-road simulators due to the challenges of replicating studies in both physical and virtual worlds. To provide a common infrastructure to port in-lab study designs on-road, we built a platform-portable infrastructure, Portobello, to enable us to run twinned physical-virtual studies. As a proof-of-concept, we extended the on-road simulator XR-OOM with Portobello. We ran a within-subjects, autonomous-vehicle crosswalk cooperation study (N=32) both in-lab and on-road to investigate study design portability and platform-driven influences on study outcomes. To our knowledge, this is the first system that enables the twinning of studies originally designed for in-lab simulators to be carried out in an on-road platform.

Automated vehicles are expected to improve traffic safety and efficiency. One approach to achieve this is via platooning, that is, (automated) vehicles can drive behind each other at very close proximity to reduce air resistance. However, this behavior could lead to difficulties in mixed traffic, for example, when manual drivers try to enter a highway. Therefore, we report the results of a within-subject Virtual Reality study (N=29) evaluating different platoon behaviors (single vs. multiple, i.e., four, gaps) and communication strategies (HUD, AR, attached displays). Results show that AR communication reduced mental workload, improved perceived safety, and a single big gap led to the safest merging behavior. Our work helps to incorporate novel behavior enabled by automation into general traffic better.

Gaze interaction is a promising interaction method to increase variety, challenge, and fun in games.We present “Shed Some Fear”, a 2D platform game including numerous eye-gaze-based interactions. “Shed Some Fear” includes control with eye-gaze and traditional keyboard input. The eye-gaze interactions are partially based on eye exercises reducing digital eye strain but also on employing peripheral vision. By employing eye-gaze as a necessary input mechanism, we explore the effects on and tradeoffs between user enjoyment and digital eye strain in a five-day longitudinal between-subject study (N=17) compared to interaction with a traditional mouse. We found that perceived competence was significantly higher with eye gaze interaction and significantly higher internal eye strain. With this work, we contribute to the not straightforward inclusion of eye tracking as a useful and fun input method for games.

External Human-Machine Interfaces (eHMIs) have been evaluated to facilitate interactions between Automated Vehicles (AVs) and pedestrians. Most eHMIs are, however, visual/ light-based solutions, and multi-modal eHMIs have received little attention to date. We ran an experimental video study (N = 29) to systematically understand the effect on pedestrian’s willingness to cross the road and user preferences of a light-based eHMI (light bar on the bumper) and two sound-based eHMIs (bell sound and droning sound), and combinations thereof. We found no objective change in pedestrians’ willingness to cross the road based on the nature of eHMI, although people expressed different subjective preferences for the different ways an eHMI may communicate, and sometimes even strong dislike for multi-modal eHMIs. This shows that the modality of the evaluated eHMI concepts had relatively little impact on their effectiveness. Consequently, this lays an important groundwork for accessibility considerations of future eHMIs, and points towards the insight that provisions can be made for taking user preferences into account without compromising effectiveness.

As automated vehicles become more widespread but lack a driver to communicate in uncertain situations, external communication, for example, via LEDs or displays, is evaluated. However, the concepts are mostly evaluated in simple scenarios, such as one person trying to cross in front of one automated vehicle. The traditional empirical approach fails to study the large-scale effects of these in this not-yet-real scenario. Therefore, we built PedSUMO, an enhancement to SUMO for the simulacra of automated vehicles' effects on public traffic, specifically how pedestrian attributes affect their respect for automated vehicle priority at unprioritized crossings. We explain the algorithms used and the derived parameters relevant to the crossing. We open-source our code under https://github.com/M-Colley/pedsumo and demonstrate an initial data collection and analysis of Ingolstadt, Germany.

Minor violations of traffic regulations are common today and partially socially accepted. Automated vehicles (AVs), however, will be obliged to keep to the letter of the law. This can lead to situations where user requests cause the AV to reach its legal boundaries, creating novel user-vehicle conflicts. To investigate whether traffic-violating driver interests are transferred to the automated context, we conducted an online survey with three conflict-prone scenarios (N=49). The results indicate that legally compliant AV behavior is desired but that users would intervene in the vehicle’s behavior to enforce interests. In a subsequent Virtual Reality study (N=30), we evaluated the effect of legal boundary-handling strategies (Responsibility and Control Shift, Responsibility Shift, No Shift) and other traffic participants’ violating traffic regulations on behavior, conflict, and trust in a legally conflict-prone parking scenario. Results show that conflict is perceived significantly higher in all strategies compared to the manual baseline, while situational trust in the vehicle is higher in the automated conditions but independent of the handling strategy.

This study examines three modalities for designing live-in van interiors. Participants (N=18) situated within an empty van were asked to explore potential designs using physical cardboard prototyping, a commercial software application (Vanspace 3D) for planning van interiors, and an augmented reality application that we developed. Participants were asked to think aloud as they designed van interiors for fictive journeys using each modality. A qualitative evaluation was conducted to assess how participants’ conceptualizations of space shifted across the use of each prototyping method. The results demonstrate that each design method influenced design outcomes due to the physicality of the task. This study highlights the importance of considering the role of physicality in the selection of prototyping modality for the design exploration process.

Drivers want to retain a sense of control when driving (partially) automated vehicles (AVs). Future AVs will continue to offer the possibility to drive manually, potentially leading to challenging driver-initiated takeovers (DITs) due to the "out-of-the-loop problem" and reduced driving performance. A driving simulator study (N=24) was conducted to explore whether cooperative systems, without full control of driving tasks, provide a sense of control to mitigate DITs in varying conflict situations. Conflict levels were operationalized by an AV performing overtaking maneuvers under free, 100m, and 50m visibility on a two-lane rural road. Participants experienced three systems: no intervention-, a cooperative choice-, and a manual control system. Results showed that participants had a similar sense of control with the cooperative system compared to the manual one and preferred it over the manual system. The likelihood of DITs increased with conflict intensity, and trust in the AV moderated the conflict-DIT association.

Automotive user interface (AUI) evaluation becomes increasingly complex due to novel interaction modalities, driving automation, heterogeneous data, and dynamic environmental contexts. Immersive analytics may enable efficient explorations of the resulting multilayered interplay between humans, vehicles, and the environment. However, no such tool exists for the automotive domain. With AutoVis, we address this gap by combining a non-immersive desktop with a virtual reality view enabling mixed-immersive analysis of AUIs. We identify design requirements based on an analysis of AUI research and domain expert interviews (N=5). AutoVis supports analyzing passenger behavior, physiology, spatial interaction, and events in a replicated study environment using avatars, trajectories, and heatmaps. We apply context portals and driving-path events as automotive-specific visualizations. To validate AutoVis against real-world analysis tasks, we implemented a prototype, conducted heuristic walkthroughs using authentic data from a case study and public datasets, and leveraged a real vehicle in the analysis process.

External communication of automated vehicles is proposed to replace driver-pedestrian communication in ambiguous crossing situations. So far, research has focused on simpler scenarios with one attentive pedestrian and one automated vehicle. This virtual reality study (N=115) investigates a more complex scenario with other crossing pedestrians, a distracting task on the smartphone, and external communication by the automated vehicle. Interaction effects were found for crossing duration, gaze behavior, and subjective measures. For attentive pedestrians, the external communication resulted in shorter crossing durations, higher perceived safety, as well as lower perceived criticality, cognitive workload, and effort. These positive effects were not found when pedestrians were distracted. Instead, distracted pedestrians benefited from other crossing pedestrians because they looked less at the stopping vehicle, felt safer, perceived the situation as less critical, and reported lower cognitive workload and effort. Pedestrians initiated crossings earlier with a group or external communication and later with a smartphone.

Novel display technologies, such as lightfield displays, have become increasingly available. In the automotive domain, these are already in use or are to be used in the future. However, their effect on the user is yet to be explored. Therefore, we conducted a within-subject study (N=15) comparing a baseline visualization of information about automated vehicle functionality with it being visualized either via a LumePad or a LookingGlass display. Interestingly, we found almost no significant differences, thus, indicating that the display technology is less relevant for conveying automated functionality.

In highly automated vehicles, passengers can engage in non-driving-related activities. Additionally, the technical advancement allows for novel interaction possibilities such as voice, gesture, gaze, touch, or multimodal interaction, both to refer to in-vehicle and outside objects (e.g., thermostat or restaurant). This interaction can be characterized by levels of urgency (e.g., based on late detection of objects) and cognitive load (e.g., because of watching a movie or working). Therefore, we implemented a Virtual Reality simulation and conducted a within-subjects study with N=11 participants evaluating the effects of urgency and cognitive load on modality usage in automated vehicles. We found that while all modalities were possible to use, participants relied on touch the most. This was followed by gaze, especially for external referencing. This work helps to further understand multimodal interaction and the requirements this poses on natural interaction in (automated) vehicles.

Today's social media (SM) platforms are toolkits consisting of features with different use cases, some strongly related to habitual and regretful use. Especially Infinite Scrolling (IS) has been reported to make users feel like they are being caught in a loop, regretfully elongating SM sessions. We investigated and defined this loop while unveiling the processes that make users break it. Based on a one-week-long field study (N=46), we unfolded and categorized general reasons for leaving social media and related those to IS. In light of our findings, we argue that SM interventions should not only focus on the app but incorporate the user's context, as most reasons to break SM sessions were not related to the app but the user's general context. Our findings and prior work also indicate the coexistence of multiple loops, which we define as inner (intra-session) loops surrounded by an outer (habitual) loop.

Automated vehicles are expected to substitute driver-pedestrian communication via LED strips or displays. This communication is expected to improve trust and the crossing process in general. However, numerous factors such as other pedestrians’ behavior, perceived time pressure, or previous experience influence crossing decisions. Therefore, we report the results of a triply subdivided Virtual Reality study (N=18) evaluating these. Results show that external communication was perceived as hedonically pleasing, increased perceived safety and trust, and also that pedestrians’ behavior affected participants’ behavior. A timer did not alter crossing behavior, however, repeated exposure increased trust and reduced crossing times, showing a habituation effect. Our work helps better to integrate research on external communication in ecologically valid settings.

Social Media (SM) has shown that we adapt our communication and disclosure behaviors to available technological opportunities. Head-mounted Augmented Reality (AR) will soon allow to effortlessly display the information we disclosed not isolated from our physical presence (e.g., on a smartphone) but visually attached to the human body. In this work, we explore how the medium (AR vs. Smartphone), our role (being augmented vs. augmenting), and characteristics of information types (e.g., level of intimacy, self-disclosed vs. non-self-disclosed) impact the users’ comfort when displaying personal information. Conducting an online survey (N=148), we found that AR technology and being augmented negatively impacted this comfort. Additionally, we report that AR mitigated the effects of information characteristics compared to those they had on smartphones. In light of our results, we discuss that information augmentation should be built on consent and openness, focusing more on the comfort of the augmented rather than the technological possibilities.

Automated vehicles are expected to substitute or even improve driver-driver communication, for example, via LED strips or displays. Numerous situations exist where ambiguities have to be resolved via gestures or implicit communication (i.e., movement). An already demanding situation is the unsignalized four-way intersection. Additionally, Vehicle-To-Everything technology enables automated vehicles to perform maneuvers impossible before such as blocking an intersection to safely let an emergency vehicle pass. Therefore, we report the results of a within-subject Virtual Reality study (N=17) evaluating these two scenarios. Results show that external communication increased perceived safety and reduced mental workload, and also that the novel behavior confused participants. Our work helps better to integrate external communication in settings with manual drivers.

Automated vehicles will change the interaction with the user drastically. While freeing the user of the driving task for most of the journey, the "final 100 meters problem'', directing the vehicle to the final parking spot, could require human intervention. Therefore, we present a classification of interaction concepts for automated vehicles based on modality and interaction mode. In a subsequent Virtual Reality study (N=16), we evaluated sixteen interaction concepts. We found that the medially abstracted interaction mode was consistently rated most usable over all modalities (joystick, speech, gaze, gesture, and tablet). While the steering wheel was still preferred, our findings indicate that other interaction concepts are usable if the steering wheel were unavailable.

With the future availability of highly automated vehicles (AVs), vulnerable road users (VRUs) will encounter vehicles without human operators. To compensate for the lack of eye contact, realizing communication via external human-machine interfaces (eHMIs) is planned. The adequacy of this regarding people with intellectual disabilities (IDs) is, however, still unknown. This work compares eHMI concepts by their perceived user experience (UX) for people with and without ID to evaluate the inclusiveness of current eHMI concepts. We analyzed related work and derived two representative concepts for a visual and an auditory eHMI. Subsequently, a survey of N=120 participants (64 with, 56 without ID) was performed, comparing the perceived UX of the selected eHMI concepts for visual, auditory, and combined modalities, and a baseline without eHMI using videos of simulations. We then had them assessed using the modified user experience questionnaire - short (UEQ-S). We found that auditory eHMIs performed worse than visual or multi-modal ones, and multi-modal concepts performed worse for people with ID in terms of pragmatic quality and crossing decisions. Our insights can be taken by both industry and academia, to make AVs more inclusive.

Infrastructure-mounted sensors that monitor roads can provide essential information for manual drivers and automated vehicles, e.g., positions of other vehicles occluded by buildings. However, human drivers and passengers have to trust and accept their use. This raises the question of how trust can be increased in such a scenario. One important factor for this is understanding the available information, including its quality and, for passengers of automated vehicles, the actions planned based on it. For this, augmented reality is a promising visualization technology because it can present the relevant information integrated into the physical world. Thus, this work develops a taxonomy of augmented reality visualizations for connected automated and manual driving. It is intended to classify and compare existing visualizations, identify novel visualizations, and provide a common language for discussions. The use case infrastructure-supported automated driving is explored by suggesting augmented reality visualizations to inform passengers of automated vehicles and are intended to increase trust. They present information available from infrastructure and onboard sensors as well as the driving decisions based on it. Finally, we evaluated the visualizations' influence on trust in an automated vehicle by conducting a driving simulator study (N=18). Results indicate a high dependency of trust on presenting driving decisions and information on road users but less on location-specific information.

The inclusion of in-vehicle sensors and increased intention and state recognition capabilities enable implicit in-vehicle interaction. Starting from a systematic literature review (SLR) on implicit in-vehicle interaction, which resulted in 82 publications, we investigated state and intention recognition methods based on (1) their used modalities, (2) their underlying level of automation, and (3) their considered interaction focus. Our SLR revealed a research gap addressing implicit interaction in highly automated vehicles (HAVs). Therefore, we discussed how the requirements for implicit state and intention recognition methods and interaction based on them are changing in HAVs. With this, open questions and opportunities for further research in this area were identified.

Automated vehicles are about to enter the mass market. However, such systems regularly meet limitations of varying criticality. Even basic tasks such as Object Identification can be challenging, for example, under bad weather or lighting conditions or for (partially) occluded objects. One common approach is to shift control to manual driving in such circumstances, however, post-automation effects can occur in these control transitions. Therefore, we present ORIAS, a system capable of asking the driver to (1) identify/label unrecognized objects or to (2) select an appropriate action to be automatically executed. ORIAS extends the automation capabilities, prevents unnecessary takeovers, and thus reduces post-automation effects. This work defines the capabilities and limitations of ORIAS and presents the results of a study in a driving simulator (N=20). Results indicate high usability and input correctness.

Autonomous vehicles (AVs) are expected to communicate to vulnerable road users as a substitution of, for example, driver-pedestrian communication, leading to increased safety and acceptance. This communication is currently one-directional, i.e., from the AV to the pedestrian. However, today’s communication between drivers and pedestrians in crossing scenarios is bidirectional. Pedestrians gesture “thank you’’ or wave drivers through in case they do not want to cross. Human drivers often acknowledge this, for example, with a nod. We present an experiment in Virtual Reality (N=20), in which the effect of such acknowledgment of the AVs via its external communication is investigated for the two described scenarios and concerning pedestrian presence. Results show that such feedback is perceived as highly necessary, depends on the scenario, and improves the perceived intelligence of the AV, confirming a Halo-Effect.

Traffic behavior and its perception is shaped by various factors such as vehicle color or size. Decals are used to express information about the owner’s beliefs or are intended to be funny. In the future, with external displays on (automated) vehicles, individualized customization could be even more pronounced. While some research looked at the messages these decals convey, it is unclear how these decals influence the perception of surrounding drivers on the operator of the vehicle. We gathered data on decals in 29 cities in 8 countries. A thematic analysis unveiled 17 dominant themes among decals. Subsequently, we investigated effects of decals of 9 supra-regional common themes in an online study (N=64) finding that participants attributed different characteristics to the driver of a vehicle with a decal based on the type of decal and the participants’ country of origin. Additionally, a Virtual Reality study (N=16) revealed diverse opinions on future usage of such personalization options.

Autonomous vehicles could improve mobility, safety, and inclusion in traffic. While this technology seems within reach, its successful introduction depends on the intended user’s acceptance. A substantial factor for this acceptance is trust in the autonomous vehicle’s capabilities. Visualizing internal information processed by an autonomous vehicle could calibrate this trust by enabling the perception of the vehicle’s detection capabilities (and its failures) while only inducing a low cognitive load. Additionally, the simultaneously raised situation awareness could benefit potential take-overs. We report the results of two comparative online studies on visualizing semantic segmentation information for the human user of autonomous vehicles. Effects on trust, cognitive load, and situation awareness were measured using a simulation (N=32) and state-of-the-art panoptic segmentation on a pre-recorded real-world video (N=41). Results show that the visualization using Augmented Reality increases situation awareness while remaining low cognitive load.

Recent automotive research often focuses on automated driving, including the interaction between automated vehicles (AVs) and so-called “vulnerable road users” (VRUs). While road safety statistics and traffic psychology at least define VRUs as pedestrians, cyclists, and motorcyclists, many publications on human-vehicle interaction use the term without even defining it. The actual target group remains unclear. Since each group already poses a broad spectrum of research challenges, a one-fits-all solution seems unrealistic and inappropriate, and a much clearer differentiation is required. To foster clarity and comprehensibility, we propose a literature-based taxonomy providing a structured separation of (vulnerable) road users, designed to particularly (but not exclusively) support research on the communication between VRUs and AVs. It consists of two conceptual hierarchies and will help practitioners and researchers by providing a uniform and comparable set of terms needed for the design, implementation, and description of HCI applications.

Augmented Reality (AR) glasses equip users with the tools to modify the visual appearance of their surrounding environment. This might severely impact interpersonal communication, as the conversational partners will no longer share the same visual perception of reality. Grounded in color-in-context theory, we present a potential AR application scenario in which users can modify the color of the environment to achieve subconscious benefits. In a consecutive online survey (N=64), we measured the user’s comfort, acceptance of altering and being altered, and how it is impacted by being able to perceive or not perceive the alteration. We identified significant differences depending on (1) who or what is the target of the alteration, (2) which body part is altered, and (3) which relationship the conversational partners share. In light of our quantitative and qualitative findings, we discuss ethical and practical implications for future devices and applications that employ visual alterations.

In (partially) automated vehicles, users will sometimes have to take over control due to system failure or reach of an operational driving domain end. In such scenarios, the user becomes the driver and has to quickly gain situational awareness. With advanced sensory, an automated vehicle could aid the user in building situational awareness by providing information. A literature analysis found no commonly conveyed information. Therefore, to evaluate the effects of four different abstraction levels (high, medium-high, medium, and low) and used modalities (visual vs. visual+auditory) on situational awareness and accompanying usability scores, we evaluated eight systems and a baseline without information display. In the between-subjects online monitor-based study (N=225), we found that while subjective measures are higher and a warning is required, providing abstract information does not improve objective situational awareness, and only being provided with visual information was preferred.

Pedestrians are very vulnerable road users. Autonomous vehicles driving on the street are expected to reduce the risks for pedestrians. However, traffic will include manually driven vehicles for the next decades; therefore, the risk remains and can even increase due to pedestrians’ overreliance on technology. Thus, we propose to use autonomous vehicles parked at the side of the road to continually survey its surroundings and to issue warnings to pedestrians in potentially risky situations. Findings of a Virtual Reality (N=20) study show that participants preferred the communication on the vehicle compared to a communication on the sidewalk, that the visualization grabbed participants’ attention, and that overall crossing time was not significantly affected. This concept highlights the potential autonomous vehicles could have in making traffic safer, even while being parked.

Autonomous vehicles carry the potential to greatly improve mobility and safety in traffic. However, this technology has to be accepted and of value for the intended users. One challenge on this way is the detection and recognition of pedestrians and their intentions. While there are technological solutions to this problem, there seems to be no research on how to make this information transparent to the user in order to calibrate the user’s trust. Our work presents a comparative study of 5 visualization techniques with Augmented Reality or tablet-based visualization technology and two or three information clarity states of pedestrian intention in the context of highly automated driving. We investigated these in a user study in Virtual Reality (N=15). We found that such a visualization was rated reasonable, necessary, and that especially the Augmented Reality-based version with three clarity states was preferred.

Automated vehicles will change the trucking industry as human drivers become more absent. In crossing scenarios, external communication concepts are already evaluated to resolve potential issues. However, automated delivery poses unique communication problems. One specific situation is the delivery to the curb with the truck remaining partially on the street, blocking sidewalks. Here, pedestrians have to walk past the vehicle with reduced sight, resulting in safety issues. To address this, we conducted a literature survey revealing the lack of addressing external communication of automated vehicles in situations other than crossings. Afterwards, a study in Virtual Reality (N=20) revealed the potential of such communication. While the visualization (e.g., arrows or text) of whether it is safe to walk past the truck only played a minor part, the information of being able to safely walk past was highly appreciated. This shows that external communication concepts carry great potential besides simple crossing scenarios.

Autonomous vehicles are on the verge of entering the mass market. Communication between these vehicles with vulnerable road users could increase safety and ease their introduction by helping to understand the vehicle’s intention. Numerous communication modalities and messages were proposed and evaluated. However, these explorations do not account for the factors described in communication theory. Therefore, we propose a two-part design space consisting of a concept part with 3 dimensions and a situation part with 6 dimensions based on a literature review on communication theory and a focus group with experts (N=4) on communication. We found that most work until now does not address situation-specific aspects of such communication.

People with vision impairments (VIP) are among the most vulnerable road users in traffic. Autonomous vehicles are believed to reduce accidents but still demand some form of external communication signaling relevant information to pedestrians. Recent research on the design of vehicle-pedestrian communication (VPC) focuses strongly on concepts for a non-disabled population. Our work presents an inclusive user-centered design for VPC, beneficial for both vision impaired and seeing pedestrians. We conducted a workshop with VIP (N=6), discussing current issues in road traffic and comparing communication concepts proposed by literature. A thematic analysis unveiled two important themes: number of communicating vehicles and content (affecting duration). Subsequently, we investigated these in a second user study in virtual reality (N=33, 8 VIP) comparing the VPC between groups of abilities. We found that trust and understanding is enhanced and cognitive load reduced when all relevant vehicles communicate; high content messages also reduce cognitive load.

When player skill levels are not matched, games provide an unsatisfying player experience. Player balancing is used across many digital game genres to address this, but has not been studied for co-located augmented reality (AR) tabletop games, where using boosts and handicaps can adjust for different player skill levels. In the setting of an AR table football game, we studied the importance of game balancing being triggered by the game system or the players, and whether player skill should be required to trigger game balancing. We implemented projected icons to prominently display game balancing mechanics in the AR table football game. In a within-subjects study (N=24), we found players prefer skill-based control over game balancing and that different triggers are perceived as having different fairness. Further, the study showed that even game balancing that is perceived as unfair can provide enjoyable game experiences. Based on our findings, we provide suggestions for player balancing in AR tabletop games.