Research has characterized mind-wandering as humans’ natural mental state, with moments of task-focused attention being the exception. With this framing, mind-wandering while driving likely occurs more than generally acknowledged, and seems poised to increase with higher levels of automation. This in turn may have adverse effects on drivers’ abilities to regain situation awareness or resume control when needed. Of the prior work on detecting mind-wandering while driving, none focuses on automation or complex urban environments. We ran an exploratory study (N = 14) of an automated drive through New York City in a two-dimensional virtual reality context, focusing on physiological measures such as gaze distribution, pupillometry, and heart rate. We also explored how drivers missing critical events may be a potential new measure. Results varied between focused and mind-wandering mental states and between moving and stopped driving contexts. These observations are an initial step toward understanding mind-wandering across diverse driving scenarios.

Automated emergency braking (AEB) systems-which alert a driver to approaching hazards and automatically brake-are currently available in some vehicles and will soon be widespread. Due to the uncertainties inherent in any environment and difficulties in processing sensor data, these systems are prone to both false alarms and system misses of hazards. A pressing design concern is whether to bias these systems toward a higher likelihood of false alarms versus system misses for non-fatal events. We investigated how drivers form mental models of the AEB system and how that influences their reliance on the system during a critical, potentially fatal, failure. In a full vehicle driving simulator, participants experienced nine interactions that reflected the system's level of bias toward false alarms or toward misses for non-fatal events or that demonstrated perfect performance. When a potentially fatal event occurred, participants trained to expect misses were better able to avoid a pedestrian in the road after a detection failure than those using a system with perfect performance or false alarms. These findings suggest that systems biased toward misses in non-fatal events encourage driver vigilance and preparedness for potentially fatal events.

Dominance is a key aspect of interpersonal relationships. To what extent do nonverbal indicators related to dominance status translate to a nonanthropomorphic robot? An experiment ( N = 25) addressed whether a mobile robot's motion style can influence people's perceptions of its status. Using concepts from improv theater literature, we developed two motion styles across three scenarios (robot makes lateral motions, approaches, and departs) to communicate a robot's dominance status through nonverbal expression. In agreement with the literature, participants described a motion style that was fast, in the foreground, and more animated as higher status than a motion style that was slow, in the periphery, and less animated. Participants used fewer negative emotion words to describe the robot with the purportedly high-status movements versus the purportedly low-status movements, but used more negative emotion words to describe the robot when it made departing motions that occurred in the same style. This result provides evidence that guidelines from improvisational theater for using nonverbal expression to perform interpersonal status can be applied to influence perception of a nonanthropomorphic robot's status, thus suggesting that useful models for more complicated behaviors might similarly be derived from performance literature and theory.

Lack of trust can arise when people do not know what autonomous vehicles perceive in the environment. To convey this information without causing alarm or compelling people to act, we designed and evaluated a way to sonify an autonomous vehicle's perception of salient driving events using abstract auditory icons, or "earcons." These are localized in space using an in-car quadraphonic speaker array to correspond with the direction of events. We describe the interaction design for these awareness cues and a validation experiment (N=28) examining the effects of sonified events on drivers' sense of situation awareness, comfort, and trust. Overall, this work suggests that our designed earcons do improve people's awareness of in-simulation events. The effect of the increased situational awareness on trust and comfort is inconclusive. However, post-study design feedback suggests that sounds should have low levels of intensity and dissonance, and a sense of belonging to a common family.

In partially automated driving, rapid transitions of control present a severe hazard. How long does it take a driver to take back control of the vehicle when engaged with other non-driving tasks? In this driving simulator study, we examined the performance of participants (N=30) after an abrupt loss of automated vehicle control. We tested three transition time conditions, with an unstructured transition of control occurring 2s, 5s, or 8s before entering a curve. As participants were occupied with an active secondary task (playing a game on a tablet) while the automated driving mode was enabled, they needed to disengage from the task and regain control of the car when the transition occurred. Few drivers in the 2 second condition were able to safely negotiate the road hazard situation, while the majority of drivers in the 5 or 8 second conditions were able to navigate the hazard situation safely.

There is a growing need to study the interactions between drivers and their increasingly autonomous vehicles. This paper describes a method of using a low-cost, portable, and versatile driver interaction system in commercial passenger vehicles to enable on-road partial and fully autonomous driving interaction studies. By conducting on-road Wizard-of-Oz studies in naturalistic settings, we can explore a range of driving conditions and scenarios far beyond what can be conducted in laboratory simulator environments. The Marionette system uses off-the-shelf components to cre- ate bidirectional communication between the driving controls of a Wizard-of-Oz vehicle operator and a driving study participant. It signals to the study participant what the car is doing and enables researchers to study participant intervention in driving activity. Mar- ionette is designed to be easily replicated for researchers studying partially autonomous driving interaction. This paper describes the design and evaluation of this system.

Information about drivers' mental states can be vital to the design of interfaces for highly automated vehicles. Functional near infrared spectroscopy (fNIRS) is a neuroimaging tool that is fast becoming popular to study the cortical activity of participants in HCI experiments and driving simulator studies in particular. The analysis methods of the fNIRS data create requirements in the experimental design such as repeated measures. In this paper, we present a study of the event related cortical activity of the drivers of manual, partially autonomous, and fully autonomous cars when performing lane changes using functional near infrared spectroscopic measures. We also present the experimental methodology that was adopted to meet the needs of the fNIRS measurement and the subsequent analysis. The study (N=28) was conducted in a driving simulator. Participants drove for approximately 7 minutes and performed 8 lane change maneuvers in each mode of automation. Multiple streams of data including 4 time-synced video recordings, NASA TLX questionnaires and fNIRS data were recorded and analyzed. It was found that the dorsolateral prefrontal cortex activation during lane changes performed in a partially autonomous mode of operation was just as high as that during a manual lane change, showing that drivers of partially automated systems are as cognitively engaged as drivers of manually operated vehicles.

In this paper, we introduce two different transforming steering wheel systems that can be utilized to augment user experience for future partially autonomous and fully autonomous vehicles. The first one is a robotic steering wheel that can mechanically transform by using its actuators to move the various components into different positions. The second system is a LED steering wheel that can visually transform by using LEDs embedded along the rim of wheel to change colors. Both steering wheel systems contain onboard microcontrollers developed to interface with our driving simulator. The main function of these two systems is to provide emergency warnings to drivers in a variety of safety critical scenarios, although the design space that we propose for these steering wheel systems also includes the use as interactive user interfaces. To evaluate the effectiveness of the emergency alerts, we conducted a driving simulator study examining the performance of participants (N=56) after an abrupt loss of autonomous vehicle control. Drivers who experienced the robotic steering wheel performed significantly better than those who experienced the LED steering wheel. The results of this study suggest that alerts utilizing mechanical movement are more effective than purely visual warnings.

Automated driving systems that share control with human drivers by using haptic feedback through the steering wheel have been shown to have advantages over fully automated systems and manual driving. Here, we describe an experiment to elicit tacit expectations of behavior from such a system. A gaming steering wheel electronically coupled to the steering wheel in a full-car driving simulator allows two participants to share control of the vehicle. One participant was asked to use the gaming wheel to act as the automated driving agent while another participant acted as the car driver. The course provided different information and visuals to the driving agent and the driver to simulate possible automation failures and conflict situations between automation and the driver. The driving agent was also given prompts that specified a communicative goal at various points along the course. Both participants were interviewed before and after the drive, and vehicle data and drive video were collected. Our results suggest that drivers were able to interpret simple trajectory intentions, such as a lane change, conveyed by the driving agent. However, the driving agent was not able to effectively communicate more nuanced, higher level ideas such as availability, primarily due to the steering wheel being the control mechanism. Torque on the steering wheel without warning was seen most often as a failure of automation. Gentle and steady steering movements were viewed more favorably.

In this study, we examined how participants (N=20) interacted and collaborated with a set of robotic drawers to accomplish an assembly task. Using Wizard of Oz techniques, the drawers' behavior varied along two dimensions - proactivity and expressivity of motions. The results of our study indicate that participants consider an expressive robot to be more involved and interested in the interaction. We also observe that while proactive or expressive robots can dominate the interaction, proactivity may negatively affect the participants' perception of their social status relative to that of the robot's, while expressiveness does not. This shows the importance of utilizing expressive movements when designing socially appropriate robots that collaborate with human users.

This tutorial is a hands-on introduction to human-centered design topics and practices for human-robot interaction. It is intended for researchers with a variety of backgrounds, particularly those with little or no prior experience in design. In the morning, participants will learn about user needs and needfinding, as ways to understand the stakeholders in research outcomes, guide the selection of participants, and as possible measures of success. We then focus on design sketching, including ways to represent objects, people and their interactions through storyboards. Design sketching is not intended to be art, rather a way to develop and build upon ideas with oneself, and quickly communicate with colleagues. In the afternoon, participants will use the tools and materials, and learn techniques for lightweight physical prototyping and improvisation. Participants will build a small paper robot (not actuated) of their own design, to practice puppeteering, explore bodily movement and prototype interactions.

While automated driving systems will become increasingly capable and common in the future, there will still be instances when human drivers want or need to make corrections to the car's automated driving behavior. We conducted two studies exploring how driving interfaces could be designed to better execute the drivers' intentions. In our first study, adult participants (N=40) experienced a simulated driving scenario that varied the behavior of the car's automation (perfect driving and imperfect driving) and the intervention modalities (takeover and takeover+influence). At certain segments, the car's automation would drive perfectly or weave within the lane. During those times, participants could intervene using the available modalities. When experiencing instances of imperfect driving, drivers who had the ability to takeover+influence intervened more often than drivers who were only given the option to takeover. As intervening would require them to resume full control, drivers in the takeover condition were more tolerant of the imperfect driving. Also, most drivers tried to intervene initially by influencing the car, even those drivers who were only given the ability to takeover. In our second study, we examined how participants (N=40) of different demographics (high school students and seniors) would respond when they were subjected to the imperfect driving scenarios. High school drivers intervened just as much as the adult drivers. However, senior drivers intervened far less. These two studies suggest that when intervention is necessary, human drivers have a desire for shared control, which allows them to act as supervisors rather than operators of automated vehicles.

How will pedestrians and bicyclists interact with autonomous vehicles when there is no human driver? In this paper, we outline a novel method for performing observational field experiments to investigate interactions with driverless cars. We provide a proof-of-concept study (N=67), conducted at a crosswalk and a traffic circle, which applies this method. In the study, participants encountered a vehicle that appeared to have no driver, but which in fact was driven by a human confederate hidden inside. We constructed a car seat costume to conceal the driver, who was specially trained to emulate an autonomous system. Data included video recordings and participant responses to post-interaction questionnaires. Pedestrians who encountered the car reported that they saw no driver, yet they managed interactions smoothly, except when the car misbehaved by moving into the crosswalk just as they were about to cross. This method is the first of its kind, and we believe that it contributes a valuable technique for safely acquiring empirical data and insights about driverless vehicle interactions. These insights can then be used to design vehicle behaviors well in advance of the broad deployment of autonomous technology.

This paper describes our approach to designing, developing behaviors for, and exploring the use of, a robotic footstool, which we named the mechanical ottoman. By approaching unsuspecting participants and attempting to get them to place their feet on the footstool, and then later attempting to break the engagement and get people to take their feet down, we sought to understand whether and how motion can be used by non-anthropomorphic robots to engage people in joint action. In several embodied design improvisation sessions, we observed a tension between people perceiving the ottoman as a living being, such as a pet, and simultaneously as a functional object, which requests that they place their feet on it-something they would not ordinarily do with a pet. In a follow-up lab study (N=20), we found that most participants did make use of the footstool, although several chose not to place their feet on it for this reason. We also found that participants who rested their feet understood a brief lift and drop movement as a request to withdraw, and formed detailed notions about the footstool's agenda, ascribing intentions based on its movement alone.

With automated driving systems, drivers may still be expected to resume full control of the vehicle. While structured transitions where drivers are given warning are desirable, it is critical to benchmark how drivers perform when transition of control is unstructured and occurs without advanced warning. In this study, we observed how participants (N=27) in a driving simulator performed after they were subjected to an emergency loss of automation. We tested three transition time conditions, with an unstructured transition of vehicle control occurring 2 seconds, 5 seconds, or 8 seconds before the participants encountered a road hazard that required the drivers' intervention. Few drivers in the 2 second condition were able to safely negotiate the road hazard situation, while the majority of drivers in 5 or 8 second conditions were able to navigate the hazard safely. Similarly, drivers in 2 second condition rated the vehicle to be less likeable than drivers in 5 and 8 second conditions. From the study results, we are able to narrow in on a minimum amount of time in which drivers can take over the control of vehicle safely and comfortably from the automated system in the advent of an impending road hazard.

In this study, we examined how participants (N=20) interacted and collaborated with a set of robotic drawers to accomplish an assembly task. The drawers' behavior varied along two dimensions - proactivity and expressivity of motions. The results of our study indicate that participants consider an expressive robot to be more involved and interested in the interaction. We also found that while proactive or expressive robots could dominate the interaction, proactivity might negatively affect the participants' perception of their social status relative to that of the robot's, while expressiveness did not. This shows the importance of utilizing expressive movements when designing socially appropriate robots that collaborate with human users.

Service robots in public places need to both understand environmental cues and move in ways that people can understand and predict. We developed and tested interactions with a trash barrel robot to better understand the implicit protocols for public interaction. In eight lunch-time sessions spread across two crowded campus dining destinations, we experimented with piloting our robot in Wizard of Oz fashion, initiating and responding to requests for impromptu interactions centered on collecting people's trash. Our studies progressed from open-ended experimentation to testing specific interaction strategies that seemed to evoke clear engagement and responses, both positive and negative. Observations and interviews show that a) people most welcome the robot's presence when they need its services and it actively advertises its intent through movement; b) people create mental models of the trash barrel as having intentions and desires; c) mistakes in navigation are indicators of autonomous control, rather than a remote operator; and d) repeated mistakes and struggling behavior polarized responses as either ignoring or endearing.

The role of human-robot interaction is becoming more important as everyday robotic devices begin to permeate into our lives. In this study, we video-prototyped a user's interactions with a set of robotic drawers. The user and robot each displayed one of five emotional states - angry, happy, indifferent, sad, and timid. The results of our study indicated that the participants of our online questionnaire preferred empathetic drawers to neutral ones. They disliked robotic drawers that displayed emotions orthogonal to the user's emotions. This showed the importance of displaying emotions, and empathy in particular, when designing robotic devices that share our living and working spaces.