In this study, we focus on different strategies drivers use in terms of interleaving between driving and non-driving related tasks (NDRT) while taking back control from automated driving. We conducted two driving simulator experiments to examine how different cognitive demands of texting, priorities, and takeover time budgets affect drivers’ takeover strategies. We also evaluated how different takeover strategies affect takeover performance. We found that the choice of takeover strategy was influenced by the priority and takeover time budget but not by the cognitive demand of the NDRT. The takeover strategy did not have any effect on takeover quality or NDRT engagement but influenced takeover timing.

The growing availability of generative AI technologies such as large language models (LLMs) has significant implications for creative work. This paper explores twofold aspects of integrating LLMs into the creative process – the divergence stage of idea generation, and the convergence stage of evaluation and selection of ideas. We devised a collaborative group-AI Brainwriting ideation framework, which incorporated an LLM as an enhancement into the group ideation process, and evaluated the idea generation process and the resulted solution space. To assess the potential of using LLMs in the idea evaluation process, we design an evaluation engine and compared it to idea ratings assigned by three expert and six novice evaluators. Our findings suggest that integrating LLM in Brainwriting could enhance both the ideation process and its outcome. We also provide evidence that LLMs can support idea evaluation. We conclude by discussing implications for HCI education and practice.

The coronavirus pandemic has significantly disrupted information work across the globe. Since March 2020 when the World Health Organization designated Covid-19 as a pandemic, workplaces across the ...

1. The advent of the COVID-19 pandemic has forced millions of workers to suddenly shift their activity out of their offices and into their homes: 5–15% of Americans worked from home before the pand...

We explore the transfer of control from an automated vehicle to the driver. Based on data from N=19 participants who participated in a driving simulator experiment, we find evidence that the transfer of control often does not take place in one step. In other words, when the automated system requests the transfer of control back to the driver, the driver often does not simply stop the non-driving task. Rather, the transfer unfolds as a process of interleaving the non-driving and driving tasks. We also find that the process is moderated by the length of time available for the transfer of control: interleaving is more likely when more time is available. Our interface designs for automated vehicles must take these results into account so as to allow drivers to safely take back control from automation.

Recent technological advancements in automation have sparked interest in how automation will affect truck drivers and the trucking industry. However, there is a gap in the literature addressing how truck drivers perceive automation and how they believe it will impact trucking. This study aims to understand truck drivers’ perspectives on automation in the trucking industry. Extending a preliminary study, we conducted a broader analysis of comments discussing automation in the r/Truckers subreddit from February 2017 to March 2021. In general, the community had negative sentiments towards automation in the trucking industry. Participants speculated when automation would become mainstream in trucking and discussed the feasibility of automation in the context of executing non-driving tasks and having accommodating infrastructure. Our findings indicate that truck drivers seek to participate in conversations about the future and to prepare themselves for when automation is more prominent in the trucking industry.

Individual and organizational computer security rests on how people interpret and use the security information they are presented. One challenge is determining whether a given URL is safe or not. This paper explores the visual behaviors that users employ to gauge URL safety. We conducted a user study on 20 participants wherein participants classified URLs as safe or unsafe while wearing an eye tracker that recorded eye gaze (where they look) and pupil dilation (a proxy for cognitive effort). Among other things, our findings suggest that: users have a cap on the amount of cognitive resources they are willing to expend on vetting a URL; they tend to believe that the presence of www in the domain name indicates that the URL is safe; and they do not carefully parse the URL beyond what they perceive as the domain name.

A Hidden Markov Model framework is introduced to formalize the beliefs that humans may have about the mode in which a semi-automated vehicle is operating. Previous research has identified various “levels of automation,” which serve to clarify the different degrees of a vehicle’s automation capabilities and expected operator involvement. However, a vehicle that is designed to perform at a certain level of automation can actually operate across different modes of automation within its designated level, and its operational mode might also change over time. Confusion can arise when the user fails to understand the mode of automation that is in operation at any given time, and this potential for confusion is not captured in models that simply identify levels of automation. In contrast, the Hidden Markov Model framework provides a systematic and formal specification of mode confusion due to incorrect user beliefs. The framework aligns with theory and practice in various interdisciplinary approaches to the field of vehicle automation. Therefore, it contributes to the principled design and evaluation of automated systems and future transportation systems.

We review the history of human-automation interaction research, assess its current status and identify future directions. We start by reviewing articles that were published on this topic in the International Journal of Human-Computer Studies during the last 50 years. We find that over the years, automated systems have been used more frequently (1) in time-sensitive or safety-critical settings, (2) in embodied and situated systems, and (3) by non-professional users. Looking to the future, there is a need for human-automation interaction research to focus on (1) issues of function and task allocation between humans and machines, (2) issues of trust, incorrect use, and confusion, (3) the balance between focus, divided attention and attention management, (4) the need for interdisciplinary approaches to cover breadth and depth, (5) regulation and explainability, (6) ethical and social dilemmas, (7) allowing a human and humane experience, and (8) radically different human-automation interaction.

As vehicles of the future take on more of the driving responsibility and the role of the driver transitions into more of a monitoring capacity, the traditional notions of interruption and attention management needs to be reconsidered for automated vehicles. We argue that the transfer of control between the automated vehicle and the human driver can be considered as an interruption handling process, and that this process goes through a series of ten explicit stages. Each stage has its own characteristics and implications for practice and future research. Therefore, in this paper we identify for each stage what is known from theory, together with important implications for safety, design, and future research, especially for human-machine interaction. More generally, the framework makes explicit that it is not appropriate to think of transfer of control as a single event or even small set of events. The framework also highlights that it might not be realistic to expect human drivers to immediately respond correctly to a system initiated request to transfer control, given that humans interleave their attention between non-driving and driving tasks, and given that a transition constitutes of multiple stages. These nuances are accounted for in the framework.

Augmented reality navigation aids have been investigated in a number of studies, and results are encouraging, especially for large, head-up displays. However, such displays are not commercially available -- in fact they are rare in laboratories as well. In this paper we ask: would drivers be well-served with a navigation aid that overlays AR content on a live feed from a camera that shows the forward road? To answer this question we conducted a simulator-based study and compared the use of such a navigation aid to the use of a head-up AR aid, as well as to the use of 2D map shown on a head-down display. Our results confirm prior results that a head-up AR navigation aid can keep drivers' visual attention on the road, and that drivers like such a navigation aid. Our results also indicate that a camera-view AR navigation aid might not be well-received by drivers.

Technological advances offer new methods of representing physical objects in tangible and virtual forms. This study compares learning outcomes from 61 students as they interact with ancient Egyptian sculptures using three increasingly popular educational technologies: HoloLens AR headset, 3D model viewing website (SketchFab), and plastic extrusion 3D prints. We explored how differences in interaction styles affect the learning process, quantitative and qualitative learning outcomes, and critical analysis.

Many automated vehicles are already on our roads, and we can expect that many more will follow soon. Yet, the legal frameworks that govern the deployment and operation of these vehicles are still under development. This paper explores legal issues related to the human-machine interaction for automated vehicles. The paper reviews the current legal landscape, with a focus on the United States, and presents some of the issues that will be of interest to researchers, developers, and regulators as the new legal frameworks take shape.

The objective of this tutorial is to provide MobileHCI newcomers to the domain of automotive user interfaces (AutomotiveUI) with an introduction and overview of the field. The tutorial will introduce the specifics and challenges of in-vehicle user interfaces that set this field apart from others. With a clear focus on the integration of mobile devices into the car, we will provide an overview of the specific requirements of AutomotiveUI, discuss the design of such interfaces, also with regard to standards and guidelines. We further outline how to evaluate interfaces in the car, discuss the challenges with upcoming automated driving and present trends and challenges in this domain.

The Microsoft HoloLens keeps track of its location and rotation relative to the environment but lacks the ability to capture eye gaze data. We assess a novel method to extend the HoloLens with a head mounted eye-tracker. Using a combination of eye gaze data and head rotation we compared gaze behavior between real and virtual objects. Results indicate that eye-tracking plays an important role in accurately determining a user's gaze for real objects in contrast to virtual objects.

We present findings from an empirical study of how groups of eight users collaborate on a decision-making task around an interactive tabletop. To our knowledge, this is the first study to examine co-located collaboration in larger groups (of 8-12 users) seated around a large-scale high-resolution multi-touch horizontal display. Our findings shed light on: 1) the effect of collaboration patterns of larger groups on equity of participation; 2) the role of participants' position around the tabletop in forming collaborations; and 3) the mechanisms, which facilitate coordination and collaboration in larger group interacting around large-scale tabletops; We also contribute computational methods that leverage image processing to analyze interaction around large-scale tabletops. Finally, we discuss implications for the design of large-scale tabletop systems for supporting co-located collaboration in larger groups.

Is there a difference in behavior when drivers start driving after parking compared to taking over from an autonomous driving car? In the former, the driving context switch (from static to driving) might be bigger than the latter, where drivers are already in a moving vehicle. This bigger difference might be paired with a decision to stop attending to any distracting task since drivers might find themselves in a different state after driving away. Participants drove a straight highway in a simulator. They either took over driving after being driven autonomously, or after being parked. Concurrently, we played distracting videos in the simulator. Participants looked more towards the road while the car was driving autonomously but there was no difference in driving performance and gazes towards the distraction after take-over compared to starting after parking. This implies that despite a difference in attention before takeover, the control switch is similar.

In this paper, we present a proof-of-concept approach to estimating mental workload by measuring the user's pupil diameter under various controlled lighting conditions. Knowing the user's mental workload is desirable for many application scenarios, ranging from driving a car, to adaptive workplace setups. Typically, physiological sensors allow inferring mental workload, but these sensors might be rather uncomfortable to wear. Measuring pupil diameter through remote eye-tracking instead is an unobtrusive method. However, a practical eye-tracking-based system must also account for pupil changes due to variable lighting conditions. Based on the results of a study with tasks of varying mental demand and six different lighting conditions, we built a simple model that is able to infer the workload independently of the lighting condition in 75% of the tested conditions.

By the nature of their jobs first responders have to interact with in-vehicle devices even as they drive under challenging road conditions. In this paper we assess the state-of-the-art in creating safe in-vehicle user interfaces for first responders, and we propose six research and development priorities for future work in this realm.

It is well established that driving while interacting with a secondary in-car device, such as a portable music player, is distracting and can lead to a decline in driver safety and performance. One aspect that has not received as much attention though is the extent to which drivers adapt their interactions with an in-car device to the changing demands of the road. Do drivers adopt compensatory strategies or tactics when driving in more demanding settings? We investigate this question by conducting a driving simulator study in which participants were required to drive either in a city or a highway environment. During these drives, participants were asked to interact with an MP3 music player and make a series of either easy or difficult selections. It was found that participants who drove in the city made shorter glances to the in-car iPod than those that drove on the highway. As a result of this tactical change, participants had better lane keeping performance in the city, which was important given the narrower traffic lanes. As expected, we also replicate the well-known effect that more complex secondary in-car tasks are more distracting than simpler in-car tasks. The contribution this paper makes to the automotive UI community is that it provides evidence that drivers adapt to the demands of the driving environment even when interacting with secondary devices.

We explore the feasibility of using pupil diameter to estimate how the cognitive load of the driver changes during a spoken dialogue task with a remote conversant. The conversants play a series of Taboo games, which do not follow a structured turn-taking nor initiative protocol. We contrast the driver's pupil diameter when the remote conversant begins speaking with the diameter right before the driver responds. Although we find a significant difference in pupil diameter for the first pair in each game, subsequent pairs show little difference. We speculate that this is due to the less structured nature of the task, where there are no set time boundaries on when the conversants work on the task. This suggests that spoken dialogue systems for in-car use might better manage the driver's cognitive load by using a more structured interaction, such as system-initiative dialogues.

Most vehicles only require a key to authenticate the driver. However, with vehicles becoming portals to digital information, many drivers might find this authentication method inadequate. In this paper we explore using tap sequences on the back of the steering wheel to authenticate drivers. Our results indicate that drivers can learn to use an authentication system that uses such taps, and that the system could provide good protection from shoulder-surfing attacks.

The Project54 mobile system for law enforcement developed at the University of New Hampshire integrates the control of disparate law enforcement devices such as radar, VHF radio, video, and emergency lights and siren. In addition it provides access to state and national law enforcement databases via wireless data queries. Officers using Project54 are free to inter-mix three different user interface modes: the device native controls; an LCD touchscreen with keyboard and mouse; and voice commands with voice feedback. The Project54 system was utilized by the New Hampshire State Police agency wide for a period of seven years spanning 2005 through 2011. This paper presents an analysis of user preferences in regard to user interface modes during the three years 2009 through 2011, obtained through logs of daily system use in approximately 200 police cruisers. Results indicate that most officers chose to use the touch screen controls frequently instead of the device native controls, but only a minority chose to use the speech command interface.

Augmented reality (AR) navigation systems are likely to improve the driving experience compared to today's personal navigation devices on the dashboard, as they don't require glances away from the road ahead. As technology is not yet capable to deliver an affordable and seamless HUD AR solution, we explore an inexpensive version of augmentation, which would have a similar benefit of reduced distraction. We propose using an LED (light emitting diode) matrix in the periphery of the driver's vision to indicate turns on the road. We find that such a system produces better results in visual attention, driving performance and in subjective measures compared to standard navigation devices.

In a driving simulator study, we explore the feasibility of using pupil diameter to estimate how the cognitive load of the driver changes during a spoken dialogue with a remote conversant. We confirm that it is feasible to use pupil diameter to differentiate between parts of the dialogue that increase the cognitive load of the driver, and those that decrease it. Our long term goal is to build a spoken dialogue system that can adapt its behavior when the driver is under high cognitive load, whether from the driving task or the dialogue task. Index Terms: dialog, cognitive load, pupil diameter, driving 1.

In driving simulator studies pupil diameter is often employed as a physiological measure of cognitive load. However, pupil size is primarily influenced by the pupillary light reflex (PLR). In this paper, we explore the influence of the size and luminance of visual targets on the PLR. Our results indicate that even for small targets (angular radius of 2.5°) changes in luminance can result in PLR that can obscure cognitive load-related pupil diameter changes. We propose a weighting function to be used to predict the PLR and present initial results that support its utility.

Pupil diameter is an important measure of cognitive load. However, pupil diameter is also influenced by the amount of light reaching the retina. In this study we explore the interaction between these two effects in a simulated driving environment. Our results indicate that it is possible to separate the effects of illumination and visual cognitive load on pupil diameter, at least in certain situations.

Today's in-car information systems are undergoing an evolution towards realistic visualization as well as to real-time telematics services. In a road study with 31 participants we explored the communication of safety information to the driver. We compared three presentation styles: audio-only, audiovisual with a conventional map, and audiovisual with augmented reality. The participants drove on a motorway route and were confronted with recommendations for route following, speed limitation, lane utilization, unexpected route change, and emergency stops. We found significant differences between these safety scenarios in terms of driving performance, eye glances and subjective preference. Comparing the presentation styles, we found that following such recommendations was highly efficient in the audio-only mode. Additional visual information did not significantly increase driving performance. As our subjective preference data also shows, augmented reality does not necessarily create an added value when following safety-related traffic recommendations. However, additional visual information did not interfere with safe driving. Importantly, we did not find evidence for a higher distraction potential by augmented reality; drivers even looked slightly less frequently on the human-machine interface screen in the augmented reality mode than with conventional maps.

Prior research has shown that when drivers look away from the road to view a personal navigation device (PND), driving performance is affected. To keep visual attention on the road, an augmented reality (AR) PND using a heads-up display could overlay a navigation route. In this paper, we compare the AR PND, a technology that does not currently exist but can be simulated, with two PND technologies that are popular today: an egocentric street view PND and the standard map-based PND. Using a high-fidelity driving simulator, we examine the effect of all three PNDs on driving performance in a city traffic environment where constant, alert attention is required. Based on both objective and subjective measures, experimental results show that the AR PND exhibits the least negative impact on driving. We discuss the implications of these findings on PND design as well as methods for potential improvement.

With the proliferation of pervasive devices and the increase in their processing capabilities, client-side speech processing has been emerging as a viable alternative. The SiMPE workshop series started in 2006 [5] with the goal of enabling speech processing on mobile and embedded devices to meet the challenges of pervasive environments (such as noise) and leveraging the context they offer (such as location). SiMPE 2010, the latest in the series brought together, very successfully, researchers from the speech and the HCI communities. We believe this is the beginning.

In this article we focus on human–human multi-tasking dialogues, in which pairs of conversants, using speech, work on an ongoing task while occasionally completing real-time tasks. The ongoing task is a poker game in which conversants need to assemble a poker hand, and the real-time task is a picture game in which conversants need to find out whether they have a certain picture on their displays. We employ empirical corpus studies and machine learning experiments to understand the mechanisms that people use in managing these complex interactions. First, we examine task interruptions: switching from the ongoing task to a real-time task. We find that generally conversants tend to interrupt at a less disruptive context in the ongoing task when possible. We also find that the discourse markers oh and wait occur in initiating a task interruption twice as often as in the conversation of the ongoing task. Pitch is also found to be statistically correlated with task interruptions; in fact, the more disruptive the task interruption, the higher the pitch. Second, we examine task resumptions: returning to the ongoing task after completing an interrupting real-time task. We find that conversants might simply resume the conversation where they left off, but sometimes they repeat the last utterance or summarize the critical information that was exchanged before the interruption. Third, we apply machine learning to determine how well task interruptions can be recognized automatically and to investigate the usefulness of the cues that we find in the corpus studies. We find that discourse context, pitch, and the discourse markers oh and wait are important features to reliably recognize task interruptions; and with non-lexical features one can improve the performance of recognizing task interruptions with more than a 50% relative error reduction over a baseline. Finally, we discuss the implication of our findings for building a speech interface that supports multi-tasking dialogue.

We report on the design of spoken tasks for a study that explored how people manage spoken multi-threaded dialogues while one of the conversants is operating a simulated vehicle. Based on a series of preliminary studies we propose a set of considerations that researchers should take into account when designing such tasks. Using these considerations, we discuss two spoken tasks, the parallel twenty questions game and the last letter game, and discuss the successful utilization of these tasks in a study exploring human-human dialogue behavior.

We report on the results of a study in which pairs of subjects were involved in spoken dialogues and one of the subjects also operat-ed a simulated vehicle. We estimated the driver’s cognitive load based on pupil size measurements from a remote eye tracker. We compared the cognitive load estimates based on the physiological pupillometric data and driving performance data. The physiologi-cal and performance measures show high correspondence suggest-ing that remote eye tracking might provide reliable driver cogni-tive load estimation, especially in simulators. We also introduced a new pupillometric cognitive load measure that shows promise in tracking cognitive load changes on time scales of several seconds.

With the proliferation of pervasive devices and the increase in their processing capabilities, client-side speech processing has been emerging as a viable alternative. The SiMPE workshop series started in 2006 [5] with the goal of enabling speech processing on mobile and embedded devices to meet the challenges of pervasive environments (such as noise) and leveraging the context they offer (such as location).

Nowadays, personal navigation devices (PNDs) that provide GPS-based directions are widespread in vehicles. These devices typically display the real-time location of the vehicle on a map and play spoken prompts when drivers need to turn. While such devices are less distracting than paper directions, their graphical display may distract users from their primary task of driving. In experiments conducted with a high fidelity driving simulator, we found that drivers using a navigation system with a graphical display indeed spent less time looking at the road compared to those using a navigation system with spoken directions only. Furthermore, glancing at the display was correlated with higher variance in driving performance measures. We discuss the implications of these findings on PND design for vehicles.

With the proliferation of cell phones around the world, governments have been enacting legislation prohibiting the use of cell phones during driving without a “hands-free” kit, bringing automotive speech recognition to the forefront of public safety. At the same time, the trend in cell phone hardware has been to create smaller and thinner devices with greater computational power and functional complexity, making speech the most viable modality for user input. Given the important role that automotive speech recognition is likely to play in consumer lives, we explore how the accuracy of the speech engine, the use of the push-to-talk button, and the type of dialog repair employed by the interface influences driving performance. In experiments conducted with a driving simulator, we found that the accuracy of the speech engine and its interaction with the use of the push-to-talk button does impact driving performance significantly, but the type of dialog repair employed does not. We discuss the implications of these findings on the design of automotive speech recognition systems.

Datacasting employs the excess bandwidth from digital television signals for use in one-way data transmission, and it is being used successfully for high-speed downloads at fixed locations. There is considerable interest in extending datacast usage to mobile users, although there are reception challenges in the mobile environment that can significantly impact system performance. To explore the feasibility of using datacasting in this environment, datacasting receivers and data logging equipment have been installed in 10 emergency vehicles to record performance characteristics over a wide range of operational conditions. Summary conclusions from that study are described in this paper along with details about the equipment used to make the test and the environmental factors that were found to have the greatest impact on system performance. The use of the low-speed (9600 baud) VHF, emergency-band data channel in conjunction with the datacast channel to provide two- way data transmission is also evaluated and discussed for the mobile environment.

In this paper, we explore the conventions that people use in managing multiple dialogue threads. In particular, we focus on where in a thread people interrupt when switching to another thread. We find that some subjects are able to vary where they switch depending on how urgent the interrupting task is. When time-allowed, they switched at the end of a discourse segment, which we hypothesize is less disruptive to the interrupted task when it is later resumed.

Project54 is an effort to integrate embedded and wireless mobile technologies into the police cruisers of the New Hampshire State Police. While performing their job, police officers often find themselves outside of their cruisers. A prototype system, called the remote access and mobile data transaction system, which provides remote accessibility to some of the functionality and the data available within the in-car Project54 system, was designed and implemented using a palm-sized computer, a two-dimensional barcode scan engine, and wireless communication modules. The system was integrated into the Project54 system using the intelligent transportation system data bus (IDB) and the IDB controllers developed by Project54. The system was successfully tested in laboratory conditions.

The Project54 effort aims to improve the ability of police to manipulate data in mobile units as well as to provide a way to seamlessly integrate all in-car electronic devices. Work is being done the integration of in-car hardware and software, user interface integration, and the integration of the cruiser into a wireless data network. The system is being tested in three New Hampshire State Police cruisers, USA. The entire New Hampshire State Police fleet of 250 cruisers will be equipped with the system.

An adaptive dynamic balance scheme was implemented and tested on an experimental biped. The control scheme used pre-planned but adaptive motion sequences. CMAC neural networks were responsible for the adaptive control of side-to-side and front-to-back balance, as well as for maintaining good foot contact. Qualitative and quantitative test results show that the biped performance improved with neural network training. The biped is able to start and stop on demand, and to walk with continuous motion on flat surfaces at a rate of up to 100 steps per minute, with up to 6 cm long step.