Interacting with in-vehicle infotainment screens (IVIS) while driving influences how drivers allocate attention to the road, which can affect task and driving proficiency. Haptic feedback in IVIs has emerged as a promising avenue to support drivers’ interactions. Little is known about the impact of diverse feedback modalities on different types of interactions commonly carried out when interacting with IVIS. Two studies were conducted utilising production vehicles to understand the effects of IVIS featuring haptic feedback modalities (vibration and force touch) compared to other modalities (audio-only and no haptic feedback) for different types of interactions. Findings indicate that vibration and audio-only feedback supported drivers’ attention to the road and increased task performance in most types of interactions. Conversely, force touch did not support the driver and yielded the same results as having no haptic feedback. These results can support the design and enhancement of IVIS systems to support the drivers.

Haptic feedback can improve safety and driving behaviour. While vibration has been widely studied, other haptic modalities have been neglected. To address this, we present two studies investigating the use of uni- and bimodal vibrotactile and thermal cues on the steering wheel. First, notifications with three levels of urgency were subjectively rated and then identified during simulated driving. Bimodal feedback showed an increased identification time over unimodal vibrotactile cues. Thermal feedback was consistently rated less urgent, showing its suitability for less time critical notifications, where vibration would be unnecessarily attention-grabbing. The second study investigated more complex thermal and bimodal haptic notifications comprised of two different types of information (Nature and Importance of incoming message). Results showed that both modalities could be identified with high recognition rates of up to 92% for both and up to 99% for a single type, opening up a novel design space for haptic in-car feedback.

The visual demand of finger-touch based interactions with touch screens has been increasingly modelled using Fitts' Law. With respect to driving, these models facilitate the prediction of mean glance duration and total glance time with an index of difficulty based on target size and location. Strong relationships between measures have been found in the controlled conditions of driving simulators. The present study aimed to validate such models in naturalistic conditions. Nineteen experienced drivers carried out a range of touchscreen button-press tasks in an instrumented car on a UK motorway. In contrast with previous simulator-based work, our on-road data produced much weaker relationships between the index of difficulty and glance times. The model improved by focusing on tasks that required one glance only. Limitations of Fitts' Law in the more complex and dynamic real-world driving environment are discussed, as are the potential drawbacks of driving simulators for conducting visual demand research.

Haptic feedback is used in cars to reduce visual inattention. While tactile feedback like vibration can be influenced by the car's movement, thermal and cutaneous push feedback should be independent of such interference. This paper presents two driving simulator studies investigating novel tactile feedback on the steering wheel for navigation. First, devices on one side of the steering wheel were warmed, indicating the turning direction, while those on the other side were cooled. This thermal feedback was compared to audio. The thermal navigation lead to 94.2% correct recognitions of warnings 200m before the turn and to 91.7% correct turns. Speech had perfect recognition for both. In the second experiment, only the destination side was indicated thermally, and this design was compared to cutaneous push feedback. The simplified thermal feedback design did not increase recognition, but cutaneous push feedback had high recognition rates (100% for 200 m warnings, 98% for turns).

Understanding how best to present information inside a semi-automated vehicle is a prevalent challenge in HMI design. There is an understanding that a driver's trust and previous driving experience can affect the information they require inside a semi-automated vehicle. However, to date little is known about how these predispositions specifically affect the types of information that should be presented and importantly, how this changes with increased exposure to an automated system. In this paper, seventeen participants experienced twenty-six minutes of an automated driving simulation once every day for a week. The information to display was carefully chosen in accordance with the Skills, Rules, Knowledge model. The information was synchronized to the driving simulation and presented on a tablet in the driving simulator. Eye tracking was used to measure the information looked at. The results showed that trust increased significantly with increased exposure, but this had no correlation to any specific piece of information viewed. Drivers who were more prone to making lapses or errors (as measured by the Driver Behavior Questionnaire) tended towards using information that was less cognitively demanding. Finally, a driver's propensity to making lapses was found to be a potential early predictor of trust, but this became less accurate with increased exposure to the semi-automated vehicle.

Predictive touch is an HMI technology that relies on inferring, early in the pointing gesture, the interface item a driver or passenger intends to select on an in-vehicle display [1, 2]. It simplifies and expedites the selection task, thereby reducing the associated interaction effort. This paper presents two studies on drivers using predictive touch and focuses on evaluating the best means to facilitate selecting the intended on-display item. This includes immediate midair selection with the system autonomously auto-selecting the predicted interface component, hover/dwell and drivers pressing a button on the steering wheel to execute the selection action. These were arrived at in an expert workshop study with twelve participants. The results of the subsequent evaluation study with twenty four participants demonstrate, using quantitative and qualitative measures, that immediate mid-air selection is a promising assistive scheme, where drivers need not touch a physical surface to select interface components, thus touch-free control.

This paper reports on two studies investigating how following a lead vehicle could act as a metaphor for an Augmented Reality (AR) system to support navigation tasks. For the first formative study, 34 participants completed a video-based evaluation of the role of a real lead vehicle when navigating a coherent journey. Verbal protocols indicated that a lead vehicle may be a valuable navigation aid at a range of different junction types, but not where drivers may desire a preview of upcoming steps or their overall orientation. A subsequent driving simulator study with 22 participants examined whether an AR lead vehicle may support drivers when navigating at complex junctions, specifically large multi-exit roundabouts. The virtual car led to good navigation and driving performance, which was comparable to a more traditional screen-fixed interface. Overall, this work demonstrates that a virtual lead vehicle may be beneficial within AR navigation devices.

Haptic feedback has been widely studied for in-car interactions. However, most of this research has used vibrotactile cues. This paper presents two studies that examine novel thermal feedback for navigation during simulated driving for a lane change task. In the first, we compare the distraction and time differences of audio and thermal feedback. The results show that the presentation of thermal stimuli does not increase lane deviation, but the time needed to complete a lane change increased by 1.82 seconds. In the second study, the influence of variable changes of thermal stimuli on the lane change task performance was tested. We found that the same stimulus design for warm and cold temperatures does not always elicit the same results. Furthermore, variable alterations can have different effects on specified tasks. This suggests that the design of thermal stimuli is highly dependent on what task result should be maximized.

Driver fatigue is one of the main causes of road accidents. It is essential to develop a reliable driver drowsiness detection system which can alert drivers without disturbing them and is robust to environmental changes. This paper explores yawning behaviour as a sign of drowsiness in spontaneous expressions of drowsy drivers in simulated driving scenarios. We analyse a labelled dataset of videos of sleep-deprived versus alert drivers and demonstrate the correlation between hand-over-face touches, face occlusions and yawning. We propose that face touches can be used as a novel cue in automated drowsiness detection alongside yawning and eye behaviour. Moreover, we present an automatic approach to detect yawning based on extracting geometric and appearance features of both mouth and eye regions. Our approach successfully detects both hand-covered and uncovered yawns with an accuracy of 95%. Ultimately, our goal is to use these results in designing a hybrid drowsiness-detection system.

This paper presents formative research to inform the design of intelligent automotive user interfaces. It describes an evaluation of dialogue-based interfaces, mediating the driver to take back control from the autonomous mode of a car. Four concepts designed to increase driver Situation Awareness were evaluated in a driving simulator. They used dialogue-based interaction, where driving-related information was either asked from or repeated by the driver, with the alternative of a countdown-based interface with no additional information. An inclusive set of participants, with a wide age spectrum, tested the interfaces. The shorter and simpler interaction of the countdown timer was most accepted. The interface seeking answers to driving-related questions came next, and the interface requiring repetition of driving-related information, even when augmented by visual and tactile cues, was least accepted. Design guidelines on utilizing dialogue as a means of keeping the driver in the loop during a takeover were thus derived.

Autonomous driving presents an exciting new development in vehicle technology. It poses a new challenge in driver-automation collaboration particularly during handover transitions between human and machine. In order to deal with this problem, this paper proposes a novel control framework for the haptic take-over system. The high-level framework of the haptic take-over control system, which takes driver cognitive workload, neuromuscular dynamics and optimal trajectory planning into consideration, is developed. Under the proposed framework, the determination approach of the optimal input sequence is introduced. The model of the allowed driver take-over authority, which is associated with driver's cognitive workload, as well as muscle readiness during take- over, is investigated and developed. The haptic feedback torque controller is then designed so as to minimize the deviation between the allowed control authority and driver's current degree of participation. A handover process, along with the proposed take-over control method, is also simulated. The simulation results validate the feasibility and effectiveness of the proposed approach.

Touchscreen human–machine interfaces (HMIs) are commonly employed as the primary control interface and touch-point of vehicles. However, there has been very little theoretical work to model the demand associated with such devices in the automotive domain. Instead, touchscreen HMIs intended for deployment within vehicles tend to undergo time-consuming and expensive empirical testing and user trials, typically requiring fully functioning prototypes, test rigs, and extensive experimental protocols. While such testing is invaluable and must remain within the normal design/development cycle, there are clear benefits, both fiscal and practical, to the theoretical modeling of human performance. We describe the development of a preliminary model of human performance that makes a priori predictions of the visual demand (total glance time, number of glances, and mean glance duration) elicited by in-vehicle touchscreen HMI designs, when used concurrently with driving. The model incorporates information theoretic components based on Hick–Hyman Law decision/search time and Fitts’ Law pointing time and considers anticipation afforded by structuring and repeated exposure to an interface. Encouraging validation results, obtained by applying the model to a real-world prototype touchscreen HMI, suggest that it may provide an effective design and evaluation tool, capable of making valuable predictions regarding the limits of visual demand/performance associated with in-vehicle HMIs, much earlier in the design cycle than traditional design evaluation techniques. Further validation work is required to explore the behavior associated with more complex tasks requiring multiple screen interactions, as well as other HMI design elements and interaction techniques. Results are discussed in the context of facilitating the design of in-vehicle touchscreen HMI to minimize visual demand.

This paper presents a multimodal approach to in-vehicle classification of driver glances. Driver glance is a strong predictor of cognitive load and is a useful input to many applications in the automotive domain. Six descriptive glance regions are defined and a classifier is trained on video recordings of drivers from a single low-cost camera. Visual features such as head orientation, eye gaze and confidence ratings are extracted, then statistical methods are used to perform failure analysis and calibration on the visual features. Non-visual features such as steering wheel angle and indicator position are extracted from a RaceLogic VBOX system. The approach is evaluated on a dataset containing multiple 60 second samples from 14 participants recorded while driving in a natural environment. We compare our multimodal approach to separate unimodal approaches using both Support Vector Machine (SVM) and Random Forests (RF) classifiers. RF Mean Decrease in Gini Index is used to rank selected features which gives insight into the selected features and improves the classifier performance. We demonstrate that our multimodal approach yields significantly higher results than unimodal approaches. The final model achieves an average F1score of 70.5% across the six classes.

In order to develop an advanced haptic take-over system for highly automated vehicles, research into the driver's neuromuscular dynamics is needed. In this paper a dynamic model of drivers' neuromuscular interaction with a steering wheel is firstly established. The transfer function and the natural frequency of the systems are analysed. In order to identify the key parameters of the driver-steering-wheel coupled system and investigate the system properties under different situations, experiments with drive-in-the-loop are carried out. For each test subject, two steering tasks, namely the passive and active steering tasks, are instructed to be completed. Furthermore, during the experiments, subjects manipulated the steering wheel with two distinct postures and three different hand positions. Based on the test results, key parameters of the transfer function and system properties are identified and investigated. The data and characteristics of the driver neuromuscular system are discussed and compared with respect to different steering tasks, hand positions and driver postures. These test results identified system properties that provide a good foundation for the development of a haptic take-over control system for automated vehicles.

In this paper, we first give an overview of the predictive display concept, which aims to minimise the demand associated with interacting with in-vehicle displays, such as touchscreens, via free hand pointing gestures. It determines the item the user intends to select, early in the pointing gesture, and accordingly simplifies-expedites the target acquisition. A study to evaluate the impact of using a predictive touchscreen in a car is then presented. The mid-air selection pointing facilitation scheme is applied, such that the user does not have to physically touch the interactive surface. Instead, the predictive display auto-selects the predicted interface icon on behalf of the user, once the required level of inference certainty is achieved. The study results, which are based on data collected from 20 participants under various driving-road conditions, demonstrate that a predictive display can significantly reduce the workload, effort and durations of completing on-screen selection tasks in vehicles.

This paper explores how navigational guidance which references the world around the driver influences their experience of workload and environmental engagement on multiple journeys. In an on-road study thirty participants drove following three intersecting routes in the city of Nottingham, UK on two separate occasions, one week apart. In a between subjects design, participants followed navigational guidance from either: an existing navigation system (SatNav); or the semi-structured wayfinding prompts of an informed passenger. Results indicated that drivers following the navigational guidance of the informed passenger benefitted from lower workload overall which also reduced significantly on their second drive. Additionally, these drivers made significantly fewer errors on route and demonstrated superior survey knowledge compared to their SatNav counterparts. This evidence supports the case for enhanced route directions in future navigation devices, referencing the driver's environment in a manner that can reduce workload and navigation errors, whilst assisting in spatial knowledge development.

Given the proliferation of digital assistants in everyday mobile technology, it appears inevitable that next generation vehicles will be embodied by similar agents, offering engaging, natural language interactions. However, speech can be cognitively captivating. It is therefore important to understand the demand that such interfaces may place on drivers. Twenty-five participants undertook four drives (counterbalanced), in a medium-fidelity driving simulator: 1. Interacting with a state-of-the-art digital driving assistant ('DDA') (presented using Wizard-of-Oz); 2. Engaged in a hands-free mobile phone conversation; 3. Undertaking the delayed-digit recall ('2-back') task and 4. With no secondary task (baseline). Physiological arousal, subjective workload assessment, tactile detection task (TDT) and driving performance measures consistently revealed the '2-back' drive as the most cognitively demanding (highest workload, poorest TDT performance). Mobile phone and DDA conditions were largely equivalent, attracting low/medium cognitive workload. Findings are discussed in the context of designing in-vehicle natural language interfaces to mitigate cognitive demand.

Touchscreen Human-Machine Interfaces (HMIs) inherently demand some visual attention. By employing a secondary device, to work in unison with a touchscreen, some of this demand may be alleviated. In a medium-fidelity driving simulator, twenty-four drivers completed four typical in-vehicle tasks, utilising each of four devices -- touchscreen, rotary controller, steering wheel controls and touchpad (counterbalanced). Participants were then able to combine devices during a final 'free-choice' drive. Visual behaviour, driving/task performance and subjective ratings (workload, emotional response, preferences), indicated that in isolation the touchscreen was the most preferred/least demanding to use. In contrast, the touchpad was least preferred/most demanding, whereas the rotary controller and steering wheel controls were largely comparable across most measures. When provided with 'free-choice', the rotary controller and steering wheel controls presented as the most popular candidates, although this was task-dependent. Further work is required to explore these devices in greater depth and during extended periods of testing.

Current navigational systems rarely consider generic road landmarks in their navigation instructions, which can lead to mistakes, frustration, and distraction. However, automatic detection of road landmarks is difficult, as current approaches to object detection focus either on out-of-context objects which have special characteristics or on very specific domains. This work presents a future direction for a user-friendly navigational system based on state-of-the-art computer vision techniques that use deep learning for object detection. We propose an automatic hierarchical approach for detecting and classifying a set of static and dynamic road landmarks that would be useful in automatic navigational systems. We further demonstrate a set of optimisations that improve performance and accuracy of the basic system. We evaluate our approach on a natural, 'in-the-wild' dataset to determine how well it handles natural automotive input. Finally, we demonstrate a use-case for our system that extracts information about a vehicle's location and intention.

Using interactive displays, such as a touchscreen, in vehicles typically requires dedicating a considerable amount of visual as well as cognitive capacity and undertaking a hand pointing gesture to select the intended item on the interface. This can act as a distractor from the primary task of driving and consequently can have serious safety implications. Due to road and driving conditions, the user input can also be highly perturbed resulting in erroneous selections compromising the system usability. In this paper, we propose intent-aware displays that utilize a pointing gesture tracker in conjunction with suitable Bayesian destination inference algorithms to determine the item the user intends to select, which can be achieved with high confidence remarkably early in the pointing gesture. This can drastically reduce the time and effort required to successfully complete an in-vehicle selection task. In the proposed probabilistic inference framework, the likelihood of all the nominal destinations is sequentially calculated by modeling the hand pointing gesture movements as a destination-reverting process. This leads to a Kalman filter-type implementation of the prediction routine that requires minimal parameter training and has low computational burden; it is also amenable to parallelization. The substantial gains obtained using an intent-aware display are demonstrated using data collected in an instrumented vehicle driven under various road conditions.

With the proliferation of the touchscreen technology, interactive displays are becoming an integrated part of the modern vehicle environment. However, due to road and driving conditions, the user input on such displays can be perturbed resulting in erroneous selections. This paper describes an evaluative study of the usability and input performance of in-vehicle touchscreens. The analysis is based on data collected in instrumented cars driven under various road/driving conditions. We assess the frequency of failed selection attempts, distances by which users miss the intended on-screen target and the durations of undertaken free hand pointing gestures to accomplish the selection tasks. It is shown that the road/driving conditions can notably undermine the usability of an interactive display when the user input is perturbed, e.g. due to the experienced vibrations and lateral accelerations in the vehicle. The distance between the location of an erroneous on-screen selection and the intended endpoint on the display, is closely related to the level of present in-vehicle noise. The conducted study can advise graphical user interfaces design for the vehicle environment where the user free hand pointing gestures can be subject to varying levels of perturbations.

Finger-touch based interactions with capacitive touchscreen devices in cars are becoming increasingly common. As such, it is critical to understand the basic human factors of target acquisition (pointing/touching) in this context. We describe a simulator study that aims to build Fitts' Law relationships for predicting the visual demands (mean glance duration and total glance time) associated with finger-touch pointing tasks as a function of target size, location and design. The observed data show strong linear relationships between all visual demand measures and Fitts' index of difficulty, indicating that finger-touch pointing tasks conform well with the Fitts' Law model under conditions of divided attention. The derived equations are discussed in the context of designing in-vehicle touchscreen interfaces for minimal visual demand.

Interactive displays are becoming an integrated part of the modern vehicle environment. Their use typically entails dedicating a considerable amount of attention and undertaking a pointing gesture to select an interface item/icon displayed on a touchscreen. This can have serious safety implications for the driver. The pointing gesture can also be highly perturbed due to the road and driving conditions, resulting in erroneous selections. In this paper, we propose a probabilistic intent prediction approach that facilitates establishing the targeted icon on the interface early in the pointing gesture. It employs a 3D vision sensory device to continuously track the pointing hand/finger in conjunction with suitable Bayesian prediction algorithms. The introduced technique can significantly reduce the pointing task completion time, the necessary associated visual, cognitive and movement efforts as well as enhance the selection accuracy. The substantial furnished gains and the pointing gesture characteristics are demonstrated using data collected in an instrumented vehicle.

Touchscreen interfaces are increasingly used on a daily basis in both mobile devices and in cars. The majority of vehicles use resistive touchscreens which, while reliable and inexpensive, may not perform as well as alternative touchscreen technologies. A simulator-based user-centred study was conducted to compare the User Experience of resistive touchscreens against capacitive and infra-red variants in a range of automotive use cases. This paper details an initial treatment of the data focusing on touchscreen task performance and subjective usability measures. Findings identified that the resistive display was clearly least preferred, with capacitive offering the best overall performance.

Emerging applications using body sensor networks (BSNs) constitute a new trend in car safety. However, the integration of heterogeneous body sensors with vehicular ad hoc networks (VANETs) poses a challenge, particularly on the detection of human behavioral states that may impair driving. This paper proposes a detector of human emotions, of which tiredness and stress (tension) could be related to traffic accidents. We present an exploratory study demonstrating the feasibility of detecting one emotional state in real time using a BSN. Based on these results, we propose middleware architecture that is able to detect emotions, which can be communicated via the onboard unit of a vehicle with city emergency services, VANETs, and roadside units, aimed at improving the driver's experience and at guaranteeing better security measures for the car driver.

Touch screens are increasingly used within modern vehicles, providing the potential for a range of gestures to facilitate interaction under divided attention conditions. This paper describes a study aiming to understand how drivers naturally make swipe gestures in a vehicle context when compared with a stationary setting. Twenty experienced drivers were requested to undertake a swipe gesture on a touch screen in a manner they felt was appropriate to execute a wide range of activate/deactivate, increase/decrease and next/previous tasks. All participants undertook the tasks when either driving within a right-hand drive, medium-fidelity simulator or whilst sitting stationary. Consensus emerged in the direction of swipes made for a relatively small number of increase/decrease and next/previous tasks, particularly related to playing music. The physical action of a swipe made in different directions was found to affect the length and speed of the gesture. Finally, swipes were typically made more slowly in the driving situation, reflecting the reduced resources available in this context and/or the handedness of the participants. Conclusions are drawn regarding the future design of swipe gestures for interacting with in-vehicle touch screens.