Robots are becoming more capable and can autonomously perform tasks such as navigating between locations. However, human oversight remains crucial. This study compared two touchless methods for directing mobile robots: voice control and gesture control, to investigate the efficiency of these methods and the preference of users. We tested these methods in two conditions: one in which participants remained stationary and one in which they walked freely alongside the robot. We hypothesized that walking alongside the robot would result in higher intuitiveness ratings and improved task performance, based on the idea that walking promotes spatial alignment and reduces the effort required for mental rotation. In a 2 × 2 within-subject design, 218 participants guided the quadruped robot Spot along a circuitous route with multiple 90 \(^{\circ}\) turns using rotate left, rotate right, and walk forward commands. After each trial, participants rated the intuitiveness of the command mapping, while post-experiment interviews were used to gather the participants’ preferences. Results showed that voice control combined with walking with Spot was the most favored and intuitive, whereas gesture control while standing caused confusion for left/right commands. Nevertheless, 29% of participants preferred gesture control, citing increased task engagement and visual congruence as reasons. An odometry-based analysis revealed that participants often followed behind Spot, particularly in the gesture control condition, when they were allowed to walk. In conclusion, voice control with walking produced the best outcomes. Improving physical ergonomics and adjusting gesture types could make gesture control more effective.

Situation awareness (SA) is critical to improving takeover performance during the transition period from automated driving to manual driving. Although many studies measured SA during or after the driving task, few studies have attempted to predict SA in real time in automated driving. In this work, we propose to predict SA during the takeover transition period in conditionally automated driving using eye-tracking and self-reported data. First, a tree ensemble machine learning model, named LightGBM (Light Gradient Boosting Machine), was used to predict SA. Second, in order to understand what factors influenced SA and how, SHAP (SHapley Additive exPlanations) values of individual predictor variables in the LightGBM model were calculated. These SHAP values explained the prediction model by identifying the most important factors and their effects on SA, which further improved the model performance of LightGBM through feature selection. We standardized SA between 0 and 1 by aggregating three performance measures (i.e., placement, distance, and speed estimation of vehicles with regard to the ego-vehicle) of SA in recreating simulated driving scenarios, after 33 participants viewed 32 videos with six lengths between 1 and 20 s. Using only eye-tracking data, our proposed model outperformed other selected machine learning models, having a root-mean-squared error (RMSE) of 0.121, a mean absolute error (MAE) of 0.096, and a 0.719 correlation coefficient between the predicted SA and the ground truth. The code is available athttps://github.com/refengchou/Situation-awareness-prediction. Our proposed model provided important implications on how to monitor and predict SA in real time in automated driving using eye-tracking data.

The future urban environment may consist of mixed traffic in which pedestrians interact with automated vehicles (AVs). However, it is still unclear how AVs should communicate their intentions to pedestrians. Augmented reality (AR) technology could transform the future of interactions between pedestrians and AVs by offering targeted and individualized communication. This paper presents nine prototypes of AR concepts for pedestrian-AV interaction that are implemented and demonstrated in a real crossing environment. Each concept was based on expert perspectives and designed using theoretically-informed brainstorming sessions. Prototypes were implemented in Unity MARS and subsequently tested on an unmarked road using a standalone iPad Pro with LiDAR functionality. Despite the limitations of the technology, this paper offers an indication of how future AR systems may support future pedestrian-AV interactions.

Non-verbal communication, such as eye contact between drivers and pedestrians, has been regarded as one way to reduce accident risk. So far, studies have assumed rather than objectively measured the occurrence of eye contact. We address this research gap by developing an eye contact detection method and testing it in an indoor experiment with scripted driver–pedestrian interactions at a pedestrian crossing. Thirty participants acted as a pedestrian either standing on an imaginary curb or crossing an imaginary one-lane road in front of a stationary vehicle with an experimenter in the driver’s seat. In half of the trials, pedestrians were instructed to make eye contact with the driver; in the other half, they were prohibited from doing so. Both parties’ gaze was recorded using eye trackers. An in-vehicle stereo camera recorded the car’s point of view, a head-mounted camera recorded the pedestrian’s point of view, and the location of the driver’s and pedestrian’s eyes was estimated using image recognition. We demonstrate that eye contact can be detected by measuring the angles between the vector joining the estimated location of the driver’s and pedestrian’s eyes, and the pedestrian’s and driver’s instantaneous gaze directions, respectively, and identifying whether these angles fall below a threshold of 4°. We achieved 100% correct classification of the trials involving eye contact and those without eye contact, based on measured eye contact duration. The proposed eye contact detection method may be useful for future research into eye contact.

Future automated vehicles may be equipped with external human-machine interfaces (eHMIs) capable of signaling to pedestrians whether or not they can cross the road. There is currently no consensus on the correct colors for eHMIs. Industry and academia have already proposed a variety of eHMI colors, including red and green, as well as colors that are said to be neutral, such as cyan. A confusion that can arise with red and green is whether the color refers to the pedestrian (egocentric perspective) or the automated vehicle (allocentric perspective). We conducted two crowdsourcing experiments (N = 2000 each) with images depicting an automated vehicle equipped with an eHMI in the form of a rectangular display on the front bumper. The eHMI had one out of 729 colors from the RGB spectrum. In Experiment 1, participants rated the intuitiveness of a random subset of 100 of these eHMIs for signaling `please cross the road', and in Experiment 2 for `please do NOT cross the road'. The results showed that for `please cross', colors close to pure green were considered the most intuitive. For `please do NOT cross', colors close to pure red were rated as the most intuitive, but with high standard deviations among participants. In addition, some participants rated green colors as intuitive for `please do NOT cross'. Results were consistent for men and women and for colorblind and non-colorblind persons. It is concluded that eHMIs should be green if the eHMI is intended to signal `please cross', but green and red should be avoided if the eHMI is intended to signal `please do NOT cross'. Various neutral colors can be used for that purpose, including cyan, yellow, and purple.

Physiological synchrony (PS) refers to the similarity in physiological responses of two or more individuals and may be an informative source of information in the field of affective computing. Up to now, PS has been assessed using either autonomic measures or neural measures. While in literature multiple physiological channels have already been combined into one composite index for PS assessment, multimodal PS, i.e., using a combination of autonomic and neural channels in a single composite index (‘A-N’ multimodal), has remained unexplored. A-N multimodal PS is promising for the robust detection of emotionally or cognitively relevant events, as both autonomic and neural activity are sensitive to these events. The aim of this study is (i) to review analytic approaches that have been used to combine multiple physiological channels into one composite index for PS, and (ii) to view these approaches in the light of their potential applicability to A-N multimodal PS. A literature search was conducted to find studies assessing PS based on a composite index of multiple autonomic channels or multiple channels in electroencephalographic (EEG) recordings. Four studies were found that assessed PS based on a composite index using multiple autonomic channels and 12 studies assessed PS based on a composite index using multiple EEG channels. We found that analytic approaches varied between studies. Some averaged over multiple channels after assessing PS separately per channel $(N = 4)$, or averaged over channels before assessing PS $(N = 1)$, while others used different linear combinations of channels based on spatio-spectral decomposition $(N = 1)$ or correlated component analysis $(CCA, N = 8)$. CCA finds linear combinations of channels that are maximally correlated between subjects and has up to now been used to assess neural PS. We suggest that this method may be most appropriate for the exploration of multimodal PS assessment.

This paper assessed four types of human-machine interfaces (HMIs), classified according to the stages of automation proposed by Parasuraman et al. [“A model for types and levels of human interaction with automation,” IEEE Trans. Syst. Man, Cybern. A, Syst. Humans, vol. 30, no. 3, pp. 286-297, May 2000]. We hypothesized that drivers would implement decisions (lane changing or braking) faster and more correctly when receiving support at a higher automation stage during transitions from conditionally automated driving to manual driving. In total, 25 participants with a mean age of 25.7 years (range 19-36 years) drove four trials in a driving simulator, experiencing four HMIs having the following different stages of automation: baseline (information acquisition-low), sphere (information acquisition-high), carpet (information analysis), and arrow (decision selection), presented as visual overlays on the surroundings. The HMIs provided information during two scenarios, namely a lane change and a braking scenario. Results showed that the HMIs did not significantly affect the drivers' initial reaction to the take-over request. Improvements were found, however, in the decision-making process: When drivers experienced the carpet or arrow interface, an improvement in correct decisions (i.e., to brake or change lane) occurred. It is concluded that visual HMIs can assist drivers in making a correct braking or lane change maneuver in a take-over scenario. Future research could be directed toward misuse, disuse, errors of omission, and errors of commission.

Shared control is an increasingly popular approach to facilitate control and communication between humans and intelligent machines. However, there is little consensus in guidelines for design and evaluation of shared control, or even in a definition of what constitutes shared control. This lack of consensus complicates cross fertilization of shared control research between different application domains. This paper provides a definition for shared control in context with previous definitions, and a set of general axioms for design and evaluation of shared control solutions. The utility of the definition and axioms are demonstrated by applying them to four application domains: automotive, robot-assisted surgery, brain-machine interfaces, and learning. Literature is discussed for each of these four domains in light of the proposed definition and axioms. Finally, to facilitate design choices for other applications, we propose a hierarchical framework for shared control that links the shared control literature with traded control, co-operative control, and other human-automation interaction methods. Future work should reveal the generalizability and utility of the proposed shared control framework in designing useful, safe, and comfortable interaction between humans and intelligent machines.

In pioneering work, Senders (1983) tasked five participants to watch a bank of six dials, and found that glance rates and times glanced at dials increase linearly as a function of the frequency bandwidth of the dial's pointer. Senders did not record the angle of the pointers synchronously with eye movements, and so could not assess participants' visual sampling behavior in regard to the pointer state. Because the study of Senders has been influential but never repeated, we replicated and extended it by assessing the relationship between visual sampling and pointer state, using modern eye-tracking equipment. Eye tracking was performed with 86 participants who watched seven 90-second videos, each video showing six dials with moving pointers. Participants had to press the spacebar when any of the six pointers crossed a threshold. Our results showed a close resemblance to Senders' original results. Additionally, we found that participants did not behave in accordance with a periodic sampling model, but rather were conditional samplers, in that the probability of looking at a dial was contingent on pointer angle and velocity. Finally, we found that participants sampled more in agreement with Nyquist sampling when the high bandwidth dials were placed in the middle of the bank rather than at its outer edges. We observed results consistent with the saliency, effort, expectancy, and value model and conclude that human sampling of multidegree of freedom systems should not only be modeled in terms of bandwidth but also in terms of saliency and effort.

The interaction between a human driver and an automated driving system may improve when the automation is designed in such a way that it behaves in a human-like manner. This paper introduces a human-like steering model, in which the driver adapts to the risk due to uncertainty in the environment. Current steering models take a risk-neutral approach, while the fields of economics and sensorimotor control suggest that humans exhibit risk-sensitive behavior. The proposed model uses a risk-sensitive optimal feedback control structure to predict steering behavior. The paper studies the effect of the risk-sensitivity parameter and compares the prediction of the risk-neutral and risk-sensitive controllers in a simulated abstraction of two scenarios: (a) driving while being subjected to lateral wind gusts and (b) overtaking an unpredictably swerving car. The simulation results show that the risk-sensitive model adapts to the uncertainty in the environment. Experimental data will be needed to validate the predictions of our model.

Conditionally automated driving systems may soon be available on the market. Even though these systems exempt drivers from the driving task for extended periods of time, drivers are expected to take back control when the automation issues a so-called take-over request. This study investigated the interaction between take-over request modality and type of non-driving task, regarding the driver's reaction time. It was hypothesized that reaction times are higher when the non-driving task and the take-over request use the same modality. For example, auditory take-over requests were expected to be relatively ineffective in situations in which the driver is making a phone call. 101 participants, divided into three groups, performed one of three non-driving tasks, namely reading (i.e., visual task), calling (auditory task), or watching a video (visual/auditory task). Results showed that auditory and tactile take-over requests yielded overall faster reactions than visual take-over requests. The expected interaction between takeover modality and the dominant modality of the non-driving task was not found. As for self-reported usefulness, auditory and tactile take-over requests yielded higher scores than visual ones. In conclusion, it seems that auditory and tactile stimuli are equally effective as take-over requests, regardless of the non-driving task. Further study into the effects of realistic non-driving tasks is needed to identify which non-driving tasks are detrimental to safety in automated driving.

Head-mounted displays (HMDs) offer immersion and binocular disparity. This study investigated whether an HMD yields better object-alignment performance than a conventional monitor in virtual environments that are rich in pictorial depth cues. To determine the effects of immersion and disparity separately, three hardware setups were compared: 1) a conventional computer monitor, yielding low immersion, 2) an HMD with binocular-vision settings (HMD stereo), and 3) an HMD with the same image presented to both eyes (HMD mono). Two virtual environments were used: a street environment in which two cars had to be aligned (target distance of about 15 m) and an office environment in which two books had to be aligned (target distance of about 0.7 m, at which binocular depth cues were expected to be important). Twenty males (mean age = 21.2, SD age = 1.6) each completed 10 object-alignment trials for each of the six conditions. The results revealed no statistically significant differences in object-alignment performance between the three hardware setups. A self-report questionnaire showed that participants felt more involved in the virtual environment and experienced more oculomotor discomfort with the HMD than with the monitor.

Automated driving vehicles of the future will most likely include multiple modes and levels of operation and thus include various transitions of control (ToC) between human and machine. Traditional activation devices (e.g., knobs, switches, buttons, and touchscreens) may be confused by operators among other system setting manipulators and also susceptible to inappropriate usage. Non-intrusive eye-tracking measures may assess driver states (i.e., distraction, drowsiness, and cognitive overload) automatically to trigger manual-to-automation ToC and serve as a driver readiness verification during automation-to-manual ToC. Our integrated driver state monitor is overviewed here within the scope of this brief system description/demonstration paper. It combines gaze position, gaze variability, eyelid opening, as well as external environmental complexity from the driving scene to facilitate ToC in automated driving. As both driver facing and forward facing cameras become increasingly commonplace and even legally mandated within various automated driving vehicles, our integrated system helps inform relevant future research and development towards improved human-computer interaction and driving safety.

The task of car driving is automated to an ever greater extent. In the foreseeable future, drivers will no longer be required to touch the steering wheel and pedals and could engage in non-driving tasks such as working or resting. Vibrotactile displays have the potential to grab the attention of the driver when the automation reaches its functional limits and the driver has to take over control. The aim of the present literature survey is to outline the key physiological and psychophysical aspects of vibrotactile sensation and to provide recommendations and relevant research questions regarding the use of vibrotactile displays for taking over control from an automated vehicle. Results showed that a distinction can be made between four dimensions for coding vibrotactile information (amplitude, frequency, timing, and location), each of which can be static or dynamic. There is a consensus that frequency and amplitude are less suitable for coding information than location and timing. Vibrotactile stimuli have been shown to be effective as simple warnings. However, vibrations can evoke annoyance, and providing vibrations in close spatial-temporal proximity might cause a lack of comprehension of the signal. We describe the sequential stages of a take-over process and argue that vibrotactile displays are a promising candidate for redirecting the attention of a distracted driver. Furthermore, vibrotactile displays hold potential for supporting cognitive processing and action selection while resuming control of an automated vehicle. Finally, we argue that multimodal feedback should be used to assist the driver in the take-over process.

This paper presents a driving simulator experiment, which evaluates a road-departure prevention (RDP) system in an emergency situation. Two levels of automation are evaluated: 1) haptic feedback (HF) where the RDP provides advisory steering torque such that the human and the machine carry out the maneuver cooperatively, and 2) drive by wire (DBW) where the RDP automatically corrects the front-wheels angle, overriding the steering-wheel input provided by the human. Thirty participants are instructed to avoid a pylon-confined area while keeping the vehicle on the road. The results show that HF has a significant impact on the measured steering wheel torque, but no significant effect on steering-wheel angle or vehicle path. DBW prevents road departure and tends to reduce self-reported workload, but leads to inadvertent human-initiated steering resulting in pylon collisions. It is concluded that a low level of automation, in the form of HF, does not prevent road departures in an emergency situation. A high level of automation, on the other hand, is effective in preventing road departures. However, more research may have to be done on the human response while driving with systems that alter the relationship between steering-wheel angle and front-wheels angle.

A car-following assisting system named the rear window notification display (RWND) was developed, with the aim of improving a driver's manual car-following performance. The RWND presented lead-car acceleration and time headway (THW) (i.e., intervehicle distance divided by the speed of the following car) on the rear window of a lead car, which was driven automatically. A simulator-based experiment with 22 participants showed that the RWND reduced both the mean and standard deviation of THW but did not increase the occurrence of potentially unsafe headways of less than 1 s. The parameter estimation of a common linear car-following model showed that drivers accomplished the performance improvements by adopting higher control gains with respect to intervehicle distance, relative speed, and acceleration. A postexperiment questionnaire revealed that the display was generally not regarded as a distraction nor did participants think that it provided too much information, with means of 4.0 and 2.9, respectively, on a scale from one (completely disagree) to ten (completely agree). The results of this study suggest that the RWND can be used along with Cooperative Adaptive Cruise Control to increase traffic flow without degrading safety.

Racecar drivers could benefit from new training methods for learning to drive fast lap times. Inspired by the learning-from-errors principle, this simulator-based study investigated the effect of the tire-road friction coefficient on the training effectiveness of a car racing task. Three groups of 15 inexperienced racecar drivers (low grip (LG), 66% of normal grip; normal grip (NG); high grip (HG), 150% of normal grip) completed four practice sessions of 10 minutes in a Formula 3 car on an oval track of 800 m. After the practice sessions, two retention sessions followed: a retention session with normal grip in a Formula 3 car and another retention session with a Formula 1 car. The results showed that LG was significantly slower than HG in the first retention session. Furthermore, LG reported a higher confidence and lower frustration than NG and HG after each of the two retention sessions. In conclusion, practicing with low grip, as compared to practicing with normal or high grip, resulted in increased confidence but slower lap times.

A driving simulator experiment is presented investigating different road departure prevention (RDP) setups. To induce the risk of road departure, thirty test drivers were asked to avoid a pylon-confined area (obstacle) while keeping the vehicle within the road limits. The RDP system intervened by applying a haptic-feedback (i.e., haptic shared control) and/or correcting the steering angle (i.e., drive-by-wire (DBW) input-mixing shared control) in the event that a vehicle road departure was likely to occur. The system that determines the correcting steering input is a RDP controller based on the driver's inputs. The results showed that DBW effectively helped drivers to stay within road limits and reduced workload. The haptic shared control had a significant influence on the measured steering torque, but limited effect on the steering wheel angle and the vehicle path. The DBW system resulted in drivers making counter-corrections demoting their performance. In conclusion, shared control for RDP is effective, although more research needs to be conducted regarding the human response in situations where the relationship between the steering wheel angle and the front wheels' steering angle is altered while driving.

Shared control (also known as continuous haptic guidance or haptically active controls) has recently been introduced in car driving. With shared control, the driver receives continuous force feedback on the gas pedal or steering wheel, so that human and machine conduct the driving task simultaneously. Experiments in driving simulators have shown that shared control reduces control variability and mental workload, and improves accuracy in path tracking and car following. Crucial to road safety, however, is not whether shared control improves performance in routine driving tasks, but what happens in dangerous situations when a conflict of authority occurs, or when the force feedback cannot be relied upon or is suddenly disengaged. Drawing on research into transfer of training, it is shown that shared control may induce aftereffects and may hamper retention of robust driving skills. Supplementary information should not be provided continuously, but on an as-needed basis, warning or assisting drivers only when deviations from acceptable tolerance limits arise.

This paper aims to provide recommendations for improving the effectiveness of automatic, student-adaptive, simulation-based driver training. Using experiments and recorded data in driving simulators, three distinct issues are discussed: 1) the student, 2) the virtual driving instructor (VDI), and 3) the student-profile. We found that: first, students seek task-relevant information themselves; not providing them with feedback can be beneficial. Second, an intelligent VDI that emulates a human driving instructor is not favored. To the contrary, regressive instruction -- a relatively simple principle -- was effective in letting students drive away autonomously. Third, constructing a student-profile based on individual characteristics, such as a strength-weakness report, is viable for providing student-adaptive feedback.

Driver assistance systems that supply force feedback (FF) on the accelerator commonly use relative distance and velocity with respect to the closest lead vehicle in front of the own vehicle. This 1-D feedback might not accurately represent the situation and can cause unwanted step-shaped changes in the FFs during lateral maneuvers. To address these shortcomings, a 2-D system is proposed that calculates FF using a weighted average of the influences of lead vehicles. Offline simulations and an experiment in a driving simulator were performed to compare no feedback, 1-D systems, and the novel 2-D system during a car-following task with cut-in maneuvers. Results show that the 2-D feedback resulted in lower mean forces, lower response times to cut-in vehicles, and favorable subjective experiences as compared to the 1-D systems.