Touch is a central aspect of social human-human interaction, yet when designing social human-robot interaction, the experience of touch is very often neglected. This work explores whether a robot should feel alive by investigating the effect of life-like haptic stimuli inspired by bio-physiological signals (heartbeat, purring, and thermal feedback) on user perception. The results of an interaction study (n=42) showed that haptic cues which simulate bio-physiological signals significantly increased perceived anthropomorphism, social warmth, and likeability of the PARO robot compared to no additional haptics or abstract vibration. We discuss how multimodal haptic cues, despite increasing perceived eeriness, also increased perceived likeability of the robot, providing novel insights into whether the uncanny valley effect may extend into non-visual modalities. These results provide the first empirical evidence that life-like tactile cues can enhance user perceptions of social robots, offering design guidance for deploying effective and engaging touch interactions between humans and zoomorphic robots.

Conditionally automated vehicles require drivers to remain hazard-aware while disengaged from driving and engaged in Non-Driving Related Tasks (NDRTs) Conventional visual warning interfaces do not fully address this attentional conflict. While psychological research demonstrates that social cues are processed more efficiently than visual information, this has never been evaluated for safety–critical warnings in conditionally automated driving. This article presents the first empirical investigation into using a virtual agent to indicate danger and aid situational awareness while driver performs an NDRT. Across two experiments (n = 48), hazard prediction ability was assessed with either a visual cue or a virtual agent cueing danger. When including an attention-capturing element, both visual and social cue types mitigated driver distraction, maintaining hazard prediction comparable with undistracted driving. These findings suggest that virtual agents can be as effective as conventional visual cues in aiding situational awareness in safety–critical automated driving contexts, offering design insights for in-vehicle attentional displays supporting driver awareness.

Mitigating motion sickness with non-invasive cortical vestibular brain stimulation (vBS) is promising because it does not require redesigning VR content or altering the motion profile of a moving platform. Instead, it aims to enhance individuals' intrinsic resistance to motion sickness. However, the mechanism underlying its online efficacy remains unclear (Here, "online efficacy" means the observed motion sickness reduction during vBS not before or after vBS). Motivated by lower motion sickness susceptibility observed in congenital nystagmus patients, we hypothesized that reductions in motion sickness under vBS condition are associated with increased eye movement instability. Leveraging sham-controlled, sex-balanced eye-tracking data from an open-access database, this study presents the first evaluation of a 20 Hz-vBS approach targeting the vestibular cortices as a countermeasure to motion sickness, using spatiotemporal eye movement metrics derived from eye-tracking data collected in VR. Results show that reduced motion sickness under active vBS indeed increased eye movement instability temporarily and spatially, compared to sham vBS. We also found that the temporal metrics was more strongly associated with female participants, whereas the spatial metric was more strongly associated with male participants.

Cyclists need interfaces such as on-vehicle displays or augmented-reality (AR) glasses for effective communication with autonomous vehicles (AVs) when human drivers are no longer present. Interfaces must handle complex situations involving multiple AVs around a cyclist. Holistic AV-Cyclist Interfaces (HACIs) are a novel solution; they group interfaces into a multimodal interconnected system to support the rider. However, the best way to present information is uncertain. We explored this in a scenario with three AVs using CycleARcade, a new multi-user AR platform for designing and evaluating HACIs. Cyclists and HCI researchers collaboratively created and tested HACIs within CycleARcade through a novel iterative participatory design method. We synthesised three HACIs from this process and assessed them with riders in CycleARcade. Participants preferred HACIs with AR displays integrated into the environment to avoid road distractions, paired with spatial audio communicating AV proximity. These findings provide crucial input for the real-world deployment of AVs.

Cultural differences influence how cyclists and drivers interact, affecting global autonomous vehicle (AV) adoption. AV-cyclist interfaces are needed to clarify AV intentions and resolve ambiguities when no human driver is present. These must adapt across cultures and road infrastructure. We conducted the first cross-cultural AV-cyclist user study across Stockholm (high segregation of cyclists from drivers), Glasgow (some segregation), and Muscat (no segregation). Cyclists used an AR simulator to cycle in physical space and experienced three holistic AV-cyclist interfaces. These integrated multiple interfaces into a larger ecosystem, e.g., a smartwatch synchronised with on-vehicle eHMI. Interfaces communicated AV location, intentions, or both. Riders from all cities preferred combined AV location and intention information but used it differently. Stockholm cyclists focused on location, validating intentions with driving behaviour. Glasgow riders valued both cues equally. Muscat cyclists trusted interfaces, prioritising intentions without relying on driving behaviour. These insights are key for global AV adoption.

The increasing use of virtual reality (VR) in public transportation enables travelers to engage in immersive entertainment or productive tasks, thereby enhancing the overall travel experience. However, mismatched VR visual motion and physical car motion can cause motion sickness (MS), leading to nausea, postural instability and reduced time for enjoyment or productive tasks. Thus, the benefits of using VR in transportation systems are currently limited to individuals who do not experience MS. Thus, effective MS mitigation is essential for improving travel safety, maximizing travel time, and expanding VR accessibility. Neuromodulation targeting the vestibular apparatus, such as bone-conducted vibration (BCV), has shown promise in reducing MS. However, it remains unclear whether neuromodulation directly targeting vestibular cortical regions, such as transcranial alternating current stimulation (tACS), is superior. This paper focuses on the design and validation of a novel vestibular cortical neuromodulation approach using tACS in mitigating MS in a novel simulated in-car VR environment. Eighty participants were recruited to evaluate the tACS approach. The results demonstrate that the proposed tACS approach effectively reduces nausea, enhances postural stability, and extends survival time in MS. Compared to BCV, tACS demonstrates a faster onset of action and longer mitigation effects. However, from an applied perspective, the effects of our tACS approach were relatively short-lived and accompanied by side effects such as tingling and itching.

AV-cyclist interaction research has focused on individual interfaces, e.g., eHMIs or AR glasses. However, cyclists using multiple independent interfaces could receive redundant or contradictory messages, causing unsafe encounters. We introduce the Holistic AV-Cyclist Interface: multiple displays facilitating communication together to ensure safe road sharing. We used participatory design with cyclists and HCI researchers (N = 12) to develop holistic interfaces around an actual vehicle equipped with an eHMI. Participants created holistic interfaces communicating blindspot warnings and AV intent and accommodating two-way communication, e.g. through on-bike directional indicators. They mostly placed interfaces on the bicycle handlebars and cycling accessories, e.g. helmet. Holistic interfaces could overcome design challenges: they can be multimodal, facilitate communication between AVs and multiple cyclists, and work in varying weather/visibility conditions. We categorised participant designs into a taxonomy and developed a novel holistic interface based on our findings. Our results are important for the safe integration of AVs into traffic.

The social cues drivers exchange with cyclists to negotiate space-sharing will disappear as autonomous vehicles (AVs) join our roads, leading to safety concerns. External Human-Machine Interfaces (eHMIs) on vehicles can replace driver social signals, but how these should be designed to communicate with cyclists is unknown. We evaluated three eHMIs across multiple traffic scenarios in two stages. First, we compared eHMI versatility, acceptability and usability in a VR cycling simulator. Cyclists preferred colour-coded signals communicating AV intent, easily seen through quick glances. Second, we refined the interfaces based on our findings and compared them outdoors. Participants cycled around a moving car with real eHMIs. They preferred eHMIs using large surfaces on the vehicle and animations reinforcing colour changes. We conclude with novel design guidelines for versatile eHMIs based on first-hand interaction feedback. Our findings establish the factors that enable AVs to operate safely around cyclists across different traffic scenarios.

Semi-autonomous vehicles allow drivers to engage with non-driving related tasks (NDRTs). However, these tasks interfere with the driver’s situational awareness, key when they need to safely retake control of the vehicle. This paper investigates if Augmented Reality (AR) could be used to present NDRTs to reduce their impact on situational awareness. Two experiments compared driver performance on a hazard prediction task whilst interacting with an NDRT, presented either as an AR Heads-Up Display or a traditional Heads-Down Display. The results demonstrate that an AR display including a novel dynamic attentional cue improves situational awareness, depending on the workload of the NDRT and design of the cue. The results provide novel insights for designers of in-car systems about how to design NDRTs to aid driver situational awareness in future vehicles.

This paper presents the first in-car VR motion sickness (VRMS) detection model based on lower face action units (LF-AUs). Initially developed in a simulated in-car environment with 78 participants, the model’s generalizability was later tested in realworld driving conditions. Motion sickness was induced using visual linear motion in the VR headset and physical horizontal rotation via a rotating chair. We used a convolutional neural network (MobileNetV3) to automatically extract LF-AUs from images of the users’ mouth region, captured by the VR headset’s built-in camera. These LF-AUs were then used to train a Support Vector Regression (SVR) model to estimate motion sickness scores. We compared the SVR model’s performance using LF-AUs, pupil diameters, and physiological features (individually and in combination) from the same VR headset. Results showed that both individual LF-AU (right dimple) and combined LF-AUs had significant Pearson correlations with self-reported motion sickness scores and achieved lower root mean squared error compared to pupil diameters. The best detection results were obtained by combining LF-AUs and pupil diameters, while physiological features alone did not yield significant results. The LF-AUs-based model demonstrated encouraging generalizability across different settings in the independent studies.

Physical interactions with socially assistive robots (SARs) positively affect user wellbeing. However, haptic experiences when touching a SAR are typically limited to perceiving the robot’s movements or shell texture, while other modalities that could enhance the touch experience with the robot, such as vibrotactile stimulation, are under-explored. In this exploratory qualitative study, we investigate the potential of enhancing human interaction with the PARO robot through vibrotactile heartbeats, with the goal to regulate subjective wellbeing during stressful situations. We conducted in-depth one-on-one interviews with 30 participants, who watched three horror movie clips alone, with PARO, and with a PARO that displayed a vibrotactile heartbeat. Our findings show that PARO’s presence and its interactive capabilities can help users regulate emotions through attentional redeployment from a stressor toward the robot. The vibrotactile heartbeat further reinforced PARO’s physical and social presence, enhancing the socio-emotional support provided by the robot and its perceived life-likeness. We discuss the impact of individual differences in user experience and implications for the future design of life-like vibrotactile stimulation for SARs.

The fight or flight phenomena is of evolutionary origin and responsible for the type of defensive behaviours enacted, when in the face of threat. This review attempts to draw the link between fear and aggression as motivational levers for fight or flight defensive behaviours. Furthermore, this review investigates whether human biological motion is modulated by the affective behaviours associated with the fight or flight phenomenon. It examines how threat informed emotion and motor systems have the potential to result in the modulation of empathetic appraisal. This is of interest to this systematic review, as empathetic modulation is crucial to prosocial drive, which has the potential to alleviate the perceived threat. Hence, this review investigates the role of affective computing in capturing the potential outcome of threat perception and associated empathetic responses. To gain a comprehensive understanding of the affective responses and biological motion evoked from threat scenarios, affective computing methods used to capture these neurophysiological and behavioural responses are discussed. A systematic review using Google Scholar and Web of Science was conducted as of 2023, and findings were supplemented by bibliographies of key articles. A total of 26 studies were analysed from initial web searches to explore the topics of empathy, threat perception, fight or flight, fear, aggression, and human motion. Relationships between affective behaviours (fear, aggression) and corresponding motor defensive behaviours (fight or flight) were examined within threat scenarios, and whether existing affective computing methods are effective in capturing these responses, identifying the varying consensus in the literature, challenges, and limitations of existing research.

Social anxiety is a prevalent mental health concern that impacts quality of life and makes social spaces less accessible. We conducted two studies with socially anxious participants, investigating using affective haptic comfort objects to provide calming support during social exposure. Participatory prototyping informed the design and use of the intervention, which was then evaluated between-groups with a social exposure task. Treatment participants held their preferred vibration-augmented prototype during this task; control participants did not. We observed no change in physiological measures, but treatment participants exhibited a significantly broader distribution of psychological anxiety scores. Participants in both studies found their objects pleasant and calming, made positive emotional associations with resonant stimuli, and used their objects to afford self-soothing tactile experiences. We discuss how future designers can facilitate calming affective haptic interfaces for socially anxious settings.

Cybersickness remains a major drawback of Virtual Reality (VR) headsets, as a breadth of stationary experiences with visual self-motion can result in visually-induced motion sickness. However, not everybody experiences the same intensity or type of adverse symptoms. Here we propose that prior experience with virtual environments can predict ones degree of cybersickness. Video gaming can enhance visuospatial abilities, which in-turn relate negatively to cybersickness - meaning that consistently engaging in virtual environments can result in protective habituation effects. In a controlled stationary VR experiment, we found that 'VR-naive' video gamers experienced significantly less cybersickness in a virtual tunnel-travel task and outperformed 'VR-naive' non-video gamers on a visual attention task. These findings strongly motivate the use of non-VR games for training VR cybersickness resilience, with future research needed to further understand the mechanism(s) by which gamers become cybersickness resilient - potentially expanding access to VR for even the most susceptible participants.

As Artificial Intelligence (AI) continues to develop, it becomes vital to understand more of the nuances of Human-AI interactions. This study aims to uncover how developers can design AI to feel more human in a work environment where only written feedback is possible. Participants will identify a location from Google Maps. To do this successfully, participants must rely on the answers provided by their teammates, one AI and one human. The experiment will run a 2x4 de-sign where AI's responses will either be designed in a human style (high humanness) or state a one-word answer (low humanness), the latter of which is more typical in machines and AI. The reliability of the AI will either be 60% or 90%, and the human will be 30%. Participants will be given a series of questionnaires to rate their opinions of the AI and rate feelings of trust, confidence and performance throughout the study. Following this study, the aim is to identify specific design elements that allow AI to feel human and successfully appear to have social intelligence in more interactive settings.

Autonomous vehicles (AVs) must communicate their intentions to nearby road users and may use external Human-Machine Interfaces (eHMIs). Most eHMIs have focused on interaction with pedestrians at crossings. However, these may not work for cyclists, who can be anywhere around vehicles and encounter them in diverse traffic scenarios. We used participatory design with cyclists and AutoUI researchers (N = 12) collaborating in a real-world setting to design eHMIs around an actual vehicle. Participants preferred eHMIs that co-exist with traditional vehicle signals and use a single design language across traffic scenarios to communicate awareness and intent quickly without distracting their attention from the road. We used our findings to develop a taxonomy of eHMI features for cyclists, allowing us to synthesise the designs and contribute versatile eHMI concepts catered to cyclists’ needs. This is important to ensure AVs can safely navigate and interact with cyclists in all road scenarios.

Cyclists encounter drivers in many traffic scenarios; good communication is key to avoiding collisions. Little is known about everyday driver-cyclist interaction and communication. This is important in designing Automated Vehicles (AVs) that must drive safely around cyclists. We explored driver-cyclist interaction across diverse scenarios through in-the-wild observations (N = 414) and a naturalistic study involving cyclists wearing eye-trackers (N = 12). Results showed cyclists attended to road markings and traffic signs in controlled traffic scenarios but to vehicle sides and windows in uncontrolled encounters. Interactions were unlikely at controlled intersections, but various techniques were used to negotiate right-of-way in uncontrolled scenarios, e.g. cyclists used arm gestures and shoulder checks to communicate their intent and awareness when lane merging. Drivers communicated these through on-vehicle signals and head movements at roundabouts. We discuss the implications of driver-cyclist interaction behaviour on AV interaction design and offer insights into system requirements to support cyclists riding in traffic.

VR (Virtual Reality) Motion Sickness (VRMS) refers to purely visually-induced motion sickness. Not everyone is susceptible to VRMS, but if experienced, nausea will often lead users to withdraw from the ongoing VR applications. VRMS represents a serious challenge in the field of VR ergonomics and human factors. Like other neuro-ergonomics researchers did before, this paper considers VRMS as a brain state problem as various etiologies of VRMS support the claim that VRMS is caused by disagreement between the vestibular and visual sensory inputs. However, what sets this work apart from the existing literature is that it explores anti-VRMS brain patterns via electroencephalogram (EEG) in VRMS-resistant individuals. Based on existing datasets of a previous study, we found enhanced theta activity in the left parietal cortex in VRMS-resistant individuals (N= 10) compared to VRMS-susceptible individuals (N=10). Even though the sample size per se is not large, this finding achieved medium effect size. This finding offers new hypotheses regarding how to reduce VRMS by the enhancement of brain functions per se (e.g., via non-invasive transcranial electrostimulation techniques) without the need to redesign the existing VR content.

This paper investigates how presenting emotionally resonant vibrotactile stimuli at cool, neutral and warm temperature levels impacts mean ratings for emotional resonance and affective response. Affective vibrotactile stimuli can elicit pleasant or calming responses, making them applicable for emotion regulation. Evoking real-world sensations via emotional resonance can widen their affective range and improve their effectiveness, and allow them to enhance immersive multimodal experiences. Thermotactile cues have been shown to affect emotional responses, but have not been combined with emotionally resonant vibrations to see how they change responses to such cues. This study (n=20) assessed the impact of 3 temperature levels (24℃, 30℃, and 34℃) on 15 emotionally resonant vibrotactile cues and observed if emotionally resonant stimuli exceeded the affective range non-resonant vibrotactile stimuli. The findings suggest that presenting specific resonant vibrations at temperatures that are appropriate for the sensation they evoke can improve emotional resonance and vice versa. In addition, temperature had a positive effect on affective response and emotionally resonant vibrations were found to have a wider affective range than traditional vibrotactile cues. These findings support using emotionally resonant vibrations and thermal cues to elicit desirable emotional responses in emotion regulation and immersive media applications.

Haptics is an essential element of interaction between humans and socially assistive robots. However, it is often limited to movements or vibrations and misses key aspects such as temperature. This mixed-methods study explores the potential of enhancing human-robot interaction [HRI] through thermal stimulation to regulate affect during a stress-inducing task. Participants were exposed to thermal stimulation while completing the Mannheim-multicomponent-stress-task (MMST). Findings yielded that human-robot emotional touch may induce comfort and relaxation during the exposure to acute stressors. User affect may be further enhanced through thermal stimulation, which was experienced as comforting, de-stressing, and altered participants’ perception of the robot to be more life-like. Allowing participants to calibrate a temperature they perceived as calming provided novel insights into the temperature ranges suitable for interaction. While neutral temperatures were the most popular amongst participants, findings suggest that cool (4 – 29 ºC), neutral (30 – 32 ºC), and warm (33ºC -36 ºC) temperatures can all induce comforting effects during exposure to stress. The results highlight the potential of thermal HRI in general and, more specifically, the advantages of personalized temperature calibration.

Virtual reality (VR) has the potential to induce cybersickness (CS), which impedes CS-susceptible VR users from the benefit of emerging VR applications. To better detect CS, the current study investigated whether/how the newly proposed human vestibular network (HVN) is involved in flagship consumer VR-induced CS by simultaneously recording autonomic physiological signals as well as neural signals generated in sensorimotor and cognitive domains. The VR stimuli were made up of one or two moderate CS-inducing entertaining task(s) as well as a mild CS-inducing cognitive task implemented before and after the moderate CS task(s). Results not only showed that CS impaired cognitive control ability, represented by the degree of attentional engagement, but also revealed that combined indicators from all three HVN domains could together establish the best regression relationship with CS ratings. More importantly, we found that every HVN domain had its unique advantage with the dynamic changes in CS severity and time. These results provide evidence for involvement of the HVN in CS and indicate the necessity of HVN-based CS detection.

This paper presents a survey informing a user-first approach to designing calming affective haptic stimuli by eliciting user preferences in different social scenarios. Prior affective haptics research presented users with stimuli and recorded emotional responses. By contrast this work focuses on the sensations users wish to experience and how these can be simulated using haptics. The survey (n=81) investigated which users preferences in four social situations to reduce social anxiety. Using thematic analysis of responses we created a coding scheme of stimuli derived from real-world experiences to emulate with affective haptics. By cross-referencing these categories with affective haptics research, we provide recommendations to designers about which calming stimuli users wish to experience socially and how they can be implemented.

This paper describes a novel category of affective vibrotactile stimuli which evoke real-world sensations and details a study into emotional responses to them. The affective properties of short and abstract vibrotactile waveforms have previously been studied and shown to have a narrow emotional range. By contrast this paper investigated emotional responses to longer waveforms and to emotionally resonant vibrotactile stimuli, stimuli which are evocative of real-world sensations such as animal purring or running water. Two studies were conducted. The first recorded emotional responses to Tactons with a duration of 20 seconds. The second investigated emotional responses to novel emotionally resonant stimuli. Stimuli that users found more emotionally resonant were more pleasant, particularly if they had prior emotional connections to the sensation represented. Results suggest that future designers could use emotional resonance to expand the affective response range of vibrotactile cues by utilising stimuli with which users bear an emotional association.

Haptic feedback can improve safety and driving behaviour. While vibration has been widely studied, other haptic modalities have been neglected. To address this, we present two studies investigating the use of uni- and bimodal vibrotactile and thermal cues on the steering wheel. First, notifications with three levels of urgency were subjectively rated and then identified during simulated driving. Bimodal feedback showed an increased identification time over unimodal vibrotactile cues. Thermal feedback was consistently rated less urgent, showing its suitability for less time critical notifications, where vibration would be unnecessarily attention-grabbing. The second study investigated more complex thermal and bimodal haptic notifications comprised of two different types of information (Nature and Importance of incoming message). Results showed that both modalities could be identified with high recognition rates of up to 92% for both and up to 99% for a single type, opening up a novel design space for haptic in-car feedback.

Haptic feedback is used in cars to reduce visual inattention. While tactile feedback like vibration can be influenced by the car's movement, thermal and cutaneous push feedback should be independent of such interference. This paper presents two driving simulator studies investigating novel tactile feedback on the steering wheel for navigation. First, devices on one side of the steering wheel were warmed, indicating the turning direction, while those on the other side were cooled. This thermal feedback was compared to audio. The thermal navigation lead to 94.2% correct recognitions of warnings 200m before the turn and to 91.7% correct turns. Speech had perfect recognition for both. In the second experiment, only the destination side was indicated thermally, and this design was compared to cutaneous push feedback. The simplified thermal feedback design did not increase recognition, but cutaneous push feedback had high recognition rates (100% for 200 m warnings, 98% for turns).

The goal of Information Retrieval (IR) systems is to satisfy searchers' Information Need (IN). Our research focuses on next-generation IR engines, which can proactively detect, identify, and serve INs without receiving explicit queries. It is essential, therefore, to be able to detect when INs occur. Previous research has established that a realisation of INs physically manifests itself with specific brain activity. With this work we take the next step, showing that monitoring brain activity can lead to accurate predictions of a realisation of IN occurrence. We have conducted experiments whereby twenty-four participants performed a Q/A Task, while their brain activity was being monitored using functional Magnetic Resonance Imaging (fMRI) technology. The questions were selected and developed from the TREC-8 and TREC 2001 Q/A Tracks. We present two methods for predicting the realisation of an IN, i.e. Generalised method (GM) and Personalised method (PM). GM is based on the collective brain activity of all twenty-four participants in a predetermined set of brain regions known to be involved in representing a realisation of INs. PM is unique to each individual and employs a 'Searchlight' analysis to locate brain regions informative for distinguishing when a “specific” user realises an information need. The results of our study show that both methods were able to predict a realisation of an IN (statistically) significantly better than chance. Our results also show that PM (statistically) significantly outperformed GM in terms of prediction accuracy. These encouraging findings make the first fundamental step towards proactive IR engines based on brain signals.

The main goal of information retrieval (IR) is to satisfy information need (IN). IN refers to a complex concept: at the very initial state of the phenomenon (that is, at a visceral level), even the searcher may not be aware of its existence. Thus, despite advances in the past few decades in both the IR and relevant scientific communities, we do not fully understand how an IN emerges and how it is physically manifested. In this article we aim to inform a holistic view of the realization of IN using functional magnetic resonance imaging. We collected new data of brain activity of 24 participants while they formulated and stated a realization of IN in a Question Answering task, focusing on a distributed set of brain regions associated with activities related to IN, found in our previous study. Results of a functional connectivity analysis led us to propose a neuropsychological model of the realization of IN. Our model consists of three components: (a) a successful memory retrieval component, (b) an information flow regulation component, and (c) a high‐level perception component. We believe this study constitutes an important step in unraveling the nature of IN and how to better satisfy IN.

Haptic feedback has been widely studied for in-car interactions. However, most of this research has used vibrotactile cues. This paper presents two studies that examine novel thermal feedback for navigation during simulated driving for a lane change task. In the first, we compare the distraction and time differences of audio and thermal feedback. The results show that the presentation of thermal stimuli does not increase lane deviation, but the time needed to complete a lane change increased by 1.82 seconds. In the second study, the influence of variable changes of thermal stimuli on the lane change task performance was tested. We found that the same stimulus design for warm and cold temperatures does not always elicit the same results. Furthermore, variable alterations can have different effects on specified tasks. This suggests that the design of thermal stimuli is highly dependent on what task result should be maximized.

Search is one of the most performed activities on the World Wide Web. Various conceptual models postulate that the search process can be broken down into distinct emotional and cognitive states of searchers while they engage in a search process. These models significantly contribute to our understanding of the search process. However, they are typically based on self-report measures, such as surveys, questionnaire, etc. and therefore, only indirectly monitor the brain activity that supports such a process. With this work, we take one step further and directly measure the brain activity involved in a search process. To do so, we break down a search process into five time periods: a realisation of Information Need, Query Formulation, Query Submission, Relevance Judgment and Satisfaction Judgment. We then investigate the brain activity between these time periods. Using functional Magnetic Resonance Imaging (fMRI), we monitored the brain activity of twenty-four participants during a search process that involved answering questions carefully selected from the TREC-8 and TREC 2001 Q/A Tracks. This novel analysis that focuses on transitions rather than states reveals the contrasting brain activity between time periods - which enables the identification of the distinct parts of the search process as the user moves through them. This work, therefore, provides an important first step in representing the search process based on the transitions between neural states. Discovering more precisely how brain activity relates to different parts of the search process will enable the development of brain-computer interactions that better support search and search interactions, which we believe our study and conclusions advance.

In this demonstration we show a thermal interaction design on the steering wheel for navigational cues in a car. Participants will be able to use a thermally enhanced steering wheel to follow instructions given in a turn-to-turn based navigation task in a virtual city. The thermal cues will be provided on both sides of the steering wheel and will indicate the turning direction by warming the corresponding side, while the opposite side is being cooled.

The raison d'etre of IR is to satisfy human information need. But, do we really understand information need? Despite advances in the past few decades in both the IR and relevant scientific communities, this question is largely unanswered. We do not really understand how an information need emerges and how it is physically manifested. Information need refers to a complex concept: at the very initial state of the phenomenon (i.e. at a visceral level), even the searcher may not be aware of its existence. This renders the measuring of this concept (using traditional behaviour studies) nearly impossible. In this paper, we investigate the connection between an information need and brain activity. Using functional Magnetic Resonance Imaging (fMRI), we measured the brain activity of twenty four participants while they performed a Question Answering (Q/A) Task, where the questions were carefully selected and developed from TREC-8 and TREC 2001 Q/A Track. The results of this experiment revealed a distributed network of brain regions commonly associated with activities related to information need and retrieval and differing brain activity in processing scenarios when participants knew the answer to a given question and when they did not and needed to search. We believe our study and conclusions constitute an important step in unravelling the nature of information need and therefore better satisfying it.

This paper presents a first evaluation of multimodal language-based warnings for handovers of control in autonomous cars. A set of possible handover situations varying in urgency is described. A set of multimodal, language-based warnings for these situations is then introduced. All combinations of audio, tactile and visual warnings for handovers were evaluated in terms of perceived urgency, annoyance and alerting effectiveness. Results showed clear recognition of the warning urgency in this new context, as well as low perceived annoyance overall, and higher perceived effectiveness for critical warnings. The time of transition from self-driving to manual mode in the presence of the warnings was then evaluated. Results showed quicker transitions for highly urgent warnings and poor driving performance for unimodal visual warnings. These results provide a novel set of guidelines for an effective transition of control between car and driver in an autonomous vehicle.

This paper examines the response of 10 typical and 10 autism spectrum participants in their reaction to a set of uni- and multisensory warning signals designed to indicate different levels of urgency. The warnings were composed of auditory, visual and tactile signals that were presented alone or in combination. Two experiments were conducted, a first that examined perceived urgency and annoyance with the warnings and a second that used a driving simulator scenario to explore recognition of the level of urgency and the speed of response. Results of Experiment 1 showed that there was no difference between groups in the perceived urgency of the warning signals, though the autism spectrum group reported less annoyance with the signals. Results of Experiment 2 showed that while both groups showed high accuracy in correctly reporting urgency level, the autism spectrum group performed better. Moreover, the fastest overall reaction times obtained were by the autism spectrum group when the warning included a visual component, with vision alone producing the quickest response. These results provide novel empirical insights on behaviour of drivers with autism when exposed to multimodal driver displays. They also highlight how consideration of characteristics of individual differences can contribute to the design of effective warning signals.

Multimodal displays are increasingly being utilized as driver warnings. Abstract warnings, without any semantic association to the signified event, and language-based warnings are examples of such displays. This paper presents a first comparison between these two types, across all combinations of audio, visual and tactile modalities. Speech, text and Speech Tactons (a novel form of tactile warnings synchronous to speech) were compared to abstract pulses in two experiments. Results showed that recognition times of warning urgency during a non-critical driving situation were shorter for abstract warnings, highly urgent warnings and warnings including visual feedback. Response times during a critical situation were shorter for warnings including audio. We therefore suggest abstract visual feedback when informing drivers during a non-critical situation and audio in a highly critical one. Language-based warnings during a critical situation performed equally well as abstract ones, so they are suggested as less annoying vehicle alerts.

Relevance is a central notion in Information Retrieval, but it is considered to be a difficult concept to define. We analyse brain signals for the first 800 milliseconds (ms) of a relevance assessment process to answer the question "when relevance is happening in the brain?" with the belief that it will lead to better operational definitions of relevance. For this purpose, we devised a user study in which we captured the brain response of 20 participants. Using a 64-channel EEG device, we measured the electrophysiological activity of the brain while the subjects were in the phase of giving an explicit judgement about the relevance of presented images according to a given topic. Analyses were then performed over different time windows of the recorded EEG signals using repeated measures ANOVA. Data reveal significant variation between relevance and non-relevance within the EEG signals from the presentation of the image to 800 milliseconds afterwards. At an early stage these differences were located at frontal and posterior electrode sites. However, at later stages these differences were located in central, centro-parietal and centro-frontal areas.Our findings are an important step towards (i) a better understanding of the concept of relevance and (ii) a more effective implicit feedback systems.

This paper describes two experiments evaluating a set of speech and tactile driver warnings. Six speech messages of three urgency levels were designed, along with their tactile equivalents, Speech Tactons. These new tactile warnings retained the rhythm of speech and used different levels of roughness and intensity to convey urgency. The perceived urgency, annoyance and alerting effectiveness of these warnings were evaluated. Results showed that bimodal (audio and tactile) warnings were rated as more urgent, more annoying and more effective compared to unimodal ones (audio or tactile). Perceived urgency and alerting effectiveness decreased along with the designed urgency, while perceived annoyance was lowest for warnings of medium designed urgency. In the tactile modality, ratings varied less as compared to the audio and audiotactile modalities. Roughness decreased and intensity increased ratings for Speech Tactons in all the measures used. Finally, Speech Tactons produced acceptable recognition accuracy when tested without their speech counterparts. These results demonstrate the utility of Speech Tactons as a new form of tactile alert while driving, especially when synchronized with speech.

Previous studies have investigated audio, visual and tactile driver warnings, indicating the importance of communicating the appropriate level of urgency to the drivers. However, these modalities have never been combined exhaustively and tested under conditions of varying situational urgency to assess their effectiveness both in the presence and absence of critical driving events. This paper describes an experiment evaluating all multimodal combinations of such warnings under two contexts of situational urgency: a lead car braking and not braking. The results showed that participants responded quicker to more urgent warnings, especially in the presence of a car braking. They also responded faster to the multimodal as opposed to unimodal signals. Driving behaviour improved in the presence of the warnings and the absence of a car braking. These results highlight the influence of urgency and number of modalities in warning design and indicate the utility of non-visual warnings in driving.

Previous studies have evaluated Audio, Visual and Tactile warnings for drivers, highlighting the importance of conveying the appropriate level of urgency through the signals. However, these modalities have never been combined exhaustively with different urgency levels and tested while using a driving simulator. This paper describes two experiments investigating all multimodal combinations of such warnings along three different levels of designed urgency. The warnings were first evaluated in terms of perceived urgency and perceived annoyance in the context of a driving simulator. The results showed that the perceived urgency matched the designed urgency of the warnings. More urgent warnings were also rated as more annoying but the effect of annoyance was lower compared to urgency. The warnings were then tested for recognition time when presented during a simulated driving task. It was found that warnings of high urgency induced quicker and more accurate responses than warnings of medium and of low urgency. In both studies, the number of modalities used in warnings (one, two or three) affected both subjective and objective responses. More modalities led to higher ratings of urgency and annoyance, with annoyance having a lower effect compared to urgency. More modalities also led to quicker responses. These results provide implications for multimodal warning design and reveal how modalities and modality combinations can influence participant responses during a simulated driving task.

This workshop will provide an overview of current research into how humans use faces, body postures and motion to recognize human activity and its social content. Additionally, we will focus on cutting edge research in brain imaging that reveals the brain processes underlying the recognition of face and gesture. These talks will highlight issues fundamental to recognizing face and gesture and thus encourage a dialogue between different approaches to understanding these complex signals.

"Sticky Hands" is a physical game for two people involving gentle contact with the hands. The aim is to develop relaxed and elegant motion together, achieve physical sensitivity-improving reactions, and experience an interaction at an intimate yet comfortable level for spiritual development and physical relaxation. We developed a control system for a humanoid robot allowing it to play Sticky Hands with a human partner. We present a real implementation including a physical system, robot control, and a motion learning algorithm based on a generalizable intelligent system capable itself of generalizing observed trajectories' translation, orientation, scale and velocity to new data, operating with scalable speed and storage efficiency bounds, and coping with contact trajectories that evolve over time. Our robot control is capable of physical cooperation in a force domain, using minimal sensor input. We analyze robot-human interaction and relate characteristics of our motion learning algorithm with recorded motion profiles. We discuss our results in the context of realistic motion generation and present a theoretical discussion of stylistic and affective motion generation based on, and motivating cross-disciplinary research in computer graphics, human motion production and motion perception.