Personal data and digital technologies increasingly shape children's daily experiences, but younger children are seldom presented with opportunities to reflect in-depth about their perspectives on personal data, beyond simple warnings about not sharing information online. We present Grasping Data, a set of portable activities aimed at helping children explore what counts as personal data, why some data feel more personal than others, and how meaning changes between audiences and purposes. The demo combines three linked tangible activities: a teddy-bear proxy for card sorting-based discussion, a wooden tangible prototype for self-generated activity data, and a set of shape-changing multi-material objects for embodied interpretation of personal data. For the IDC community, the demo contributes a 10-minute encounter highlighting concrete, research-based, low-tech methods to discuss personal data with young children across formal and informal learning settings.

Automated vehicles (AVs) must communicate their yielding intentions to pedestrians at crossings. External Human-Machine Interfaces (eHMIs, on-vehicle displays) are promising solutions, but were primarily tested with walking pedestrians. Runners are a significant pedestrian group who move faster and face distinct bodily and perceptual demands, raising questions about how pedestrian activity influences eHMI use. We conducted an outdoor study using an augmented reality simulator. Participants navigated a virtual crossing while walking and running; an approaching AV displayed one of three eHMIs: red/green colour-changing lights, animated cyan lights, or no-eHMI. No-eHMI consistently underperformed. Walkers mostly stopped and validated eHMI signals with vehicle behaviour; they processed both eHMI animations and colour changes effectively. Runners experienced greater time pressure to cross, increasing reliance on the eHMI over vehicle behaviour. They preferred colour changes over animation for rapid decisions. These findings are crucial for promoting eHMI inclusivity and physical wellbeing as AVs join our roads.

While Augmented Reality (AR) promises to transform remote collaboration, many aspects remain underexplored, particularly where to place remote avatars in messy, everyday environments. Two mixed-methods within-subjects studies examined avatar placement preferences during cooperative (brainstorming) and competitive (negotiation) tasks between participant pairs, focusing on the influence of physical objects (chairs, box, tree) on user preferences. Results showed a strong preference for frontal or slightly off-centre avatar placements, independent of task type. Participants preferred avatar placements that mirrored real-life behaviour, with chairs inviting placements and the tree deterring them. Notably, the large and visually simple box elicited mixed reactions, being viewed alternately as an obstacle to avoid when placing avatars or as an inviting physical anchor for them, despite causing a clear physicality conflict. We term this the "(Anti-)Affordance Problem", highlighting the complexity of avatar placement within physical contexts, and the necessity for AR collaboration platforms to respond to real-world constraints, offering flexibility in avatar placements to accommodate diverse user preferences.

Data physicalization is a promising approach for empowering children to understand and enjoy their own data. However, it relies on embodied metaphors to convey information effectively. This paper explores how to elicit children’s embodied experiences using a set of shape-changing objects that can inform the design of dynamic physicalizations. We propose a set of auxetic metamaterials, which can bend, twist, scale and shear. Following principles of tangible interaction, we conducted a study with 59 children who participated in four movement-based games before being introduced to the collection of shape-changing tangibles. Children expressed metaphors based on these activities related to concepts such as containers, rhythm, resistance, and semantic analogies, which we categorised into embodied schemas. Our findings reveal that characteristics of the shape-changing tangibles aid children in connecting their bodily experiences to dynamic transformations. Translating these insights into idea sketches, we outline how to tailor these affordable shape-changing mechanisms into usable prototypes.

While zoos deploy technologies for animals’ enrichment and visitors’ education, little research has investigated how technology can support joint computer use by animals and people working together. To bridge this gap, we developed CreatureConnect, a distributed device with which lemurs and zoo visitors alike can control the intensity of sounds, smells, and visuals on either side of the enclosure boundary. Over 20 days of subsequent observation, we recorded 541 lemur-system interactions, observed 16,139 zoo visitors, and collected 696 sets of questionnaire responses to examine the effects of distributing control on both species across baseline, visitor-control-only, lemur-control-only, shared-control, and no-control conditions. While lemurs used CreatureConnect significantly less when controlling it alone, humans exhibited significantly greater engagement, education, empathy, and overall-experience value under shared-control conditions, which outperformed all other conditions. In light of the results and the fundamental role of interaction and interfaces in animal–computer and human–computer interaction, the paper examines its vital implications for between-species collaboration and control.

Touch is a central aspect of social human-human interaction, yet when designing social human-robot interaction, the experience of touch is very often neglected. This work explores whether a robot should feel alive by investigating the effect of life-like haptic stimuli inspired by bio-physiological signals (heartbeat, purring, and thermal feedback) on user perception. The results of an interaction study (n=42) showed that haptic cues which simulate bio-physiological signals significantly increased perceived anthropomorphism, social warmth, and likeability of the PARO robot compared to no additional haptics or abstract vibration. We discuss how multimodal haptic cues, despite increasing perceived eeriness, also increased perceived likeability of the robot, providing novel insights into whether the uncanny valley effect may extend into non-visual modalities. These results provide the first empirical evidence that life-like tactile cues can enhance user perceptions of social robots, offering design guidance for deploying effective and engaging touch interactions between humans and zoomorphic robots.

Mental health issues are a huge health concern for undergraduate students, a problem which worsened during the COVID-19 pandemic. In a time when access to in-person support became scarce, we surveyed 156 undergraduate students from a UK university to understand their perceptions and use of online resources with regard to their mental health. Our findings show that, while search engines are primarily used to search for symptoms and diagnoses, video-based social media platforms are predominantly used to find support from people with lived experiences, while typically text-based platforms are used to find support from friends and family. Our findings inform recommendations that can support researchers and professionals in placing and tailoring mental health interventions and resources to the online locations which are most beneficial to students.

Conditionally automated vehicles require drivers to remain hazard-aware while disengaged from driving and engaged in Non-Driving Related Tasks (NDRTs) Conventional visual warning interfaces do not fully address this attentional conflict. While psychological research demonstrates that social cues are processed more efficiently than visual information, this has never been evaluated for safety–critical warnings in conditionally automated driving. This article presents the first empirical investigation into using a virtual agent to indicate danger and aid situational awareness while driver performs an NDRT. Across two experiments (n = 48), hazard prediction ability was assessed with either a visual cue or a virtual agent cueing danger. When including an attention-capturing element, both visual and social cue types mitigated driver distraction, maintaining hazard prediction comparable with undistracted driving. These findings suggest that virtual agents can be as effective as conventional visual cues in aiding situational awareness in safety–critical automated driving contexts, offering design insights for in-vehicle attentional displays supporting driver awareness.

Electrotactile haptics is a novel form of skin stimulation. It is versatile and allows manipulation of many design parameters, broadening the range of haptic experiences. However, how parameter changes influence qualitative experiences is not well understood. We systematically investigated (n = 45) the influence of single parameter changes (e.g., pulse width, frequency), capturing the breadth of experiences elicited. We compared differences between calibration and ordering task performance, first between left and right hands, then between single and both hands simultaneously, all with an open hand and while holding an object, fully exploring the design space. Results showed the wide range of qualitative experience, from concrete to abstract. Calibration should be performed on both hands simultaneously to ensure perception, as participants preferred lower pulse width values for simulation on both hands as compared to each hand alone. The results present the first detailed exploration into the qualitative aspects of electrotactile stimulation across the hands.

Mitigating motion sickness with non-invasive cortical vestibular brain stimulation (vBS) is promising because it does not require redesigning VR content or altering the motion profile of a moving platform. Instead, it aims to enhance individuals' intrinsic resistance to motion sickness. However, the mechanism underlying its online efficacy remains unclear (Here, "online efficacy" means the observed motion sickness reduction during vBS not before or after vBS). Motivated by lower motion sickness susceptibility observed in congenital nystagmus patients, we hypothesized that reductions in motion sickness under vBS condition are associated with increased eye movement instability. Leveraging sham-controlled, sex-balanced eye-tracking data from an open-access database, this study presents the first evaluation of a 20 Hz-vBS approach targeting the vestibular cortices as a countermeasure to motion sickness, using spatiotemporal eye movement metrics derived from eye-tracking data collected in VR. Results show that reduced motion sickness under active vBS indeed increased eye movement instability temporarily and spatially, compared to sham vBS. We also found that the temporal metrics was more strongly associated with female participants, whereas the spatial metric was more strongly associated with male participants.

Unlike any previous generation, children's lives are now highly datafied, tracked, and digitally monitored.They are increasingly vulnerable to data collection from seemingly innocuous toys and devices with cameras, sensors, voice recognition, and geolocation.Yet, children typically lack awareness or control over these data exchanges, as consent is usually given by adult caregivers (who may also lack data literacy).Despite regulatory improvements, interdisciplinary approaches remain crucial to empowering children to understand, value, and manage their personal data.How can the IDC community involve children in the design and use of their personal data?This workshop unites experts to explore current practices.1 CCS Concepts• Security and privacy → Social aspects of security and privacy; • Human-centered computing → Interaction design 1 This workshop merged with "Towards a Research Agenda for Including Children and their Care Ecosystems in HCI" to run a joint workshop: Grasping Data Together: Interdisciplinary Agendas for Engaging Children and their Care Ecosystems with their personal data

Attachment is a crucial part of children's psychological development and can impact social relationships and overall mental health and well-being.However, measures to assess attachment are traditionally costly in terms of time and use.Though previous interventions have tried to support the assessment of attachment in children, there have been few developments of a low-cost, fast, and efficient tool.Therefore, we present the demo for a School Attachment Monitor (SAM).The SAM is a digital tool that utilises artificial intelligence to assess attachment in children with 80% accuracy [21].Our demo showcases the SAM at a key point of development, exploring alternative methods of interaction to suit young children today and its deployability in schools.

Cyclists need interfaces such as on-vehicle displays or augmented-reality (AR) glasses for effective communication with autonomous vehicles (AVs) when human drivers are no longer present. Interfaces must handle complex situations involving multiple AVs around a cyclist. Holistic AV-Cyclist Interfaces (HACIs) are a novel solution; they group interfaces into a multimodal interconnected system to support the rider. However, the best way to present information is uncertain. We explored this in a scenario with three AVs using CycleARcade, a new multi-user AR platform for designing and evaluating HACIs. Cyclists and HCI researchers collaboratively created and tested HACIs within CycleARcade through a novel iterative participatory design method. We synthesised three HACIs from this process and assessed them with riders in CycleARcade. Participants preferred HACIs with AR displays integrated into the environment to avoid road distractions, paired with spatial audio communicating AV proximity. These findings provide crucial input for the real-world deployment of AVs.

Cultural differences influence how cyclists and drivers interact, affecting global autonomous vehicle (AV) adoption. AV-cyclist interfaces are needed to clarify AV intentions and resolve ambiguities when no human driver is present. These must adapt across cultures and road infrastructure. We conducted the first cross-cultural AV-cyclist user study across Stockholm (high segregation of cyclists from drivers), Glasgow (some segregation), and Muscat (no segregation). Cyclists used an AR simulator to cycle in physical space and experienced three holistic AV-cyclist interfaces. These integrated multiple interfaces into a larger ecosystem, e.g., a smartwatch synchronised with on-vehicle eHMI. Interfaces communicated AV location, intentions, or both. Riders from all cities preferred combined AV location and intention information but used it differently. Stockholm cyclists focused on location, validating intentions with driving behaviour. Glasgow riders valued both cues equally. Muscat cyclists trusted interfaces, prioritising intentions without relying on driving behaviour. These insights are key for global AV adoption.

Zoos aim to uphold high animal welfare standards while educating the public, yet the direct interactions that attract visitors can negatively impact the animals. Exploring technological solutions to reshape this human-animal relationship in zoos, we developed a novel device allowing lemurs to trigger olfactory, auditory, and visual stimuli in their enclosure. Over 63 days, lemurs engaged most with multimodal stimuli and with visual the least. We then created a similar device for zoo visitors to educate them about lemurs and their stimuli choices. Deploying for 20 days (no devices, lemur-only, visitor-only, and both devices), we examined the impact on visitor behaviour, education, empathy, and experience. From 968 questionnaires and 25,782 visitors, we found that using technology on the lemur and visitor sides jointly significantly enhanced all measured visitor factors, even if the visitors did not directly interact with the device or observe lemurs using theirs. This approach supports long-term conservation and visitor education efforts.

Rotational gain has been studied largely under active self-motion, where users control their own movement. In multiple VR scenarios, the user is under passive self-motion: their body is moved by a training simulator, a motorised chair, or a vehicle-based VR application. Users may be less sensitive to manipulation under passive motion - especially when engaged in a secondary task - meaning motion experiences could be expanded by high gains and even opposed virtual-physical motion. We identified both the perceptible and maximum comfortable thresholds of rotational gain when passively turned in a motorised chair, with and without a task, for the first time. We then applied those thresholds to an ’unbounded’ in-car VR game where the user experiences an entirely different route to their physical movement. We provide the first guidelines for creating enhanced passive motion experiences and open the design space to new applications not restricted by physical motion.

Transitioning to manual control following a Take-Over Request (TOR) in Level 3 autonomous cars is challenging, requiring drivers to re-engage with driving after engaging with Non-Driving Related Tasks (NDRTs). Effective TOR design can mitigate this challenge. We present the first study on how culture, age, and NDRT intersect to shape TOR design. In a cross-cultural study across the UK (high traffic-law compliance) and Israel (low compliance), involving older and younger drivers, participants designed TORs for four NDRTs in a real car setting. Results revealed a universal preference for re-purposing NDRT-devices to issue TORs. Older drivers preferred tri-modal TORs that suspend the NDRT; younger drivers favoured bi-modal TORs allowing NDRT interruption management. Due to altered alert sensitivity and low law compliance, Israeli participants included a RiskMeter to assess hazard criticality. We introduce novel TOR designs and taxonomy features to guide culturally and age-sensitive TOR development, key for global Level 3 adoption.

Augmented reality (AR) applications present virtual elements that are connected to the real world. While these may be aware of a user's geographical or visual context, the real sounds in a user's environment are rarely used in the AR experience. We investigate audio augmented reality (AAR) with sound as the primary output. We present the first evaluation of sonic linking, where an AAR application uses real sounds (bird calls and car engines) in a user's surroundings to drive the interaction. We developed two AAR applications to investigate how to design and use such links: a game where entities are spawned based on real-world sounds and a music player with sound-reactive filtering and volume adjustment. Design variations are compared to cover different AAR scenarios, types of sonic link, and existing unlinked equivalents. The results show that sonic linking can create a more augmented, engaging AAR experience, and may alter a user's relationship with their real-world surroundings, enabling new types of augmented reality applications.

Augmented Reality Television (ARTV) affords new ways to experience media – from displaying auxiliary information to increasing immersion with additional 3D models. Many people consume video content when travelling or commuting; accordingly, the benefits of ARTV in this context merit investigation. We conducted a study (N=20) exploring the effects of using ARTV Ornamentations when watching Game of Thrones on a train. We used Virtual Reality to simulate use of AR in a train carriage and evaluated four different levels of ARTV Ornamentation, from a simple floating panel through to themed models and interior textures that change to match the on-screen visuals. Results showed that Ornamentations matching the show’s genre were sufficient to provide an engaging and immersive experience, but Ornamentations themed to the show which changed based on the scene further elevated the experience. This means it is possible for passengers to have an immersive experience while commuting without significantly reducing their awareness of their environment. We also reflect on the implications of our work for an existing ARTV framework and propose the inclusion of several new aspects to encourage designers to consider the unique challenges and affordances of ARTV Ornamentations.

Mobile phones are most likely the subject of targeted attacks, such as software exploits. The resources needed to carry out such attacks are becoming increasingly available and, hence, easily executable, putting users’ privacy at risk. We conducted a systematic literature analysis to understand the relationship between resources and attack feasibility and present a categorisation of social engineering and side-channel attacks on mobile phones focusing on the resources attackers require. Our proposed categorisation levels facilitate an in-depth understanding of how mobile phone attacks can be executed using different combinations of partly simple resources. The analysis reveals that discrete protection mechanisms are insufficient to provide all-inclusive protection. The proposed categorisation assists in building novel solutions for safeguarding users’ privacy from diverse attacks by carefully considering the potential misuse of resources. We conclude by outlining future research directions highlighting the urgent need for a holistic user defense.

Audio augmented reality (AAR) games allow for audio-only game experiences which blend real and virtual worlds. Despite being sound-based in their output, these games utilise a small number of control schemes, and rarely deploy control schemes that are also sonic in nature. We present the first evaluation of sonic control schemes for AAR gameplay. We present a systematic literature review that collates the control schemes and game scenarios deployed in existing AAR games, and a user study comparing these traditional control schemes with sonic controls -- speech, music, and sonic gesture -- in minigames corresponding to the most common game scenarios. Our results show that while there are some key differences between these control schemes, sonic controls show promise for use in AAR gameplay, and could be deployed in scenarios where AAR developers currently reach for established control schemes.

The increasing use of virtual reality (VR) in public transportation enables travelers to engage in immersive entertainment or productive tasks, thereby enhancing the overall travel experience. However, mismatched VR visual motion and physical car motion can cause motion sickness (MS), leading to nausea, postural instability and reduced time for enjoyment or productive tasks. Thus, the benefits of using VR in transportation systems are currently limited to individuals who do not experience MS. Thus, effective MS mitigation is essential for improving travel safety, maximizing travel time, and expanding VR accessibility. Neuromodulation targeting the vestibular apparatus, such as bone-conducted vibration (BCV), has shown promise in reducing MS. However, it remains unclear whether neuromodulation directly targeting vestibular cortical regions, such as transcranial alternating current stimulation (tACS), is superior. This paper focuses on the design and validation of a novel vestibular cortical neuromodulation approach using tACS in mitigating MS in a novel simulated in-car VR environment. Eighty participants were recruited to evaluate the tACS approach. The results demonstrate that the proposed tACS approach effectively reduces nausea, enhances postural stability, and extends survival time in MS. Compared to BCV, tACS demonstrates a faster onset of action and longer mitigation effects. However, from an applied perspective, the effects of our tACS approach were relatively short-lived and accompanied by side effects such as tingling and itching.

Immersive technologies allow us to personalize our reality while traveling; however, widespread adoption remains limited. This study investigates the challenges of adopting immersive devices in transit, particularly when passengers encounter asymmetric experiences using disparate devices. These situations arise as co-located passengers adopt devices with varying levels of immersion, environmental information, and interactive capabilities. Using an enactment methodology in a transit scenario involving disparate devices, our study (N = 21) reveals that asymmetric passenger experiences create a sense of disconnection among passengers, especially with varying device immersion capabilities. When interactions require more than a basic response, for example complex verbal exchanges relying on social cues like gaze, adoption is further hindered. The incorporation of cues from reality addresses safety concerns but requires further refinement to support interactions that require complex or extended user involvement. This research advances the understanding of asymmetric passenger dynamics, contributing to the design of immersive technologies for transit settings.

Auditory Augmented Reality (AAR) is a way of augmenting our aural perception of reality. AAR could offer new ways of expressing information embedded in everyday objects by augmenting their auditory contact properties in real-time. Many acoustic augmentations (e.g., altering the sound, reverb) are available, but how they impact the perception of the physical object and its audio properties are unknown. In two experiments, we explore the effects of different audio feedback (surface materials, musical instruments) and reverb parameters (mixing, depth) on perceived simultaneity, unity, agency, and object properties (object size, material, room size), along with their recognition. We found that audio feedback had a significant but small effect on interaction and altered perceived object properties. Mixing in reverb reduced audio recognition but boosted reverb determination. We offer guidelines on combining audio feedback and reverb based on acoustic properties and reverb parameter adjustment to provide multi-level feedback for AAR.

AV-cyclist interaction research has focused on individual interfaces, e.g., eHMIs or AR glasses. However, cyclists using multiple independent interfaces could receive redundant or contradictory messages, causing unsafe encounters. We introduce the Holistic AV-Cyclist Interface: multiple displays facilitating communication together to ensure safe road sharing. We used participatory design with cyclists and HCI researchers (N = 12) to develop holistic interfaces around an actual vehicle equipped with an eHMI. Participants created holistic interfaces communicating blindspot warnings and AV intent and accommodating two-way communication, e.g. through on-bike directional indicators. They mostly placed interfaces on the bicycle handlebars and cycling accessories, e.g. helmet. Holistic interfaces could overcome design challenges: they can be multimodal, facilitate communication between AVs and multiple cyclists, and work in varying weather/visibility conditions. We categorised participant designs into a taxonomy and developed a novel holistic interface based on our findings. Our results are important for the safe integration of AVs into traffic.

The social cues drivers exchange with cyclists to negotiate space-sharing will disappear as autonomous vehicles (AVs) join our roads, leading to safety concerns. External Human-Machine Interfaces (eHMIs) on vehicles can replace driver social signals, but how these should be designed to communicate with cyclists is unknown. We evaluated three eHMIs across multiple traffic scenarios in two stages. First, we compared eHMI versatility, acceptability and usability in a VR cycling simulator. Cyclists preferred colour-coded signals communicating AV intent, easily seen through quick glances. Second, we refined the interfaces based on our findings and compared them outdoors. Participants cycled around a moving car with real eHMIs. They preferred eHMIs using large surfaces on the vehicle and animations reinforcing colour changes. We conclude with novel design guidelines for versatile eHMIs based on first-hand interaction feedback. Our findings establish the factors that enable AVs to operate safely around cyclists across different traffic scenarios.

Semi-autonomous vehicles allow drivers to engage with non-driving related tasks (NDRTs). However, these tasks interfere with the driver’s situational awareness, key when they need to safely retake control of the vehicle. This paper investigates if Augmented Reality (AR) could be used to present NDRTs to reduce their impact on situational awareness. Two experiments compared driver performance on a hazard prediction task whilst interacting with an NDRT, presented either as an AR Heads-Up Display or a traditional Heads-Down Display. The results demonstrate that an AR display including a novel dynamic attentional cue improves situational awareness, depending on the workload of the NDRT and design of the cue. The results provide novel insights for designers of in-car systems about how to design NDRTs to aid driver situational awareness in future vehicles.

To prevent motion sickness, Virtual Reality (VR) experiences for vehicle passengers typically present “matched motion”: real vehicle movements are replicated 1:1 by movements in VR. This significantly limits virtual applications. We provide foundations for in-car VR experiences that break this constraint by manipulating the passenger’s visual perception of linear velocity through amplifying and reducing the virtual speed. In two on-the-road studies, we examined the application of Vehicular Translational Gain (1.5-9.5x) and Attenuation (0.66-0.14x) to real car speeds (~50km/h) across two VR tasks (reading and gaming), exploring journey perception, impact on motion sickness, travel experience and tasks. We found that vehicular gain/attenuation can be applied without significantly increasing motion sickness. Gain was more noticeable and affected perceived speed, distance, safety, relaxation and excitement, being well-suited to gaming, while attenuation was more suitable for productivity. Our work unlocks new ways that VR applications can enhance and alter the passenger experience through novel perceptual manipulations of vehicle velocity.

While noise-cancelling headphones can block out or mask environmental noise with digital sound, this costs the user situational awareness and information. With the advancement of acoustically transparent personal audio devices (e.g. headphones, open-ear audio frames), Auditory Augmented Reality (AAR), and real-time audio processing, it is feasible to preserve user situational awareness and relevant information whilst diminishing the perception of the noise. Through an online survey (n=124), this research explored users’ attitudes and preferred AAR strategy (keep the noise, make the noise more pleasant, obscure the noise, reduce the noise, remove the noise, and replace the noise) toward different types of noises from a range of categories (living beings, mechanical, and environmental) and varying degrees of relevance. It was discovered that respondents’ degrees of annoyance varied according to the kind of noise and its relevance to them. Additionally, respondents had a strong tendency to reduce irrelevant noise and retain more relevant noise. Based on our findings, we discuss how AAR can assist users in coping with noise whilst retaining relevant information through selectively suppressing or altering the noise, as appropriate.

To the best of our knowledge, this article presents the first experiments on speech-based Automatic Personality Perception performed in a virtual immersive audio environment. The key-difference compared to all previous works in the literature is that, in a virtual immersive environment, people perceive not only the voice of the speakers, but also their position and distance in space. Therefore, it is possible to investigate for the first time whether people tend to attribute different traits to people speaking at different distances and, if so, whether this makes a difference in terms of Automatic Personality Perception. The experiments were performed over 360 recordings rendered at different distances (120 speakers including 60 female and 60 male). The results show that there are correlations between perceived distance and personality judgments. Furthermore, the experiments show that the performance in Automatic Personality Perception improves when taking perceived distance into account. These results are important because immersive environments are likely to become one of the main technological interfaces through which people interact with one another and with machines.

This paper presents the first in-car VR motion sickness (VRMS) detection model based on lower face action units (LF-AUs). Initially developed in a simulated in-car environment with 78 participants, the model’s generalizability was later tested in realworld driving conditions. Motion sickness was induced using visual linear motion in the VR headset and physical horizontal rotation via a rotating chair. We used a convolutional neural network (MobileNetV3) to automatically extract LF-AUs from images of the users’ mouth region, captured by the VR headset’s built-in camera. These LF-AUs were then used to train a Support Vector Regression (SVR) model to estimate motion sickness scores. We compared the SVR model’s performance using LF-AUs, pupil diameters, and physiological features (individually and in combination) from the same VR headset. Results showed that both individual LF-AU (right dimple) and combined LF-AUs had significant Pearson correlations with self-reported motion sickness scores and achieved lower root mean squared error compared to pupil diameters. The best detection results were obtained by combining LF-AUs and pupil diameters, while physiological features alone did not yield significant results. The LF-AUs-based model demonstrated encouraging generalizability across different settings in the independent studies.

Virtual Reality (VR) headsets allow us to personalise how we experience reality while travelling, offering benefits over traditional devices. VR that incorporates elements of reality could be crucial to creating safe and socially acceptable VR experiences by supporting VR users to be aware of the ever-changing transit environment, and the presence and actions of bystander passengers. Our study (N=14) examines VR in-the-wild on real train journeys to explore how VR users desire to employ passthrough style ‘portals’ to reality, and what impact these portals have on their perceived safety, social acceptability, and more in a real passenger context. We utilised a qualitative methodology with user interviews to evaluate experiences after real train journeys. Our findings indicate a favourable response to VR use on public transport, particularly when travelling alone. Portals effectively mitigate safety, awareness, and social concerns, but also pose challenges in simultaneously engaging with real and virtual environments. Users show a preference for passive monitoring of real-world changes over the more demanding active checking of the portals. However, this can lead to 'information wormholes,' where changes in the real-world slip past the portals. This study provides ecological validity to adopting VR in real transit settings and offers insights for the further development of reality-awareness systems.

Physical interactions with socially assistive robots (SARs) positively affect user wellbeing. However, haptic experiences when touching a SAR are typically limited to perceiving the robot’s movements or shell texture, while other modalities that could enhance the touch experience with the robot, such as vibrotactile stimulation, are under-explored. In this exploratory qualitative study, we investigate the potential of enhancing human interaction with the PARO robot through vibrotactile heartbeats, with the goal to regulate subjective wellbeing during stressful situations. We conducted in-depth one-on-one interviews with 30 participants, who watched three horror movie clips alone, with PARO, and with a PARO that displayed a vibrotactile heartbeat. Our findings show that PARO’s presence and its interactive capabilities can help users regulate emotions through attentional redeployment from a stressor toward the robot. The vibrotactile heartbeat further reinforced PARO’s physical and social presence, enhancing the socio-emotional support provided by the robot and its perceived life-likeness. We discuss the impact of individual differences in user experience and implications for the future design of life-like vibrotactile stimulation for SARs.

Head-mounted displays (HMDs) offer new possibilities for reclaiming and improving passenger travel time. Common Virtual Reality (VR) HMDs provide an immersive mobile media viewing experience, overcoming the screen size constraints of mobile and seat-back displays. However, they also reduce environmental awareness as they completely occlude reality. Conversely, Augmented Reality (AR) glasses afford environmental awareness by default, allowing unrestricted presentation of media through large virtual planar displays, but without the same immersion as VR. Our work transposes the immersive benefits of VR to AR, whilst preserving awareness of the environment. We present results from a user study (N=24) exploring the addition of plants and textures as augmentations of a simulated VR train environment to enhance passenger media immersion when watching a nature documentary. We evaluated two different fields of view (50°, 104°) and three different levels of ARTV ornamentation (from none to a fully textured jungle environment in the train). We found that on high FOV devices, ARTV ornamentation results in levels of media immersion that are not significantly lower than those achieved with full VR whilst retaining environmental awareness not possible with typical VR solutions. On low FOV devices, ARTV ornamentation loses effectiveness and distract users. Our results show that the immersiveness of AR can be significantly increased while still allowing the awareness of the environment that travellers need to manage their journeys.

Personal audio technologies empower individuals to manage their own auditory environment, affording control over both digital and real-world sounds. However, this control comes at a potential cost to social interaction, awareness of surroundings, and safety. This paper explores and investigates leveraging personal audio technologies as mediators in establishing personal audio spaces in homes. Our research methodology consisted of two key phases. Initially, a series of design workshops involving 16 participants were conducted to explore potential enhancements to everyday listening situations through technology. Subsequently, a within-subjects mixed-method study with 18 participants was undertaken to investigate the user experience of employing personal audio technologies to create a personalised audio space. The findings provide insights into future opportunities for listening practices, audio content, interpersonal listening, enhancing listening experiences as well as experiential components. Drawing from our results, we engage in a discussion on the potential manifestations of personal audio spaces, while also addressing current technological limitations and suggesting avenues for future research directions.

Extended Reality (XR) offers the potential to transform the passenger experience by allowing users to inhabit varied virtual spaces for entertainment, work or social interaction, whilst escaping the constrained transit environment. XR allows remote collaborators to feel like they are together and enables them to perform complex 3D tasks. However, the social and physical constraints of the passenger space pose unique challenges to productive and socially acceptable collaboration. Using a collaborative VR puzzle task, we examined the effects of five different f-formations of collaborator placement and orientation in an interactive workspace on social presence, task workload, and implications for social acceptability. Our quantitative and qualitative results showed that face-to-face formations were preferred for tasks with a high need for verbal communication but may lead to social collisions, such as inadvertently staring at a neighbouring passenger, or physical intrusions, such as gesturing in another passenger’s personal space. More restrictive f-formations, however, were preferred for passenger use as they caused fewer intrusions on other passengers’ visual and physical space.

It is challenging for older drivers to transition to manual control after a Take-Over Request (TOR) has been issued by a Level 3 car. This study investigated if the presentation source of the TOR affects driver performance when resuming control. We measured take-over performance, hazard perception, and user acceptance when the TOR was presented on (1) a smartphone displaying a Non-Driving Related Task (NDRT) simultaneously with the In-Vehicle Information System (IVIS), or (2) presented on the IVIS only. Two NDRTs that varied in cognitive demand were tested with older drivers aged 60-69 and 70+. For the lower cognitive demand NDRT, presenting the TOR on the smartphone+IVIS improved takeover performance, hazard perception, and user acceptance, with greater benefits observed in the 70+ group. For the cognitively demanding NDRT, the smartphone+IVIS presentation did not benefit either group of drivers. TOR designers can apply these findings to enhance TORs and assist older drivers in managing control transitions considering the NDRT cognitive demand.

Social anxiety is a prevalent mental health concern that impacts quality of life and makes social spaces less accessible. We conducted two studies with socially anxious participants, investigating using affective haptic comfort objects to provide calming support during social exposure. Participatory prototyping informed the design and use of the intervention, which was then evaluated between-groups with a social exposure task. Treatment participants held their preferred vibration-augmented prototype during this task; control participants did not. We observed no change in physiological measures, but treatment participants exhibited a significantly broader distribution of psychological anxiety scores. Participants in both studies found their objects pleasant and calming, made positive emotional associations with resonant stimuli, and used their objects to afford self-soothing tactile experiences. We discuss how future designers can facilitate calming affective haptic interfaces for socially anxious settings.

Cybersickness remains a major drawback of Virtual Reality (VR) headsets, as a breadth of stationary experiences with visual self-motion can result in visually-induced motion sickness. However, not everybody experiences the same intensity or type of adverse symptoms. Here we propose that prior experience with virtual environments can predict ones degree of cybersickness. Video gaming can enhance visuospatial abilities, which in-turn relate negatively to cybersickness - meaning that consistently engaging in virtual environments can result in protective habituation effects. In a controlled stationary VR experiment, we found that 'VR-naive' video gamers experienced significantly less cybersickness in a virtual tunnel-travel task and outperformed 'VR-naive' non-video gamers on a visual attention task. These findings strongly motivate the use of non-VR games for training VR cybersickness resilience, with future research needed to further understand the mechanism(s) by which gamers become cybersickness resilient - potentially expanding access to VR for even the most susceptible participants.

Autonomous vehicles (AVs) must communicate their intentions to nearby road users and may use external Human-Machine Interfaces (eHMIs). Most eHMIs have focused on interaction with pedestrians at crossings. However, these may not work for cyclists, who can be anywhere around vehicles and encounter them in diverse traffic scenarios. We used participatory design with cyclists and AutoUI researchers (N = 12) collaborating in a real-world setting to design eHMIs around an actual vehicle. Participants preferred eHMIs that co-exist with traditional vehicle signals and use a single design language across traffic scenarios to communicate awareness and intent quickly without distracting their attention from the road. We used our findings to develop a taxonomy of eHMI features for cyclists, allowing us to synthesise the designs and contribute versatile eHMI concepts catered to cyclists’ needs. This is important to ensure AVs can safely navigate and interact with cyclists in all road scenarios.

The use of Virtual Reality technology in vehicles is poised to bring about a new era of transport experiences, but its use can cause or enhance motion sickness. However, with careful design, VR devices have the potential to mitigate motion sickness and provide immersive experiences while travelling. We propose a novel implicit motion cue for manipulating a virtual display presented in Virtual and Extended Reality. Our design mitigates motion sickness by providing awareness of physical orientation changes through changes in the visual orientation of the virtual planar content. We performed two experiments on a rotating chair, testing mitigation properties of different levels of rotational gain applied to the display. Our results showed that the technique significantly reduced motion sickness, without negatively affecting task performance. Our findings show that we can subtly interleave motion cues in existing spatial content like planar displays and this can contribute to lessening motion sickness experienced.

Extended Reality (XR) technology brings exciting possibilities for aeroplane passengers, allowing them to escape their limited cabin space. Using nearby physical surfaces enables a connection with the real world while improving the XR experience through touch. However, available surfaces may be located in awkward positions, reducing comfort and input performance and thus limiting their long-term use. We explore the usability of passive haptic surfaces in different orientations, assessing their effects on input performance, user experience and comfort. We then overcome ergonomic issues caused by the confined space by using perceptual manipulation techniques that remap the position and rotation of physical surfaces and user movements, assessing their effects on task workload, comfort and presence. Our results show that the challenges posed by constrained seating environments can be overcome by a combination of passive haptics and remapping the workspace with moderate translation and rotation manipulations. These manipulations allow for good input performance, low workload and comfortable interaction, opening up XR use while in transit.

Cyclists encounter drivers in many traffic scenarios; good communication is key to avoiding collisions. Little is known about everyday driver-cyclist interaction and communication. This is important in designing Automated Vehicles (AVs) that must drive safely around cyclists. We explored driver-cyclist interaction across diverse scenarios through in-the-wild observations (N = 414) and a naturalistic study involving cyclists wearing eye-trackers (N = 12). Results showed cyclists attended to road markings and traffic signs in controlled traffic scenarios but to vehicle sides and windows in uncontrolled encounters. Interactions were unlikely at controlled intersections, but various techniques were used to negotiate right-of-way in uncontrolled scenarios, e.g. cyclists used arm gestures and shoulder checks to communicate their intent and awareness when lane merging. Drivers communicated these through on-vehicle signals and head movements at roundabouts. We discuss the implications of driver-cyclist interaction behaviour on AV interaction design and offer insights into system requirements to support cyclists riding in traffic.

Motion sickness is a problem for many in everyday travel and will become more prevalent with the rise of automated vehicles. Virtual Reality (VR) headsets have shown significant promise in-transit, enabling passengers to engage in immersive entertainment and productivity experiences. In a controlled multi-session motion sickness study using an actuated rotating chair, we examine the potential of multi-sensory visual and auditory motion cues, presented during a VR reading task, for mitigating motion sickness. We found that visual cues are most efficient in reducing symptoms, with auditory cues showing some beneficial effects when combined with the visual. Motion sickness had negative effects on presence as well as task performance, and despite the cognitive demand and multi-sensory cues, motion sickness still reached problematic levels. Our work emphasises the need for effective mitigations and the design of stronger multi-sensory motion cues if VR is to fulfil its potential for passengers.

Research on cyclists‘ safety and comfort is a growing topic. Existing works address support systems with novel interaction concepts such as augmented reality but also the design and evaluation of high-fidelity bicycle simulators. Since the field is still in its exploratory phase, there have been few attempts to systematically provide guidance for conducting experiments. For example, there is no consensus on the choice of representative driving scenarios, the proper choice of different bicycle simulators, and measurement standards to systematically compare the results of different studies. With this workshop, we want the community to gather and discuss a roadmap for the future of HCI bicycle research so that these issues can be overcome.

VR (Virtual Reality) Motion Sickness (VRMS) refers to purely visually-induced motion sickness. Not everyone is susceptible to VRMS, but if experienced, nausea will often lead users to withdraw from the ongoing VR applications. VRMS represents a serious challenge in the field of VR ergonomics and human factors. Like other neuro-ergonomics researchers did before, this paper considers VRMS as a brain state problem as various etiologies of VRMS support the claim that VRMS is caused by disagreement between the vestibular and visual sensory inputs. However, what sets this work apart from the existing literature is that it explores anti-VRMS brain patterns via electroencephalogram (EEG) in VRMS-resistant individuals. Based on existing datasets of a previous study, we found enhanced theta activity in the left parietal cortex in VRMS-resistant individuals (N= 10) compared to VRMS-susceptible individuals (N=10). Even though the sample size per se is not large, this finding achieved medium effect size. This finding offers new hypotheses regarding how to reduce VRMS by the enhancement of brain functions per se (e.g., via non-invasive transcranial electrostimulation techniques) without the need to redesign the existing VR content.

Immersive technologies allow us to control and customise how we experience reality, but are not widely used in transit due to safety, social acceptability, and comfort barriers. We propose that cues from reality can create reference points in virtuality, which we call Reality Anchors, will reduce these barriers. We used simulated public transportation journeys in a lab setting to explore Reality Anchors using speculative methods in two studies. Our first study (N=20) explored how elements of reality like objects, furniture, and people could be used as anchors, demonstrating that visibility of other passengers and personal belongings could reduce barriers. Our second study (N=19) focused on journey types that emerged from the first study - self-managed vs. externally managed journeys - revealing that self-managed journeys increased the need for anchors. We conclude that Reality Anchors can reduce concerns associated with immersive technology use in transit, especially for self-managed journeys.

Standalone Virtual Reality (VR) headsets can be used when travelling in cars, trains and planes. However, the constrained spaces around transport seating can leave users with little physical space in which to interact using their hands or controllers, and can increase the risk of invading other passengers' personal space or hitting nearby objects and surfaces. This hinders transport VR users from using most commercial VR applications, which are designed for unobstructed 1-2m 360° home spaces. In this paper, we investigated whether three at-a-distance interaction techniques from the literature could be adapted to support common commercial VR movement inputs and so equalise the interaction capabilities of at-home and on-transport users: Linear Gain, Gaze-Supported Remote Hand, and AlphaCursor. First, we analysed commercial VR experiences to identify the most common movement inputs so that we could create gamified tasks based on them. We then investigated how well each technique could support these inputs from a constrained $50\mathrm{x}50\text{cm}$ space (representative of an economy plane seat) through a user study $(\mathrm{N}=16)$, where participants played all three games with each technique. We measured task performance, unsafe movements (play boundary violations, total arm movement) and subjective experience and compared results to a control 'at-home' condition (with unconstrained movement) to determine how similar performance and experience were. Results showed that Linear Gain was the best technique, with similar performance and user experience to the 'at-home' condition, albeit at the expense of a high number of boundary violations and large arm movements. In contrast, AlphaCursor kept users within bounds and minimised arm movement, but suffered from poorer performance and experience. Based on the results, we provide eight guidelines for the use of, and research into, at-a-distance techniques and constrained spaces.

While traditional interfaces in human-computer interaction mainly rely on vision and audio, haptics becomes more and more important. Haptics cannot only increase the user experience and make technology more immersive, it can also transmit information that is hard to interpret only through vision and audio, such as the softness of a surface or other material properties. In this workshop, we aim at discussing how we could interact with technology if haptics is strongly supported and which novel research areas could emerge.

I will present our work into improving passenger journeys using immersive Virtual and Augmented Reality (together XR) to support entertainment, work and collaboration on the move. In Europe, people travel an average of 12,000km per year on private and public transport, in cars, buses, planes and trains. These journeys are often repetitive and wasted time. This total will rise with the arrival of fully autonomous cars, which free drivers to become passengers. The potential to recover this lost time is impeded by 3 significant challenges:

Electrotactile stimulation is a novel form of haptic feedback. There is little work investigating its basic design parameters and how they create effective tactile cues. This paper describes two experiments that extend our knowledge of two key parameters. The first investigated the combination of pulse width and amplitude (Intensity) on sensations of urgency, annoyance, valence and arousal. Results showed significant effects: increasing Intensity caused higher ratings of urgency, annoyance and arousal but reduced valence. We established clear levels for differentiating each sensation. A second study then investigated Intensity and Pulse Frequency to find out how many distinguishable levels could be perceived. Results showed that both Intensity and Pulse Frequency significantly affected perception, with four distinguishable levels of Intensity and two of Pulse Frequency. These results add significant new knowledge about the parameter space of electrotactile cue design and help designers select suitable properties to use when creating electrotactile cues.

This paper investigates how presenting emotionally resonant vibrotactile stimuli at cool, neutral and warm temperature levels impacts mean ratings for emotional resonance and affective response. Affective vibrotactile stimuli can elicit pleasant or calming responses, making them applicable for emotion regulation. Evoking real-world sensations via emotional resonance can widen their affective range and improve their effectiveness, and allow them to enhance immersive multimodal experiences. Thermotactile cues have been shown to affect emotional responses, but have not been combined with emotionally resonant vibrations to see how they change responses to such cues. This study (n=20) assessed the impact of 3 temperature levels (24℃, 30℃, and 34℃) on 15 emotionally resonant vibrotactile cues and observed if emotionally resonant stimuli exceeded the affective range non-resonant vibrotactile stimuli. The findings suggest that presenting specific resonant vibrations at temperatures that are appropriate for the sensation they evoke can improve emotional resonance and vice versa. In addition, temperature had a positive effect on affective response and emotionally resonant vibrations were found to have a wider affective range than traditional vibrotactile cues. These findings support using emotionally resonant vibrations and thermal cues to elicit desirable emotional responses in emotion regulation and immersive media applications.

Haptics is an essential element of interaction between humans and socially assistive robots. However, it is often limited to movements or vibrations and misses key aspects such as temperature. This mixed-methods study explores the potential of enhancing human-robot interaction [HRI] through thermal stimulation to regulate affect during a stress-inducing task. Participants were exposed to thermal stimulation while completing the Mannheim-multicomponent-stress-task (MMST). Findings yielded that human-robot emotional touch may induce comfort and relaxation during the exposure to acute stressors. User affect may be further enhanced through thermal stimulation, which was experienced as comforting, de-stressing, and altered participants’ perception of the robot to be more life-like. Allowing participants to calibrate a temperature they perceived as calming provided novel insights into the temperature ranges suitable for interaction. While neutral temperatures were the most popular amongst participants, findings suggest that cool (4 – 29 ºC), neutral (30 – 32 ºC), and warm (33ºC -36 ºC) temperatures can all induce comforting effects during exposure to stress. The results highlight the potential of thermal HRI in general and, more specifically, the advantages of personalized temperature calibration.

We present PassengXR, an open-source toolkit for creating passenger eXtended Reality (XR) experiences in Unity. XR allows travellers to move beyond the physical limitations of in-vehicle displays, rendering immersive virtual content based on - or ignoring - vehicle motion. There are considerable technical challenges to using headsets in moving environments: maintaining the forward bearing of IMU-based headsets; conflicts between optical and inertial tracking of inside-out headsets; obtaining vehicle telemetry; and the high cost of design given the necessity of testing in-car. As a consequence, existing vehicular XR research typically relies on controlled, simple routes to compensate. PassengXR is a cost-effective open-source in-car passenger XR solution. We provide a reference set of COTS hardware that enables the broadcasting of vehicle telemetry to multiple headsets. Our software toolkit then provides support to correct vehicle-headset alignment, and then create a variety of passenger XR experiences, including: vehicle-locked content; motion- and location-based content; and co-located multi-passenger applications. PassengXR also supports the recording and playback of vehicle telemetry, assisting offline design without resorting to costly in-car testing. Through an evaluation-by-demonstration, we show how our platform can assist practitioners in producing novel, multi-user passenger XR experiences.

Virtual reality (VR) has the potential to induce cybersickness (CS), which impedes CS-susceptible VR users from the benefit of emerging VR applications. To better detect CS, the current study investigated whether/how the newly proposed human vestibular network (HVN) is involved in flagship consumer VR-induced CS by simultaneously recording autonomic physiological signals as well as neural signals generated in sensorimotor and cognitive domains. The VR stimuli were made up of one or two moderate CS-inducing entertaining task(s) as well as a mild CS-inducing cognitive task implemented before and after the moderate CS task(s). Results not only showed that CS impaired cognitive control ability, represented by the degree of attentional engagement, but also revealed that combined indicators from all three HVN domains could together establish the best regression relationship with CS ratings. More importantly, we found that every HVN domain had its unique advantage with the dynamic changes in CS severity and time. These results provide evidence for involvement of the HVN in CS and indicate the necessity of HVN-based CS detection.

This article discusses the Keyboard Augmentation Toolkit (KAT), which supports the creation of virtual keyboards that can be used both for standalone input (e.g., for mid-air text entry) and to augment physically tracked keyboards/surfaces in mixed reality. In a user study, we firstly examine the impact and pitfalls of visualising shortcuts on a tracked physical keyboard, exploring the utility of virtual per-keycap displays. Supported by this and other recent developments in XR keyboard research, we then describe the design, development, and evaluation-by-demonstration of KAT. KAT simplifies the creation of virtual keyboards (optionally bound to a tracked physical keyboard) that support enhanced display —2D/3D per-key content that conforms to the virtual key bounds; enhanced interactivity —supporting extensible per-key states such as tap, dwell, touch, swipe; flexible keyboard mappings that can encapsulate groups of interaction and display elements, e.g., enabling application-dependent interactions; and flexible layouts —allowing the virtual keyboard to merge with and augment a physical keyboard, or switch to an alternate layout (e.g., mid-air) based on need. Through these features, KAT will assist researchers in the prototyping, creation and replication of XR keyboard experiences, fundamentally altering the keyboard’s form and function.

Passengers spend considerable periods of time in shared transit spaces, relying on smartphones and laptops for work. However, these displays are limited in size and ergonomics compared to typical multi-monitor setups used in the office, impairing productivity. Augmented Reality (AR) headsets could provide large, flexible virtual workspaces during travel, enabling passengers to work more efficiently. This paper investigates the factors affecting how passengers choose to layout virtual displays in car, train, subway and plane environments, studying the affordances of each mode of transport and the presence of others. Results from our experiment showed: significant usage of the physical environment to align displays; strong social effects meant avoiding placing displays over other passengers or their belongings; and use of displays for shielding oneself from others. Our findings show the unique challenges posed by the mode of transport and presence of others on the use of AR for mobile productivity in the future.

The story-stem approach is a reliable and valid method in child psychiatry for investigating children’s social understanding, such as in family/peer relationships. Research is on-going into automating the use of story-stems as the approach takes a lot of time and there are often few administrators trained to administer them. A key part of automation is detecting if a child is engaged in the story-stems; if the child is not engaged, then automated techniques will not work. The study focuses on measuring child engagement using observed facial expressions and specifically on understanding the effect of multimedia digital story-stems on their engagement levels to create a better and more engaging digital story-stem. This ensures that psychiatrists who assess story stems with videos will know whether the child involved is engaged with the story-stems and can be assessed properly. The spontaneous facial expressions of 40 children were collected while watching digital story-stems using an RGB video camera and their engagement levels can be automatically classified using observed facial expressions. Meanwhile, we look at different styles of voice (gender and expressiveness) and video format (animation vs live-action recorded) in the story-stems to see how this affected child engagement. The distribution of child engagement across different media types was compared and the results showed that children were more engaged in a story-stem narrated by a female expressive voice. While comparing the two video formats, although children were more engaged in the current version of animated videos, there were still some factors, such as doll appearance, to be investigated for the design of story-stems.

Segmenting audio into homogeneous sections such as music and speech helps us understand the content of audio. It is useful as a pre-processing step to index, store, and modify audio recordings, radio broadcasts and TV programmes. Deep learning models for segmentation are generally trained on copyrighted material, which cannot be shared. Annotating these datasets is time-consuming and expensive and therefore, it significantly slows down research progress. In this study, we present a novel procedure that artificially synthesises data that resembles radio signals. We replicate the workflow of a radio DJ in mixing audio and investigate parameters like fade curves and audio ducking. We trained a Convolutional Recurrent Neural Network (CRNN) on this synthesised data and outperformed state-of-the-art algorithms for music-speech detection. This paper demonstrates the data synthesis procedure as a highly effective technique to generate large datasets to train deep neural networks for audio segmentation.

When users are undertaking mentally demanding visuals tasks, it can be beneficial to convey information through the auditory or tactile modality instead. A fundamental problem when mapping information to sound or vibration is establishing which polarity the mapping should use. Magnitude estimation is a popular method of establishing polarity preferences, however the effectiveness of this approach remains unclear, especially in more ecologically valid contexts. We investigate what impact the polarity of a data-sound or data-vibration mapping has on how well users can interpret these mappings, under two different levels of mental workload. Our results show that polarity does not affect error rate or cognitive workload, although may affect response time. We also found that induced cognitive load may influence usability. An implication of this is that commonly used methods of establishing data mappings need to be revisited, with cognitive load in mind, to help designers create more usable auditory and vibrotactile displays.

This paper presents a survey informing a user-first approach to designing calming affective haptic stimuli by eliciting user preferences in different social scenarios. Prior affective haptics research presented users with stimuli and recorded emotional responses. By contrast this work focuses on the sensations users wish to experience and how these can be simulated using haptics. The survey (n=81) investigated which users preferences in four social situations to reduce social anxiety. Using thematic analysis of responses we created a coding scheme of stimuli derived from real-world experiences to emulate with affective haptics. By cross-referencing these categories with affective haptics research, we provide recommendations to designers about which calming stimuli users wish to experience socially and how they can be implemented.

Augmented / Mixed Reality headsets will in-time see adoption and use in a variety of mobility and transit contexts, allowing users to view and interact with virtual content and displays for productivity and entertainment. However, little is known regarding how multi-display virtual workspaces should be presented in a transit context, nor to what extent the unique affordances of transit environments (e.g. the social presence of others) might influence passenger perception of virtual display layouts. Using a simulated VR passenger airplane environment, we evaluated three different AR-driven virtual display configurations (Horizontal, Vertical, and Focus main display with smaller secondary windows) at two different depths, exploring their usability, user preferences, and the underlying factors that influenced those preferences. We found that the perception of invading other’s personal space significantly influenced preferred layouts in transit contexts. Based on our findings, we reflect on the unique challenges posed by passenger contexts, provide recommendations regarding virtual display layout in the confined airplane environment, and expand on the significant benefits that AR offers over physical displays in said environments.

This paper reports a study investigating the effects of tactile input and output from the steering wheel and the centre console on non-driving task performance. While driving, participants were asked to perform list selection tasks using tactile switches and to experience tactile feedback on either the non-dominant, dominant or both hands as they were browsing the list. Our results show the average duration for selecting an item is 30% shorter when interacting with the steering wheel. They also show a 20% increase in performance when tactile feedback is provided. Our findings reveal that input prevails over output location when designing interaction for drivers. However, tactile feedback on the steering wheel is beneficial when provided at the same location as the input or to both hands. The results will help designers understand the trade-offs of using different interaction locations in the car.

Auditory headsets capable of actively or passively intermixing both real and virtual sounds are in-part acoustically transparent. This paper explores the consequences of acoustic transparency, both on the perception of virtual audio content, given the presence of a real-world auditory backdrop, and more broadly in facilitating a wearable, personal, private, always-available soundspace. We experimentally compare passive acoustically transparent, and active noise cancelling, orientation-tracked auditory headsets across a range of content types, both indoors and outdoors for validity. Our results show differences in terms of presence, realness and externalization for select content types. Via interviews and a survey, we discuss attitudes toward acoustic transparency (e.g. being perceived as safer), the potential shifts in audio usage that might be precipitated by adoption, and reflect on how such headsets and experiences fit within the area of Mixed Reality.

This paper describes a novel category of affective vibrotactile stimuli which evoke real-world sensations and details a study into emotional responses to them. The affective properties of short and abstract vibrotactile waveforms have previously been studied and shown to have a narrow emotional range. By contrast this paper investigated emotional responses to longer waveforms and to emotionally resonant vibrotactile stimuli, stimuli which are evocative of real-world sensations such as animal purring or running water. Two studies were conducted. The first recorded emotional responses to Tactons with a duration of 20 seconds. The second investigated emotional responses to novel emotionally resonant stimuli. Stimuli that users found more emotionally resonant were more pleasant, particularly if they had prior emotional connections to the sensation represented. Results suggest that future designers could use emotional resonance to expand the affective response range of vibrotactile cues by utilising stimuli with which users bear an emotional association.

Haptic feedback can improve safety and driving behaviour. While vibration has been widely studied, other haptic modalities have been neglected. To address this, we present two studies investigating the use of uni- and bimodal vibrotactile and thermal cues on the steering wheel. First, notifications with three levels of urgency were subjectively rated and then identified during simulated driving. Bimodal feedback showed an increased identification time over unimodal vibrotactile cues. Thermal feedback was consistently rated less urgent, showing its suitability for less time critical notifications, where vibration would be unnecessarily attention-grabbing. The second study investigated more complex thermal and bimodal haptic notifications comprised of two different types of information (Nature and Importance of incoming message). Results showed that both modalities could be identified with high recognition rates of up to 92% for both and up to 99% for a single type, opening up a novel design space for haptic in-car feedback.

This work investigates the interplay between Child-Computer Interaction and attachment, a psychological construct that accounts for how children perceive their parents to be. In particular, the article makes use of a multimodal approach to test whether children with different attachment conditions tend to use differently the same interactive system. The experiments show that the accuracy in predicting usage behaviour changes, to a statistically significant extent, according to the attachment conditions of the 52 experiment participants (age-range 5 to 9). Such a result suggests that attachment-relevant processes are actually at work when people interact with technology, at least when it comes to children.

Mixed Reality (MR), Augmented Reality (AR) and Virtual Reality (VR) headsets can improve upon existing physical multi-display environments by rendering large, ergonomic virtual display spaces whenever and wherever they are needed. However, given the physical and ergonomic limitations of neck movement, users may need assistance to view these display spaces comfortably. Through two studies, we developed new ways of minimising the physical effort and discomfort of viewing such display spaces. We first explored how the mapping between gaze angle and display position could be manipulated, helping users view wider display spaces than currently possible within an acceptable and comfortable range of neck movement. We then compared our implicit control of display position based on head orientation against explicit user control, finding significant benefits in terms of user preference, workload and comfort for implicit control. Our novel techniques create new opportunities for productive work by leveraging MR headsets to create interactive wide virtual workspaces with improved comfort and usability. These workspaces are flexible and can be used on-the-go, e.g., to improve remote working or make better use of commuter journeys.

Developing educational tools aimed at children with disabilities is a challenging process for designers and developers because existing methodologies or frameworks do not provide any pedagogical information and/or do not take into account the particular needs of users with some type of impairment. In this study, we propose a framework for the design of tools to support teaching to children with disabilities. The framework provides the necessary stages for the development of tools (hardware-based or software-based) and must be adapted for a specific disability and educational goal. For this study, the framework was adapted to support literacy teaching and contributes to the design of educational/interactive technology for deaf people while making them part of the design process and taking into account their particular needs. The experts' evaluation of the framework shows that it is well structured and may be adapted for other types of disabilities.

This article presents the School Attachment Monitor, a novel interactive system that can reliably administer the Manchester Child Attachment Story Task (a standard psychiatric test for the assessment of attachment in children) without the supervision of trained professionals. Attachment problems in children cause significant mental health issues and costs to society which technology has the potential to reduce. SAM collects, through instrumented doll-play games, enough information to allow a human assessor to manually identify the attachment status of children. Experiments show that the system successfully does this in 87.5% of cases. In addition, the experiments show that an automatic approach based on deep neural networks can map the information collected into the attachment condition of the children. The outcome SAM matches the judgment of expert human assessors in 82.8% of cases. This is the first time an automated tool has been successful in measuring attachment. This work has significant implications for psychiatry as it allows professionals to assess many more children cost effectively and to direct healthcare resources more accurately and efficiently to improve mental health.

Haptic feedback is used in cars to reduce visual inattention. While tactile feedback like vibration can be influenced by the car's movement, thermal and cutaneous push feedback should be independent of such interference. This paper presents two driving simulator studies investigating novel tactile feedback on the steering wheel for navigation. First, devices on one side of the steering wheel were warmed, indicating the turning direction, while those on the other side were cooled. This thermal feedback was compared to audio. The thermal navigation lead to 94.2% correct recognitions of warnings 200m before the turn and to 91.7% correct turns. Speech had perfect recognition for both. In the second experiment, only the destination side was indicated thermally, and this design was compared to cutaneous push feedback. The simplified thermal feedback design did not increase recognition, but cutaneous push feedback had high recognition rates (100% for 200 m warnings, 98% for turns).

Child cyclists are at greater risk for car-to-cyclist accidents than adults. This is in part due to developmental differences in the motor and perceptual-motor abilities of children and adults, and missing cycling infrastructure. To address these issues, we examine unimodal and projection-based techniques to support children in maintaining a good lane position in the absence of bicycle lanes. We present safety-relevant information using unimodal cues: vibration on the handlebar, ambient light in a cycle helmet, projected heads-up display indicators, and on-road laser projection. As a first step, we interviewed twelve children about their cycling issues. We then conducted a lab experiment (N=25) in a bicycle simulator using the unimodal cues in the presence of a visual search task, followed by a controlled test-track experiment (N=15). We found that cycling performance with lane keeping cues was comparable to situations without them, however children found them helpful and expressed subjective preferences for the LED helmet and vibration on the handlebar.

The use of ultrasound haptic feedback for mid-air gestures in cars has been proposed to provide a sense of control over the user's intended actions and to add touch to a touchless interaction. However, the impact of ultrasound feedback to the gesturing hand regarding lane deviation, eyes-off-the-road time (EORT) and perceived mental demand has not yet been measured. This paper investigates the impact of uni- and multimodal presentation of ultrasound feedback on the primary driving task and the secondary gesturing task in a simulated driving environment. The multimodal combinations of ultrasound included visual, auditory, and peripheral lights. We found that ultrasound feedback presented unimodally and bimodally resulted in significantly less EORT compared to visual feedback. Our results suggest that multimodal ultrasound feedback for mid-air interaction decreases EORT whilst not compromising driving performance nor mental demand and thus can increase safety while driving.

Haptic feedback has been widely studied for in-car interactions. However, most of this research has used vibrotactile cues. This paper presents two studies that examine novel thermal feedback for navigation during simulated driving for a lane change task. In the first, we compare the distraction and time differences of audio and thermal feedback. The results show that the presentation of thermal stimuli does not increase lane deviation, but the time needed to complete a lane change increased by 1.82 seconds. In the second study, the influence of variable changes of thermal stimuli on the lane change task performance was tested. We found that the same stimulus design for warm and cold temperatures does not always elicit the same results. Furthermore, variable alterations can have different effects on specified tasks. This suggests that the design of thermal stimuli is highly dependent on what task result should be maximized.

The relationships between sounds and their perceived meaning and connotations are complex, making auditory perception an important factor to consider when designing sonification systems. Listeners often have a mental model of how a data variable should sound during sonification and this model is not considered in most data:sound mappings. This can lead to mappings that are difficult to use and can cause confusion. To investigate this issue, we conducted a magnitude estimation experiment to map how roughness, noise and pitch relate to the perceived magnitude of stress, error and danger. These parameters were chosen due to previous findings which suggest perceptual congruency between these auditory sensations and conceptual variables. Results from this experiment show that polarity and scaling preference are dependent on the data:sound mapping. This work provides polarity and scaling values that may be directly utilised by sonification designers to improve auditory displays in areas such as accessible and mobile computing, process-monitoring and biofeedback.

Acoustic levitation enables a radical new type of human-computer interface composed of small levitating objects. For the first time, we investigate the selection of such objects, an important part of interaction with a levitating object display. We present Point-and-Shake, a mid-air pointing interaction for selecting levitating objects, with feedback given through object movement. We describe the implementation of this technique and present two user studies that evaluate it. The first study found that users could accurately (96%) and quickly (4.1s) select objects by pointing at them. The second study found that users were able to accurately (95%) and quickly (3s) select occluded objects. These results show that Point-and-Shake is an effective way of initiating interaction with levitating object displays.

Room-scale Virtual Reality (VR) has become an affordable consumer reality, with applications ranging from entertainment to productivity. However, the limited physical space available for room-scale VR in the typical home or office environment poses a significant problem. To solve this, physical spaces can be extended by amplifying the mapping of physical to virtual movement (translational gain). Although amplified movement has been used since the earliest days of VR, little is known about how it influences reach-based interactions with virtual objects, now a standard feature of consumer VR. Consequently, this paper explores the picking and placing of virtual objects in VR for the first time, with translational gains of between 1x (a one-to-one mapping of a 3.5m*3.5m virtual space to the same sized physical space) and 3x (10.5m*10.5m virtual mapped to 3.5m*3.5m physical). Results show that reaching accuracy is maintained for up to 2x gain, however going beyond this diminishes accuracy and increases simulator sickness and perceived workload. We suggest gain levels of 1.5x to 1.75x can be utilized without compromising the usability of a VR task, significantly expanding the bounds of interactive room-scale VR.

Secure Attachment relationships have been shown to minimise social and behavioural problems in children and boosts resilience to risks later on such as antisocial behaviour, heart pathologies, and suicide. Attachment assessment is an expensive and time-consuming process that is not often performed. The School Attachment Monitor (SAM) automates Attachment assessment to support expert assessors. It uses doll-play activities with the dolls augmented with sensors and the child's play recorded with cameras to provide data for assessment. Social signal processing tools are then used to analyse the data and to automatically categorize Attachment patterns. This paper presents the current SAM interactive prototype.

An in-car study was conducted to examine different input techniques for list-based scrolling tasks and the effectiveness of haptic feedback for in-car touchscreens. The use of physical switchgear on centre consoles is decreasing which allows designers to develop new ways to interact with in-car applications. However, these new methods need to be evaluated to ensure they are usable. Therefore, three input techniques were tested: direct scrolling, pressure-based scrolling and scrolling using onscreen buttons on a touchscreen. The results showed that direct scrolling was less accurate than using onscreen buttons and pressure input, but took almost half the time when compared to the onscreen buttons and was almost three times quicker than pressure input. Vibrotactile feedback did not improve input performance but was preferred by the users. Understanding the speed vs. accuracy trade-off between these input techniques will allow better decisions when designing safer in-car interfaces for scrolling applications.

This paper presents an investigation into the effects of different feedback modalities on mid-air gesture interaction for infotainment systems in cars. Car crashes and near-crash events are most commonly caused by driver distraction. Mid-air interaction is a way of reducing driver distraction by reducing visual demand from infotainment. Despite a range of available modalities, feedback in mid-air gesture systems is generally provided through visual displays. We conducted a simulated driving study to investigate how different types of multimodal feedback can support in-air gestures. The effects of different feedback modalities on eye gaze behaviour, and the driving and gesturing tasks are considered. We found that feedback modality influenced gesturing behaviour. However, drivers corrected falsely executed gestures more often in non-visual conditions. Our findings show that non-visual feedback can reduce visual distraction significantly

Young children with visual impairments tend to engage less with their surroundings, limiting the benefits from activities at school. We investigated novel ways of using sound from a bracelet, such as speech or familiar noises, to tell children about nearby people, places and activities, to encourage them to engage more during play and help them move independently. We present a series of studies, the first two involving visual impairment educators, that give insight into challenges faced by visually impaired children at school and how sound might help them. We then present a focus group with visually impaired children that gives further insight into the effective use of sound. Our findings reveal novel ways of combining sounds from wearables with sounds from the environment, motivating audible beacons, devices for audio output and proximity estimation. We present scenarios, findings and a design space that show the novel ways such devices could be used alongside wearables to help visually impaired children at school.

Smartphone reminding apps can compensate for memory impairment after acquired brain injury (ABI). In the absence of a caregiver, users must enter reminders themselves if the apps are going to help them. Poor memory and apathy associated with ABI can result in failure to initiate such configuration behaviour and the benefits of reminder apps are lost. ForgetMeNot takes a novel approach to address this problem by periodically encouraging the user to enter reminders with unsolicited prompts (UPs). An in situ case study investigated the experience of using a reminding app for people with ABI and tested UPs as a potential solution to initiating reminder entry. Three people with severe ABI living in a post-acute rehabilitation hospital used the app in their everyday lives for four weeks to collect real usage data. Field observations illustrated how difficulties with motivation, insight into memory difficulties and anxiety impact reminder app use in a rehabilitation setting. Results showed that when 6 UPs were presented throughout the day, reminder-setting increased, showing UPs are an important addition to reminder applications for people with ABI. This study demonstrates that barriers to technology use can be resolved in practice when software is developed with an understanding of the issues experienced by the user group.

This paper explores the use of VR Head Mounted Displays (HMDs) in-car and in-motion for the first time. Immersive HMDs are becoming everyday consumer items and, as they offer new possibilities for entertainment and productivity, people will want to use them during travel in, for example, autonomous cars. However, their use is confounded by motion sickness caused in-part by the restricted visual perception of motion conflicting with physically perceived vehicle motion (accelerations/rotations detected by the vestibular system). Whilst VR HMDs restrict visual perception of motion, they could also render it virtually, potentially alleviating sensory conflict. To study this problem, we conducted the first on-road and in motion study to systematically investigate the effects of various visual presentations of the real-world motion of a car on the sickness and immersion of VR HMD wearing passengers. We established new baselines for VR in-car motion sickness, and found that there is no one best presentation with respect to balancing sickness and immersion. Instead, user preferences suggest different solutions are required for differently susceptible users to provide usable VR in-car. This work provides formative insights for VR designers and an entry point for further research into enabling use of VR HMDs, and the rich experiences they offer, when travelling.

The way drivers operate in-car systems is rapidly changing as traditional physical controls, such as buttons and dials, are being replaced by touchscreens and touch-sensing surfaces. This has the potential to increase driver distraction and error as controls may be harder to find and use. This paper presents an in-car, on the road driving study which examined three key types of input controls to investigate their effects: a physical dial, pressure-based input on a touch surface and touch input on a touchscreen. The physical dial and pressure-based input were also evaluated with and without haptic feedback. The study was conducted with users performing a list-based targeting task using the different controls while driving on public roads. Eye-gaze was recorded to measure distraction from the primary task of driving. The results showed that target accuracy was high across all input methods (greater than 94%). Pressure-based targeting was the slowest while directly tapping on the targets was the faster selection method. Pressure-based input also caused the largest number of glances towards to the touchscreen but the duration of each glance was shorter than directly touching the screen. Our study will enable designers to make more appropriate design choices for future in-car interactions.

This paper explores the combination of multiple concurrent modalities for conveying emotional information in HCI: temperature, vibration and abstract visual displays. Each modality has been studied individually, but can only convey a limited range of emotions within two-dimensional valence-arousal space. This paper is the first to systematically combine multiple modalities to expand the available affective range. Three studies were conducted: Study 1 measured the emotionality of vibrotactile feedback by itself; Study 2 measured the perceived emotional content of three bimodal combinations: vibrotactile + thermal, vibrotactile + visual and visual + thermal. Study 3 then combined all three modalities. Results show that combining modalities increases the available range of emotional states, particularly in the problematic top-right and bottom-left quadrants of the dimensional model. We also provide a novel lookup resource for designers to identify stimuli to convey a range of emotions

Sonification designers have little theory or experimental evidence to guide the design of data-to-sound mappings. Many mappings use acoustic representations of data values which do not correspond with the listener's perception of how that data value should sound during sonification. This research evaluates data-to-sound mappings that are based on psychoacoustic sensations, in an attempt to move towards using data-to-sound mappings that are aligned with the listener's perception of the data value's auditory connotations. Multiple psychoacoustic parameters were evaluated over two experiments, which were designed in the context of a domain-specific problem - detecting the level of focus of an astronomical image through auditory display. Recommendations for designing sonification systems with psychoacoustic sound parameters are presented based on our results.

We demonstrate a technique for rendering textured haptic surfaces in mid-air, using an ultrasound haptic display. Our technique renders tessellated 3D `haptic' shapes with different waveform properties, creating surfaces with distinct perceptions.

We present rhythmic micro-gestures, micro-movements of the hand that are repeated in time with a rhythm. We present a user study that investigated how well users can perform rhythmic micro-gestures and if they can use them eyes-free with non-visual feedback. We found that users could successfully use our interaction technique (97% success rate across all gestures) with short interaction times, rating them as low difficulty as well. Simple audio cues that only convey the rhythm outperformed animations showing the hand movements, supporting rhythmic micro-gestures as an eyes-free input technique.

This demonstration showcases novel multimodal feedback designs for in-car mid-air gesture interaction. It explores the potential of multimodal feedback types for mid-air gestures in cars and how these can reduce eyes-off-the-road time thus make driving safer. We will show four different bimodal feedback combinations to provide effective information about interaction with systems in a car. These feedback techniques are visual-auditory, auditory-ambient (peripheral vision), ambient-tactile, and tactile-auditory. Users can interact with the system after a short introduction, creating an exciting opportunity to deploy these displays in cars in the future.

In this demonstration we show a thermal interaction design on the steering wheel for navigational cues in a car. Participants will be able to use a thermally enhanced steering wheel to follow instructions given in a turn-to-turn based navigation task in a virtual city. The thermal cues will be provided on both sides of the steering wheel and will indicate the turning direction by warming the corresponding side, while the opposite side is being cooled.

Secure Attachment relationships have been shown to minimise social and behavioural problems in children and boosts resilience to risks such as antisocial behaviour, heart pathologies, and suicide later in life. Attachment assessment is an expensive and time-consuming process that is not often performed. The School Attachment Monitor (SAM) automates Attachment assessment to support expert assessors. It uses doll-play activities with the dolls augmented with sensors and the child's play recorded with cameras to provide data for assessment. Social signal processing tools are then used to analyse the data and to automatically categorize Attachment patterns. This paper presents the current SAM interactive prototype.

Acoustic levitation enables new types of human-computer interface, where the content that users interact with is made up from small objects held in mid-air. We show that acoustically-levitated objects can form mid-air widgets that respond to interaction. Users can interact with them using in-air hand gestures. Sound and widget movement are used as feedback about the interaction.

Blind children engage with their immediate environment much less than sighted children, particularly through self-initiated movement or exploration. Research has suggested that providing dynamic feedback about the environment and the child's actions within/against it may help to encourage reaching activity and support spatial cognitive learning. This paper investigated whether the accuracy of peripersonal reaching (space within arm's reach) can be improved by the use of dynamic sound from both the objects to reach for and the reaching hand itself (via a worn speaker). We ran two studies that tested the efficacy of static and dynamic audio feedback designs with blind and visually impaired young people, to identify optimal feedback designs. Study 1 was with young adults aged 18 to 22 and Study 2 involved children aged 12 to 17. The results showed that dynamic audio feedback helps to build spatial connections between the objects and the reaching hand and participants were able to reach more accurately, compared to unchanging feedback.

The way drivers interact with in-car centre console controls is undergoing radical change as physical switchgear are replaced by virtual counterparts with the use of touchscreens. This provides the opportunity to design new input techniques to improve the way on-screen widgets are operated in driving situations. This paper investigates the effectiveness of pressure-based input with haptic feedback as an alternative touch modality for in-car interactions. Two user studies were conducted: one using a driving simulator and the other inside a vehicle driven on public roads, to evaluate two main pressure-based input techniques: positional and rate-based control. The results from a list-based targeting task showed that rate-based control performed well and was comparable to standard touch input and the physical dial while users had difficulties with positional pressure input. These findings from our studies will help engineers make more appropriate design decisions when developing in-car interactions with touchscreens and touch surfaces.

Infotainment Systems can increase mental workload and divert visual attention away from looking ahead on the roads. When these systems give information to the driver, providing it through the tactile channel on the steering wheel might improve driving behaviour and safety. This paper describes an investigation into the perceivability of haptic feedback patterns using an actuated surface on a steering wheel. Six solenoids were embedded along the rim creating three bumps under each palm. A simulated driving study was conducted to test for recognition accuracy of the haptic patterns (81.3%). There was no significant increase in lane deviation or steering angle during haptic pattern presentation. These results suggest that drivers can reliably distinguish between cutaneous patterns presented on the steering wheel. Our findings can assist in delivering non-critical messages to the driver (e.g. driving performance, incoming text messages, etc.) without decreasing driving performance or increasing perceived mental workload.

Thermal stimulation is a feedback channel that has the potential to influence the emotional response of people to media such as images. While previous work has demonstrated that thermal stimuli might have an effect on the emotional perception of images, little is understood about the exact emotional responses different thermal properties and presentation techniques can elicit towards images. This paper presents two user studies that investigate the effect thermal stimuli parameters (e.g. intensity) and timing of thermal stimuli presentation have on the emotional perception of images. We found that thermal stimulation increased valence and arousal in images with low valence and neutral to low arousal. Thermal augmentation of images also reduced valence and arousal in high valence and arousal images. We discovered that depending on when thermal augmentation is presented, it can either be used to create anticipation or enhance the inherent emotion an image is capable of evoking.

When users want to interact with an in-air gesture system, they must first address it. This involves finding where to gesture so that their actions can be sensed, and how to direct their input towards that system so that they do not also affect others or cause unwanted effects. This is an important problem which lacks a practical solution. We present an interaction technique which uses multimodal feedback to help users address in-air gesture systems. The feedback tells them how ("do that") and where ("there") to gesture, using light, audio and tactile displays. By doing that there, users can direct their input to the system they wish to interact with, in a place where their gestures can be sensed. We discuss the design of our technique and three experiments investigating its use, finding that users can "do that" well (93.2%-99.9%) while accurately (51mm-80mm) and quickly (3.7s) finding "there".

There are inherent associations between temperature and emotion in language, cognition and subjective experience [22,42]. However, there exists no systematic mapping of thermal feedback to models of emotion that could be used by designers and users to convey a range of emotions in HCI. A common way of classifying emotions and quantifying emotional experience is through ratings along valence and arousal dimensions, originating from Russell's circumplex model [32]. Therefore, the research in this paper mapped subjective ratings of a range of thermal stimuli to the circumplex model to understand the range of emotions that might be conveyed through thermal feedback. However, as the suitability of the model varies depending on the type of emotional stimuli [31], we also compared the goodness of fit of ratings between the circumplex and vector [8,31] models of emotion. The results showed that thermal feedback was interpreted as representing a limited range of emotions concentrated in just two quadrants or categories of the circumplex: high valence, low arousal and low valence, high arousal. Warm stimuli were perceived as more pleasant/positive than cool stimuli and altering either the rate or extent of temperature change affected both valence and arousal axes simultaneously. The results showed a significantly better fit to a vector model than to the circumplex.

Visually impaired children often have difficulty with everyday activities like locating items, e.g. favourite toys, and moving safely around the home. It is important to assist them during activities like these because it can promote independence from adults and helps to develop skills. Our demonstration shows our work towards a multimodal sensing and output system that adapts the lighting conditions at home to help visually impaired children with such tasks.

In this paper we describe a demonstration of our multimodal affective feedback designs, used in research to expand the emotional expressivity of interfaces. The feedback leverages inherent associations and reactions to thermal, vibrotactile, auditory and abstract visual designs to convey a range of affective states without any need for learning feedback encoding. All combinations of the different feedback channels can be utilised, depending on which combination best conveys a given state. All the signals are generated from a mobile phone augmented with thermal and vibrotactile stimulators, which will be available to conference visitors to see, touch, hear and, importantly, feel.

Auditory interfaces offer a solution to the problem of effective eyes-free mobile interactions. In this article, we investigate the use of multilevel auditory displays to enable eyes-free mobile interaction with indoor location-based information in non-guided audio-augmented environments. A top-level exocentric sonification layer advertises information in a gallery-like space. A secondary interactive layer is used to evaluate three different conditions that varied in the presentation (sequentialversussimultaneous) and spatialisation (non-spatialisedversusegocentric/exocentric spatialisation) of multiple auditory sources. Our findings show that (1) participants spent significantly more time interacting with spatialised displays; (2) using the same design for primary and interactive secondary display (simultaneous exocentric) showed a negative impact on the user experience, an increase in workload and substantially increased participant movement; and (3) the other spatial interactive secondary display designs (simultaneous egocentric, sequential egocentric, and sequential exocentric) showed an increase in time spent stationary but no negative impact on the user experience, suggesting a more exploratory experience. A follow-up qualitative and quantitative analysis of user behaviour support these conclusions. These results provide practical guidelines for designing effective eyes-free interactions for far richer auditory soundscapes.

Smartphone reminder applications (apps) have the potential to help people with memory impairment after acquired brain injury (ABI) to perform everyday tasks. Issues impacting the uptake of reminder apps for this group are still poorly understood. To address this, three focus groups were held with people with memory impairments after ABI and ABI caregivers (N=12). These involved a discussion about perceptions of, and attitudes towards, reminder apps combined with usability reflections during a user-centred design session (Keep Lose Change) after a walkthrough of an existing reminder app -- Google Calendar. Framework analysis revealed six key themes that impact uptake of reminder apps; Perceived Need, Social Acceptability, Experience/Expectation, Desired Content and Functions, Cognitive Accessibility and Sensory/Motor Accessibility. Analysis of themes revealed issues that should be considered by designers and researchers when developing and testing reminding software for people with memory impairment following ABI.

Blind children engage with their immediate environment much less than sighted children, particularly through self-initiated movement or exploration. Research has suggested that providing dynamic feedback about the environment and the child--s actions within/against it may help to encourage reaching activity and support spatial cognitive learning. This paper presents an initial study suggesting the accuracy of peripersonal reaching can be improved by the use of dynamic sound from both the objects to reach for and the reaching hand itself (via a worn speaker) that changes based on the proximity of the hand to the object. The demonstration will let attendees try the interaction and feedback designs

This paper presents a first evaluation of multimodal language-based warnings for handovers of control in autonomous cars. A set of possible handover situations varying in urgency is described. A set of multimodal, language-based warnings for these situations is then introduced. All combinations of audio, tactile and visual warnings for handovers were evaluated in terms of perceived urgency, annoyance and alerting effectiveness. Results showed clear recognition of the warning urgency in this new context, as well as low perceived annoyance overall, and higher perceived effectiveness for critical warnings. The time of transition from self-driving to manual mode in the presence of the warnings was then evaluated. Results showed quicker transitions for highly urgent warnings and poor driving performance for unimodal visual warnings. These results provide a novel set of guidelines for an effective transition of control between car and driver in an autonomous vehicle.

This paper examines the response of 10 typical and 10 autism spectrum participants in their reaction to a set of uni- and multisensory warning signals designed to indicate different levels of urgency. The warnings were composed of auditory, visual and tactile signals that were presented alone or in combination. Two experiments were conducted, a first that examined perceived urgency and annoyance with the warnings and a second that used a driving simulator scenario to explore recognition of the level of urgency and the speed of response. Results of Experiment 1 showed that there was no difference between groups in the perceived urgency of the warning signals, though the autism spectrum group reported less annoyance with the signals. Results of Experiment 2 showed that while both groups showed high accuracy in correctly reporting urgency level, the autism spectrum group performed better. Moreover, the fastest overall reaction times obtained were by the autism spectrum group when the warning included a visual component, with vision alone producing the quickest response. These results provide novel empirical insights on behaviour of drivers with autism when exposed to multimodal driver displays. They also highlight how consideration of characteristics of individual differences can contribute to the design of effective warning signals.

We identify usability challenges facing consumers adopting Virtual Reality (VR) head-mounted displays (HMDs) in a survey of 108 VR HMD users. Users reported significant issues in interacting with, and being aware of their real-world context when using a HMD. Building upon existing work on blending real and virtual environments, we performed three design studies to address these usability concerns. In a typing study, we show that augmenting VR with a view of reality significantly corrected the performance impairment of typing in VR. We then investigated how much reality should be incorporated and when, so as to preserve users' sense of presence in VR. For interaction with objects and peripherals, we found that selectively presenting reality as users engaged with it was optimal in terms of performance and users' sense of presence. Finally, we investigated how this selective, engagement-dependent approach could be applied in social environments, to support the user's awareness of the proximity and presence of others.

Multimodal displays are increasingly being utilized as driver warnings. Abstract warnings, without any semantic association to the signified event, and language-based warnings are examples of such displays. This paper presents a first comparison between these two types, across all combinations of audio, visual and tactile modalities. Speech, text and Speech Tactons (a novel form of tactile warnings synchronous to speech) were compared to abstract pulses in two experiments. Results showed that recognition times of warning urgency during a non-critical driving situation were shorter for abstract warnings, highly urgent warnings and warnings including visual feedback. Response times during a critical situation were shorter for warnings including audio. We therefore suggest abstract visual feedback when informing drivers during a non-critical situation and audio in a highly critical one. Language-based warnings during a critical situation performed equally well as abstract ones, so they are suggested as less annoying vehicle alerts.

Research has shown that thermal feedback can be an engaging and convincing means of conveying experimenter-predefined meanings, e.g., material properties or message types. However, thermal perception is subjective and its meaning in interaction can be ambiguous. Interface designers may not be sure how users could naïvely interpret thermal feedback during interaction. Little is also known about how users would choose thermal cues to convey their own meanings. The research in this paper tested subjective interpretations of thermal stimuli in three different scenarios: social media activity, a colleague's presence and the extent of use of digital content. Participants were also asked to assign their own thermal stimuli to personal experiences, to help us understand what kinds of stimuli people associate with different meanings. The results showed strong agreement among participants concerning what warmth (presence, activity, quality) and cool mean (absence, poor quality). Guidelines for the design of thermal feedback are presented to help others create effective thermal interfaces.

While mid-air gestures offer new possibilities to interact with or around devices, some situations, such as interacting with applications, playing games or navigating, may require visual attention to be focused on a main task. Ultrasonic haptic feedback can provide 3D spatial haptic cues that do not demand visual attention for these contexts. In this paper, we present an initial study of active exploration of ultrasonic haptic virtual points that investigates the spatial localization with and without the use of the visual modality. Our results show that, when providing haptic feedback giving the location of a widget, users perform 50% more accurately compared to providing visual feedback alone. When provided with a haptic location of a widget alone, users are more than 30% more accurate than when given a visual location. When aware of the location of the haptic feedback, active exploration decreased the minimum recommended widget size from 2cm2to 1cm2when compared to passive exploration from previous studies. Our results will allow designers to create better mid-air interactions using this new form of haptic feedback.

In this paper we present the results of a study investigating the control of simultaneous non-dominant hand (NDH) pressure input and dominant hand (DH) pinch, swipe and rotate gesture input on a touchscreen tablet device. We investigate the effect of NDH pressure input on DH gesture accuracy, and vice versa, to support the design of bimanual interaction techniques for tablet devices. Our results suggest that pressure negatively affects DH touchscreen gesture performance; however, pressure targeting accuracy for each gesture was still high (86% overall). Our results show that participants were able to achieve high levels of accuracy using DH swipe gestures (the simplest gesture in the study) suggesting that the ability to perform a simultaneous combination of pressure and touchscreen gesture input depends on the complexity of the dominant hand action involved.

In this paper, we investigate the effects of mobility and encumbrance (holding objects such as shopping bags) on standard gestures commonly performed on touchscreens: tapping, dragging, spreading & pinching and rotating clockwise & anticlockwise when completed using a two-handed input posture. These one- and two- finger on-screen gesture inputs have become common but previous research has only examined tapping performance in everyday walking and encumbered situations. Therefore, a series of Fitts' Law style targeting tasks was designed to measure the performance of each gesture with users walking only and walking while carrying bags. The results showed that encumbrance and walking had a negative impact on each gesture in terms of accuracy except for rotational actions, which were performed well. Tapping and dragging both performed poorly which shows the input difficulties of single finger interactions when encumbered and on the move. Our findings will help designers choose the appropriate input techniques for future mobile user interfaces and apps in physically demanding contexts.

This paper describes two experiments evaluating a set of speech and tactile driver warnings. Six speech messages of three urgency levels were designed, along with their tactile equivalents, Speech Tactons. These new tactile warnings retained the rhythm of speech and used different levels of roughness and intensity to convey urgency. The perceived urgency, annoyance and alerting effectiveness of these warnings were evaluated. Results showed that bimodal (audio and tactile) warnings were rated as more urgent, more annoying and more effective compared to unimodal ones (audio or tactile). Perceived urgency and alerting effectiveness decreased along with the designed urgency, while perceived annoyance was lowest for warnings of medium designed urgency. In the tactile modality, ratings varied less as compared to the audio and audiotactile modalities. Roughness decreased and intensity increased ratings for Speech Tactons in all the measures used. Finally, Speech Tactons produced acceptable recognition accuracy when tested without their speech counterparts. These results demonstrate the utility of Speech Tactons as a new form of tactile alert while driving, especially when synchronized with speech.

Increasing amounts of geo-tagged social media have led to interest in how that media can be re-integrated into the physical environment. Yet, although location information is often automatically appended to media, little is know about how users consider location in its creation and viewing. Using Graffiti as a design meme, we developed a novel social media service to investigate these issues. A two week field study showed how users incorporated both utilitarian and playful aspects of location into their social media creation, as well as revealing a disconnect between the location-media relationship intended by creators and perceived by viewers. We outline implications of our work for services that seek to repurpose existing geo-tagged social media in the design of novel services.

A novel investigation of pressure input is presented where it is characterised as a transient modality, one that has a natural inverse, bounce-back and a state that only persists during interaction. Three empirical studies are described that evaluate pressure for use as a non-dominant hand input modality, where the ability to target and maintain pressure while simultaneously performing a dominant-hand targeting task is investigated. Pressure accuracy was high (93%) and the impact on dominant-hand targeting was low. Mean pressure accuracy when selecting targets by releasing pressure was also high (89%) as was selecting targets by applying pressure from a non-zero starting point (94.4%). The ability to accurately maintain pressure over time was better with larger target pressures. Example applications and design guidelines are presented that enable designers to exploit the transient properties of pressure input in interaction design.

In this paper, we investigate the effects of encumbrance (carrying typical objects such as shopping bags during interaction) and walking on target acquisition on a touchscreen mobile phone. Users often hold objects and use mobile devices at the same time and we examined the impact encumbrance has on one- and two- handed interactions. Three common input postures were evaluated: two-handed index finger, one-handed preferred thumb and two-handed both thumbs, to assess the effects on performance of carrying a bag in each hand while walking. The results showed a significant decrease in targeting performance when users were encumbered. For example, input accuracy dropped to 48.1% for targeting with the index finger when encumbered, while targeting error using the preferred thumb to input was 4.2mm, an increase of 40% compared to unencumbered input. We also introduce a new method to evaluate the user's preferred walking speed when interacting - PWS&I, and suggest future studies should use this to get a more accurate measure of the user's input performance.

Previous studies have investigated audio, visual and tactile driver warnings, indicating the importance of communicating the appropriate level of urgency to the drivers. However, these modalities have never been combined exhaustively and tested under conditions of varying situational urgency to assess their effectiveness both in the presence and absence of critical driving events. This paper describes an experiment evaluating all multimodal combinations of such warnings under two contexts of situational urgency: a lead car braking and not braking. The results showed that participants responded quicker to more urgent warnings, especially in the presence of a car braking. They also responded faster to the multimodal as opposed to unimodal signals. Driving behaviour improved in the presence of the warnings and the absence of a car braking. These results highlight the influence of urgency and number of modalities in warning design and indicate the utility of non-visual warnings in driving.

Ultrasonic haptic feedback is a promising means of providing tactile sensations in mid-air without encumbering the user with an actuator. However, controlled and rigorous HCI research is needed to understand the basic characteristics of perception of this new feedback medium, and so how best to utilise ultrasonic haptics in an interface. This paper describes two experiments conducted into two fundamental aspects of ultrasonic haptic perception: 1) localisation of a static point and 2) the perception of motion. Understanding these would provide insight into 1) the spatial resolution of an ultrasonic interface and 2) what forms of feedback give the most convincing illusion of movement. Results show an average localisation error of 8.5mm, with higher error along the longitudinal axis. Convincing sensations of motion were produced when travelling longer distances, using longer stimulus durations and stimulating multiple points along the trajectory. Guidelines for feedback design are given.

Above-device gesture interfaces let people interact in the space above mobile devices using hand and finger movements. For example, users could gesture over a mobile phone or wearable without having to use the touchscreen. We look at how above-device interfaces can also give feedback in the space over the device. Recent haptic and wearable technologies give new ways to provide tactile feedback while gesturing, letting touchless gesture interfaces give touch feedback. In this paper we take a first detailed look at how tactile feedback can be given during above-device interaction. We compare approaches for giving feedback (ultrasound haptics, wearables and direct feedback) and also look at feedback design. Our findings show that tactile feedback can enhance above-device gesture interfaces.

Gestures above a mobile phone would let users interact with their devices quickly and easily from a distance. While both researchers and smartphone manufacturers develop new gesture sensing technologies, little is known about how best to design these gestures and interaction techniques. Our research looks at creating usable and socially acceptable above-device interaction techniques. We present an initial gesture collection, a preliminary evaluation of these gestures and some design recommendations. Our findings identify interesting areas for future research and will help designers create better gesture interfaces.

In this paper, two walking evaluation methods were compared to evaluate the effects of encumbrance while the preferred walking speed (PWS) is controlled. Users frequently carry cumbersome objects (e.g. shopping bags) and use mobile devices at the same time which can cause interaction difficulties and erroneous input. The two methods used to control the PWS were: walking on a treadmill and walking around a predefined route on the ground while following a pacesetter. The results from our target acquisition experiment showed that for ground walking at 100% of PWS, accuracy dropped to 36% when carrying a bag in the dominant hand while accuracy reduced to 34% for holding a box under the dominant arm. We also discuss the advantages and limitations of each evaluation method when examining encumbrance and suggest treadmill walking is not the most suitable approach to use if walking speed is an important factor in future mobile studies.

Pressing a virtual button is still the major interaction method in touchscreen mobile phones. Although phones are becoming more and more powerful, operating system software is getting more and more complex, causing latency in interaction. We were interested in gaining insight into touch-feedback simultaneity and the effects of latency on the perceived quality of touchscreen buttons. In an experiment, we varied the latency between touch and feedback between 0 and 300 ms for tactile, audio, and visual feedback modalities. We modelled the proportion of simultaneity perception as a function of latency for each modality condition. We used a Gaussian model fitted with the maximum likelihood estimation method to the observations. These models showed that the point of subjective simultaneity (PSS) was 5ms for tactile, 19ms for audio, and 32ms for visual feedback. Our study included the scoring of perceived quality for all of the different latency conditions. The perceived quality dropped significantly between latency conditions 70 and 100 ms when the feedback modality was tactile or audio, and between 100 and 150 ms when the feedback modality was visual. When the latency was 300ms for all feedback modalities, the quality of the buttons was rated significantly lower than in all of the other latency conditions, suggesting that a long latency between a touch on the screen and feedback is problematic for users. Together with PSS and these quality ratings, a 75% threshold was established to define a guideline for the recommended latency range between touch and feedback. Our guideline suggests that tactile feedback latency should be between 5 and 50 ms, audio feedback latency between 20 and 70 ms, and visual feedback latency between 30 and 85 ms. Using these values will ensure that users will perceive the feedback as simultaneous with the finger's touch. These values also ensure that the users do not perceive reduced quality. These results will guide engineers and designers of touchscreen interactions by showing the trade-offs between latency and user preference and the effects that their choices might have on the quality of the interactions and feedback they design.

Previous studies have evaluated Audio, Visual and Tactile warnings for drivers, highlighting the importance of conveying the appropriate level of urgency through the signals. However, these modalities have never been combined exhaustively with different urgency levels and tested while using a driving simulator. This paper describes two experiments investigating all multimodal combinations of such warnings along three different levels of designed urgency. The warnings were first evaluated in terms of perceived urgency and perceived annoyance in the context of a driving simulator. The results showed that the perceived urgency matched the designed urgency of the warnings. More urgent warnings were also rated as more annoying but the effect of annoyance was lower compared to urgency. The warnings were then tested for recognition time when presented during a simulated driving task. It was found that warnings of high urgency induced quicker and more accurate responses than warnings of medium and of low urgency. In both studies, the number of modalities used in warnings (one, two or three) affected both subjective and objective responses. More modalities led to higher ratings of urgency and annoyance, with annoyance having a lower effect compared to urgency. More modalities also led to quicker responses. These results provide implications for multimodal warning design and reveal how modalities and modality combinations can influence participant responses during a simulated driving task.

Multimodal interaction can make home care reminder systems more accessible to their users, most of whom are older and/or have sensory impairments. Existing research into the properties of different notification modalities have used younger participants rather than members of the older population at which they are aimed. This paper presents the results of a user study with older adults that examined how different notification modalities affected (a) performance in a card matching game and (b) how effective the different modalities were at delivering information. Participants were all aged over 50 and notifications were delivered using textual, pictographic, abstract visual, speech, Earcon, Auditory Icon, tactile and olfactory modalities while playing the game. The results showed that older users were influenced by the same factors as younger users, yet there were subjective differences. The implications for the design of multimodal reminder systems for home care are discussed.

The user experience of performing gesture-based interactions in public spaces is highly dependent on context, where users must decide which gestures they will use and how they will perform them. In order to complete a realistic evaluation of how users make these decisions, the evaluation of such user experiences must be completed "in the wild." Furthermore, studies need to be completed within different cultural contexts in order to understand how users might adopt gesture differently in different cultures. This paper presents such a study using a mobile gesture-based game, where users in the UK and India interacted with this game over the span of 6 days. The results of this study demonstrate similarities between gesture use in these divergent cultural settings, illustrate factors that influence gesture acceptance such as perceived size of movement and perceived accuracy, and provide insights into the interaction design of mobile gestures when gestures are distributed across the body.

Thermal feedback is a new area of research in HCI and, as such, there has been very little investigation of the impact of environmental factors on its use for interaction. To address this shortcoming we conducted an experiment to investigate how ambient temperature and humidity could affect the usability of thermal feedback. If environmental conditions affect perception significantly, then it may not be suitable for mobile interactions. Evaluations were conducted outdoors in varying environmental conditions over a period of 5 months. Results showed that the ambient temperature has a significant impact on people's ability to detect stimuli and also their perception of these stimuli. Humidity has a negligible effect for most humidity values. Despite this, previous thermal feedback design recommendations still hold in varying temperatures and humidities showing that thermal feedback is a useful tool for mobile interaction.

We present PULSE, a mobile application designed to allow users to gain a 'vibe', an intrinsic understanding of the people, places and activities around their current location, derived from messages on the Twitter social networking site. We compared two auditory presentations of the vibe. One presented message metadata implicitly through modification of spoken message attributes. The other presented the same metadata, but through additional auditory cues. We compared both techniques in a lab and real world study. Additional auditory cues were found to allow for smaller changes in metadata to be more accurately detected, but were least preferred when PULSE was used in context. Results also showed that PULSE enhanced and shaped user understanding, with audio presentation allowing a closer coupling of digital data to the physical world.

Reminder systems are a specific range of technologies for care at home that can deliver notifications or reminders (such as 'take your medication') to assist with daily living. How to best deliver these reminders is an interesting research challenge. Today's technologies have the potential to deliver the notifications in a range of output modalities. The delivery methods can be selected both by the system (depending on what devices are available in the home) and on the users' needs, capabilities and preferences. This paper describes a user-centred approach to the design of multimodal output for reminders in the home. In a focus group study (N=15), this paper explores how six output modalities could be used to present reminders in a home setting. The results demonstrate user requirements that can be incorporated into the early phases of the design of a multimodal reminder system.

This paper expands the repertoire of non-visual feedback for mobile interaction, established through Earcons and Tactons, by designing structured thermal cues for conveying information. Research into the use of thermal feedback for HCI has not looked beyond basic 'yes-no' detection of stimuli to the unique identification of those stimuli. We first designed thermal icons that varied along two parameters to convey two pieces of information. We also designed intramodal tactile icons, combining one thermal and one vibrotactile parameter, to test perception of different tactile cues and so evaluate the possibility of augmenting vibrotactile displays with thermal feedback. Thermal icons were identified with 82.8% accuracy, while intramodal icons had 96.9% accuracy, suggesting thermal icons are a viable means of conveying information in mobile HCI, for when audio and/or vibrotactile feedback is not suitable or desired.

There is growing interest internationally in the development and effective integration of efficient and useable electronic health information management systems. However, it is essential that researchers, designers and implementers have a clear understanding of the current care processes in place before they start to develop or implement new electronic systems within health care settings. A critical first step to designing such systems is the development of accurate models depicting existing care pathways and clinical processes. Here, we introduce the concepts of process-mapping, and discuss how these tools are currently being applied to a comprehensive process mapping project of pre-operative assessment services across Scotland.

While there has been a lot of research on the usability of reminders and alarms in the work context, the home has been somewhat neglected despite the importance of reminder systems for telecare and assistive living systems. We conducted a comprehensive mixed-methods study into the requirements for useable and acceptable reminders in the home. The study consisted of a questionnaire (N=379), 6 focus groups, and 7 home tour interviews. Our results highlight the need for highly flexible and contextualized multimodal and multi-device reminder solutions that build on existing successful strategies for remembering in and around the home. We suggest that developers of home care reminder systems should design for diversity, context, priorities, autonomy, shared spaces, and optimal care.

In this paper we argue that Musicons, short samples from pieces of music are a useful way to present private but memorable reminder messages. We investigated accuracy, memorability and response times for short, medium, and long Musicons. User performance on the Musicons was also compared to short spoken reminders. The study consisted of two sessions a week apart. Quantitative measures were augmented with qualitative questions about associations and memories. Overall, participants achieved a high level of accuracy (89%) on the Musicons. Recognition was stable at 90% or better across sessions for users who were able to construct meaningful links between Musicons and the associated tasks. Optimal response times were achieved for medium-length 0.5 sec. Musicons. We conclude that Musicons are a viable option for alarms and notifications that combine the high learnability of Auditory Icons with the more private nature of Earcons.

As mobile devices increase in functionality, users perform more tasks when on the move. Spatial audio interfaces offer a solution for eyes-free interaction. However, such interfaces face a number of challenges when supporting multiple and simultaneous tasks, namely: 1) interference amongst multiple audio streams, and 2) the constraints of cognitive load. We present a comparative study of spatial audio techniques evaluated in a divided- and selective-attention task. A podcast was used for high cognitive load (divided-attention) and classical music for low cognitive load (selective-attention), while interacting with an audio menu. Results showed that spatial audio techniques were preferred when cognitive load was kept low, while a baseline technique using an interruptible single audio stream was significantly less preferred. Conversely, when cognitive load was increased the preferences reversed. Thus, given an appropriate task structure, spatial techniques offer a means of designing effective audio interfaces to support eyes-free mobile multitasking.

Thermal stimulation is a rich, emotive and salient feedback channel that is well suited to HCI, but one that is yet to be fully investigated. Thermal feedback may be suited to environments that are too loud for audio or too bumpy for vibrotactile feedback. This paper presents two studies into how well users could detect hot and cold stimuli presented to the fingertips, the palm, the dorsal surface of the forearm and the dorsal surface of the upper arm. Evaluations were carried out in static and mobile settings. Results showed that the palm is most sensitive, cold is more perceivable and comfortable than warm and that stronger and faster-changing stimuli are more detectable but less comfortable. Guidelines for the design of thermal feedback are outlined, with attention paid to perceptual and hedonic factors.

Awareness of other people's activity is an important part of shared-workspace collaboration, and is typically supported using visual awareness displays such as radar views. These visual presentations are limited in that the user must be able to see and attend to the view in order to gather awareness information. Using audio to convey awareness information does not suffer from these limitations, and previous research has shown that audio can provide valuable awareness in distributed settings. In this paper we evaluate the effectiveness of synthesized dynamic audio information, both on its own and as an adjunct to a visual radar view. We developed a granular-synthesis engine that produces realistic chalk sounds for off-screen activity in a groupware workspace, and tested the audio awareness in two ways. First, we measured people's ability to identify off-screen activities using only sound, and found that people are almost as accurate with synthesized sounds as with real sounds. Second, we tested dynamic audio awareness in a realistic groupware scenario, and found that adding audio to a radar view significantly improved awareness of off-screen activities in situations where it was difficult to see or attend to the visual display. Our work provides new empirical evidence about the value of dynamic synthesized audio in distributed groupware.

Thermal feedback is a new area of research in HCI. To date, studies investigating thermal feedback for interaction have focused on virtual reality, abstract uses of thermal output or on use in highly controlled lab settings. This paper is one of the first to look at how environmental factors, in our case clothing, might affect user perception of thermal feedback and therefore usability of thermal feedback. We present a study into how well users perceive hot and cold stimuli on the hand, thigh and waist. Evaluations were carried out with cotton and nylon between the thermal stimulators and the skin. Results showed that the presence of clothing requires higher intensity thermal changes for detection but that these changes are more comfortable than direct stimulation on skin.

Multimodal interaction can be used to make home care technology more effective and appropriate, particularly for people with sensory impairments. Previous work has revealed how disruptive notifications in different modalities are to a home-based task, but has not investigated how disruptive unwanted notifications might be. An experiment was conducted which evaluated the disruptive effects of unwanted notifications when delivered in textual, pictographic, abstract visual, speech, earcon, auditory icon, tactile and olfactory modalities. It was found that for all the modalities tested, both wanted and unwanted notifications produced similar reductions in error rate and task success, independent of modality. The results demonstrate the need to control and limit the number of unwanted notifications delivered in the home and contribute to a large body of work advocating the inclusion of multiple interaction modalities.

This paper presents a study that explores the issues of mobile multimodal interactions while on the move in the real world. Because multimodal interfaces allow new kinds of eyes and hands free interactions, usability issues while moving through different public spaces becomes an important issue in user experience and acceptance of multimodal interaction. This study focuses on these issues by deploying an RSS reader that participants used during their daily commute every day for one week. The system allows users on the move to access news feeds eyes free through head- phones playing audio and speech and hands free through wearable sensors attached to the wrists. The results showed participants were able to interact with the system on the move and became more comfortable performing these interactions as the study progressed. Users were also far more comfortable gesturing on the street than on public transport, which was reflected in the number of interactions and the perceived social acceptability of the gestures in different contexts.

Studies have reported negative effects of walking on mobile human---computer interaction when compared to standing or sitting. However, the quantitative relationship between walking speed and user performance is unknown. In the study described here, we varied walking speed on a treadmill and measured effects on discrete aiming movements on a touchscreen interface. Their relationship was found to be non-linear with a local optimum: when walking at 40--80% of one's preferred walking speed (PWS), target acquisition performance plateaus, indicating optimal trade-off between speed and interaction. Accelerometer data showed that, despite increasing hand oscillation, users were able to maintain stable interaction performance at 74% of PWS. Interestingly, this speed coincides with the speed users spontaneously walk when interacting with a mobile device.

"In all the wonderful worlds that writing opens, the spoken word still resides and lives. Written texts all have to be related somehow, directly or indirectly, to the world of sound, the natural habitat of language, to yield their meanings."

This paper presents a study looking into the effects of walking and the use of visual and audio feedback on the application of pressure for linear targeting. Positional and Rate-based control methods are compared in order to determine which allows for more stable and accurate selections, both while sitting and mobile. Results suggest that Rate-based control is superior for both mobile (walking) and static (sitting) linear targeting, and that mobility significantly increases errors, selection time and subjective workload. The use of only audio feedback significantly increased errors and task time for Positional control and static Rate-based control, but not mobile Rate-based control. Despite this, the results still suggest that audio control of pressure interaction while walking is highly accurate and usable.

We present McSig, a multimodal system for teaching blind children cursive handwriting so that they can create a personal signature. For blind people handwriting is very difficult to learn as it is a near-zero feedback activity that is needed only occasionally, yet in important situations; for example, to make an attractive and repeatable signature for legal contracts. McSig aids the teaching of signatures by translating digital ink from the teacher's stylus gestures into three non-visual forms: (1) audio pan and pitch represents the x and y movement of the stylus; (2) kinaesthetic information is provided to the student through a force-feedback haptic pen that mimics the teacher's stylus movement; and (3) a physical tactile line on the writing sheet is created by the haptic pen. McSig has been developed over two major iterations of design, usability testing and evaluation. The final step of the first iteration was a short evaluation with eight visually impaired children. The results suggested that McSig had the highest potential benefit for congenitally and totally blind children and also indicated some areas where McSig could be enhanced. The second prototype incorporated significant modifications to the system, improving the audio, tactile and force-feedback. We then ran a detailed, longitudinal evaluation over 14 weeks with three of the congenitally blind children to assess McSig's effectiveness in teaching the creation of signatures. Results demonstrated the effectiveness of McSig—they all made considerable progress in learning to create a recognizable signature. By the end of ten lessons, two of the children could form a complete, repeatable signature unaided, the third could do so with a little verbal prompting. Furthermore, during this project, we have learnt valuable lessons about providing consistent feedback between different communications channels (by manual interactions, haptic device, pen correction) that will be of interest to others developing multimodal systems.

We report the results of an exploratory 8-day field study of CrossTrainer: a mobile game with crossmodal audio and tactile feedback. Our research focuses on the longitudinal effects on performance with audio and tactile feedback, the impact of context such as location and situation on performance and personal modality preference. The results of this study indicate that crossmodal feedback can aid users in entering answers quickly and accurately using a variety of different widgets. Our study shows that there are times when audio is more appropriate than tactile and vice versa and for this reason devices should support both tactile and audio feedback to cover the widest range of environments, user preference, locations and tasks.

We present a tangible user interface (TUI) called Tangible Graph Builder, that has been designed to allow visually impaired users to access graph and chart-based data. We describe the current paper-based materials used to allow independent graph construction and browsing, before discussing how researchers have applied virtual haptic and non-speech audio techniques to provide more flexible access. We discuss why, although these technologies overcome many of the problems of non-visual graph access, they also introduce new issues and why the application of TUIs is important. An evaluation of Tangible Graph Builder with 12 participants (8 sight deprived, 4 blind) revealed key design requirements for non-visual TUIs, including phicon design and handling marker detection failure. We finish by presenting future work and improvements to our system.

Gesture-based mobile interfaces require users to change the way they use technology in public settings. Since mobile phones are part of our public appearance, designers must integrate gestures that users perceive as acceptable for public use. This topic has received little attention in the literature so far. The studies described in this paper begin to look at the social acceptability of a set of gestures with respect to location and audience in order to investigate possible ways of measuring social acceptability. The results of the initial survey showed that location and audience had a significant impact on a user's willingness to perform gestures. These results were further examined through a user study where participants were asked to perform gestures in different settings (including a busy street) over repeated trials. The results of this work provide gesture design recommendations as well as social acceptability evaluation guidelines.

We describe a virtual "tether" for mobile devices that allows groups to have quick, simple and privacy-preserving meetups. Our design provides cues which allow dynamic coordination of rendezvous without revealing users' positions. Using accelerometers and magnetometers, combined with GPS positioning and non-visual feedback, users can probe and sense a dynamic virtual object representing the nearest meeting point. The Social Gravity system makes social bonds tangible in a virtual world which is geographically grounded, using haptic feedback to help users rendezvous. We show dynamic navigation using this physical model-based system to be efficient and robust in significant field trials, even in the presence of low-quality positioning. The use of simulators to build models of mobile geolocated systems for pre-validation purposes is discussed, and results compared with those from our trials. Our results show interesting behaviours in the social coordination task, which lead to guidelines for geosocial interaction design. The Social Gravity system proved to be very successful in allowing groups to rendezvous efficiently and simply and can be implemented using only commercially available hardware.

Camera phones are now very common but there are some usability issues that affect their use. These can occur because the users look through the LCD to frame the image and can often miss the icons displayed around the edges that present important information about the status of the camera. This may lead to shots being missed or poorly exposed. Most camera phones do not take full advantage of the features of the underlying phone platform to enhance their interfaces. We created a camera application for the Nokia N95 that featured novel interface elements and made use of the features of the platform to provide a rich variety of information in more usable forms, such as: sonifications of the luminance histogram to ensure better exposure before a picture is taken; phone orientation to give a level indicator to ensure the camera is straight; measuring phone movement to ensure the phone is being held steady; and the detection of image motion to support panning We also present a scenario for how these features could be used in conjunction with each other during the photo taking process.

Auditory interfaces offer a solution to the problem of effective eyes-free mobile interactions. However, a problem with audio, as opposed to visual displays, is dealing with multiple simultaneous outputs. Any audio interface needs to consider: 1) simultaneous versus sequential presentation of multiple audio streams, 2) 3D audio techniques to place sounds in different spatial locations versus a single point of presentation, 3) dynamic movement versus fixed locations of audio sources. We present an experiment using a divided-attention task where a continuous podcast and an audio menu compete for attention. A sequential presentation baseline assessed the impact of cognitive load, and as expected, dividing attention had a significant effect on overall performance. However, spatial audio still increased the users' ability to attend to two streams, while dynamic movement of streams led to higher perceived workload. These results will provide guidelines for designers when building eyes-free auditory interfaces for mobile applications.

Despite many successes in desktop applications, little work has looked at the use of pressure input on mobile devices and the different issues associated with mobile interactions e.g. non-visual feedback. This study examined pressure input on a mobile device using a single Force Sensing Resistor (FSR) with linearised output as a means of target selection within a menu, where target menu items varied in size and location along the z-axis. Comparing visual and audio feedback, results showed that, overall, eyes-free pressure interaction reached a mean level of 74% accuracy. With visual feedback mean accuracy reached 85%. Participants could accurately distinguish up to 10 pressure levels when given adequate feedback indicating a high level of control.

User participation has transformed the way news travel the globe. With the rise of the 'Web 2.0' phenomenon users have been empowered with the means of creating and distributing informational items, which we call social feeds. Platforms like Twitter and Facebook provide a variety of tools to facilitate real-time communication among people. But social sites are not limited to personal chat; they also provide an effective means for organizing large groups of people in response to catastrophic disasters. Monitoring these feeds can provide time-critical information, but can easily lead to information overload due to the large amount of data being shared.

When designing interfaces for mobile devices it is import-ant to take into account the variety of contexts of use. We present a study that examines how changing noise and dis-turbance in the environment affects user performance in a touchscreen typing task with the interface being presented through visual only, visual and tactile, or visual and audio feedback. The aim of the study is to show at what exact environmental levels audio or tactile feedback become inef-fective. The results show significant decreases in perform-ance for audio feedback at levels of 94dB and above as well as decreases in performance for tactile feedback at vibration levels of 9.18g/s. These results suggest that at these levels, feedback should be presented by a different modality. These findings will allow designers to take advantage of sensor enabled mobile devices to adapt the provided feed-back to the user's current context.

Mobile user interfaces are commonly based on techniques developed for desktop computers in the 1970s, often including buttons, sliders, windows and progress bars. These can be hard to use on the move, which then limits the way we use our devices and the applications on them. This talk will look at the possibility of moving away from these kinds of interactions to ones more suited to mobile devices and their dynamic contexts of use where users need to be able to look where they are going, carry shopping bags and hold on to children. Multimodal (gestural, audio and haptic) interactions provide us new ways to use our devices that can be eyes and hands free, and allow users to interact in a 'head up' way. These new interactions will facilitate new services, applications and devices that fit better into our daily lives and allow us to do a whole host of new things.

We report the results of a study focusing on the meanings that can be conveyed by audio and tactile icons. Our research considers the following question: how can audio and tactile icons be designed to optimise congruence between crossmodal feedback and the type of information this feedback is intended to convey? For example, if we have a set of system warnings, confirmations, progress up-dates and errors: what audio and tactile representations best match the information or type of message? Is one modality more appropriate at presenting certain types of information than the other modality? The results of this study indicate that certain parameters of the audio and tactile modalities such as rhythm, texture and tempo play an important role in the creation of congruent sets of feedback when given a specific type of information to transmit. We argue that a combination of audio or tactile parameters derived from our results allows the same type of information to be derived through touch and sound with an intuitive match to the content of the message.

This paper describes the design and evaluation of a touch screen-based pressure keyboard to investigate the possibilities of pressure as a new method of input for mobile devices. A soft press on the touchscreen generated a lowercase letter, a hard press an uppercase one. The aim was to improve input performance when entering mixed-case text, or shifted characters often used for emoticons, etc. An experiment compared two different forms of pressure input (Dwell and Quick Release) against a standard shift key keyboard, with users both sitting and walking. Results showed that Quick Release was the fastest for input of mixed case text with Dwell being the most accurate, even when users were mobile. The results demonstrate that pressure input can outperform a standard shift-key keyboard design for mobile text entry.

Developing interfaces for mobile situations requires that devices are useable on the move. Here, we explore head tilting as an input technique to allow a user to interact with a mobile device 'hands free'. A Fitts' Law style evaluation is described where a user acquires targets, moving the cursor by head tilt. We explore d position and velocity control cursor mechanisms in both static and mobile situations to see which provided the best level of performance. Results show that participants could successfully acquire targets using head tilting. Position control was shown to be si gnificantly faster and more accurate in a static context, but exhi bited significantly poorer accuracy and longer target acquisition times when the user was on the move. We further demonstrate how analysis of user's gait shows consistent targeting biases at different stages in the gait cycle.

We outline two new auditory interaction techniques which build upon existing visual techniques to display off-screen points of interest (POI) in map based mobile computing applications. SonicPie uses a pie menu and compass metaphor, allowing a user to scroll around the environment, hearing off-screen POIs in a spatialised auditory environment. EdgeTouch integrates with the Wedge technique of Gustafson et al. [2], sonifying the POIs as the user comes into contact with them when moving his or her finger around a "sonification border".

Gesture based interfaces provide a new way for us to interact with mobile devices, but also require us to make new decisions about how we feel about this new technology and which gestures we decide are usable and appropriate. These decisions are based on the social and public settings where these devices are used on a daily basis. Our ideas about which gestures are socially acceptable or not are an important factor in whether or not these gestures will be adopted. The ways in which users evaluate social acceptability is not only highly variable, but with drastically different results amongst different users. These differences are not dependant on factors such as age, gender, occupation, geographic location, or previous technology usage. Future work into the social acceptability perceptions of users will focus on personality traits as a new way of understanding how social acceptability is determined.

This paper presents a study of finger-based text entry for mobile devices with touchscreens. Many devices are now coming to market that have no physical keyboards (the Apple iPhone being a very popular example). Touchscreen keyboards lack any tactile feedback and this may cause problems for entering text and phone numbers. We ran an experiment to compare devices with a physical keyboard, a standard touchscreen and a touchscreen with tactile feedback added. We tested this in both static and mobile environments. The results showed that the addition of tactile feedback to the touchscreen significantly improved finger-based text entry, bringing it close to the performance of a real physical keyboard. A second experiment showed that higher specification tactile actuators could improve performance even further. The results suggest that manufacturers should use tactile feedback in their touchscreen devices to regain some of the feeling lost when interacting on a touchscreen with a finger.

"McSig" is a multimodal teaching and learning environ-ment for visually-impaired students to learn character shapes, handwriting and signatures collaboratively with their teachers. It combines haptic and audio output to realize the teacher's pen input in parallel non-visual modalities. McSig is intended for teaching visually-impaired children how to handwrite characters (and from that signatures), something that is very difficult without visual feedback. We conducted an evaluation with eight visually-impaired children with a pretest to assess their current skills with a set of character shapes, a training phase using McSig and then a post-test of the same character shapes to see if there were any improvements. The children could all use McSig and we saw significant improvements in the character shapes drawn, particularly by the completely blind children (many of whom could draw almost none of the characters before the test). In particular, the blind participants all expressed enjoyment and excitement about the system and using a computer to learn to handwrite.

Our research considers the following question: how can visual, audio and tactile feedback be combined in a congruent manner for use with touchscreen graphical widgets? For example, if a touchscreen display presents different styles of visual buttons, what should each of those buttons feel and sound like? This paper presents the results of an experiment conducted to investigate methods of congruently combining visual and combined audio/tactile feedback by manipulating the different parameters of each modality. The results indicate trends with individual visual parameters such as shape, size and height being combined congruently with audio/tactile parameters such as texture, duration and different actuator technologies. We draw further on the experiment results using individual quality ratings to evaluate the perceived quality of our touchscreen buttons then reveal a correlation between perceived quality and crossmodal congruence. The results of this research will enable mobile touchscreen UI designers to create realistic, congruent buttons by selecting the most appropriate audio and tactile counterparts of visual button styles.

Camera phones are now very common but there are some issues that affect their usability. These can occur because users look through the LCD to frame the image and can often miss the icons displayed around the edge that present important information about exposure, battery life, number of shots remaining, etc. This may lead to shots being missed or poorly exposed. We created a sonified luminance histogram to present exposure information, a sound cue to indicate memory card space remaining and a tactile cue for battery charge status. A user study showed that participants were able to use the sonified histogram to identify exposure successfully and could recognise the status of the battery and memory card well, suggesting that alternative forms of output could free-up the screen for framing the image.

In this paper, we investigate wrist rotation as a hands-free method of interaction with a mobile device. To evaluate this technique, a Fitts' Law targeting study is described in four different postures: resting, seated, standing and walking. Results show correlations in movement time and the Index of Difficulty of the task and similarities in the targeting performance for the first three conditions, but show walking and targeting using this method was significantly more difficult.

Mobile device user interface elements tend to be based on desktop widgets that were not originally intended for small screen finger-based interfaces. Mobile usage scenarios afford many completely different interactions, so should be designed accordingly. This paper presents a new type of widget, the T-Bar, designed specifically for finger-based touchscreen interfaces using tactile feedback. Using the tactile feedback for orientation, the user's fingertip is guided along the T-Bar until an item is successfully selected. This paper offers observations on our finger-based touchscreen widget and two applications of the T-Bar widget. Both, File-o-Feel and Touch 'n' Twist are multi-touch information browsing applications that deviate from traditional desktop GUI paradigms and are tailored for fingertip input where all interaction takes place through the use of T-Bars; eliminating the need for any other widgets.

We present a study investigating the use of vibrotactile feedback for touch-screen keyboards on PDAs. Such key-boards are hard to use when mobile as keys are very small. We conducted a laboratory study comparing standard but-tons to ones with tactile feedback added. Results showed that with tactile feedback users entered significantly more text, made fewer errors and corrected more of the errors they did make. We ran the study again with users seated on an underground train to see if the positive effects trans-ferred to realistic use. There were fewer beneficial effects, with only the number of errors corrected significantly im-proved by the tactile feedback. However, we found strong subjective feedback in favour of the tactile display. The results suggest that tactile feedback has a key role to play in improving interactions with touch screens.

This paper reports an experiment into the design of crossmodal icons which can provide an alternative form of output for mobile devices using audio and tactile modalities to communicate information. A complete set of crossmodal icons was created by encoding three dimensions of information in three crossmodal auditory/tactile parameters. Earcons were used for the audio and Tactons for the tactile crossmodal icons. The experiment investigated absolute identification of audio and tactile crossmodal icons when a user is trained in one modality and tested in the other (and given no training in the other modality) to see if knowledge could be transferred between modalities. We also compared performance when users were static and mobile to see any effects that mobility might have on recognition of the cues. The results showed that if participants were trained in sound with Earcons and then tested with the same messages presented via Tactons they could recognize 85% of messages when stationary and 76% when mobile. When trained with Tactons and tested with Earcons participants could accurately recognize 76.5% of messages when stationary and 71% of messages when mobile. These results suggest that participants can recognize and understand a message in a different modality very effectively. These results will aid designers of mobile displays in creating effective crossmodal cues which require minimal training for users and can provide alternative presentation modalities through which information may be presented if the context requires.

We present a study into the use of smell for searching digi-tal photo collections. Many people now have large photo libraries on their computers and effective search tools are needed. Smell has a strong link to memory and emotion so may be a good way to cue recall when searching. Our study compared text and smell based tagging. For the first stage we generated a set of smell and tag names from user de-scriptions of photos, participants then used these to tag pho-tos, returning two weeks later to answer questions on their photos. Results showed that participants could tag effec-tively with text labels, as this is a common and familiar task. Performance with smells was lower but participants performed significantly above chance, with some partici-pants using smells well. This suggests that smell has poten-tial. Results also showed that some smells were consistently identified and useful, but some were not and highlighted issues with smell delivery devices. We also discuss some practical issues of using smell for interaction.

We present the results of an empirical study investigating the effect of feedback, mobility and index of difficulty on a deictic spatial audio target acquisition task in the horizontal plane in front of a user. With audio feedback, spatial audio display elements are found to enable usable deictic interac-tion that can be described using Fitts law. Feedback does not affect perceived workload or preferred walking speed compared to interaction without feedback. Mobility is found to degrade interaction speed and accuracy by 20%. Participants were able to perform deictic spatial audio target acquisition when mobile while walking at 73% of their pre-ferred walking speed. The proposed feedback design is ex-amined in detail and the effects of variable target widths are quantified. Deictic interaction with a spatial audio display is found to be a feasible solution for future interface designs.

Access to digitally stored numerical data is currently very limited for sight impaired people. Graphs and visualizations are often used to analyze relationships between numerical data, but the current methods of accessing them are highly visually mediated. Representing data using audio feedback is a common method of making data more accessible, but methods of navigating and accessing the data are often serial in nature and laborious. Tactile or haptic displays could be used to provide additional feedback to support a point-and-click type interaction for the visually impaired. A requirements capture conducted with sight impaired computer users produced a review of current accessibility technologies, and guidelines were extracted for using tactile feedback to aid navigation. The results of a qualitative evaluation with a prototype interface are also presented. Providing an absolute position input device and tactile feedback allowed the users to explore the graph using tactile and proprioceptive cues in a manner analogous to point-and-click techniques.

Tactons are structured vibrotactile messages which can be used for non-visual information presentation when visual displays are limited, unavailable or inappropriate, such as in mobile phones and other mobile devices. Little is yet known about how to design them effectively. Previous studies have investigated the perception of Tactons which encode two dimensions of information using two different vibrotactile parameters (rhythm and roughness) and found recognition rates of around 70. When more dimensions of information are required it may be necessary to extend the parameter-space of these Tactons. Therefore this study investigates recognition rates for Tactons which encode a third dimension of information using spatial location. The results show that identification rate for three-parameter Tactons is just 48, but that this can be increased to 81 by reducing the number of values of one of the parameters. These results will aid designers to select suitable Tactons for use when designing mobile displays.

This paper describes an alternative form of interaction for mobile devices using crossmodal output. These crossmodal displays allow alternative senses such as hearing and touch to be used to perceive information normally presented to the visual modality. Initial experiments show that roughness and spatial location can be perceived as equivalent in both the auditory and tactile domain. This paper discusses how crossmodal displays can be constructed using the results from these experiments and the benefits they bring to mobile human computer interfaces

An empirical study that compared six different feedback cue types to enhance pointing efficiency in deictic spatial audio displays is presented. Participants were asked to select a sound using a physical pointing gesture, with the help of a loudness cue, a timbre cue and an orientation update cue as well as with combinations of these cues. Display content was varied systematically to investigate the effect of increasing display population. Speed, accuracy and throughput ratings are provided as well as effective target widths that allow for minimal error rates. The results showed direct pointing to be the most efficient interaction technique; however large effective target widths reduce the applicability of this technique. Movement-coupled cues were found to significantly reduce display element size, but resulted in slower interaction and were affected by display content due to the requirement of continuous target attainment. The results show that, with appropriate design, it is possible to overcome interaction uncertainty and provide solutions that are effective in mobile human computer interaction.

Although landmarks are an integral part of navigation, they have rarely been used explicitly within electronic pedestrian navigation aids. We describe a two-part study into the use of landmarks in such aids, using a set of field experiments. The first part investigated whether such devices can be effective for older adults (over 60 years old), who might particularly benefit from them due to declines in sensory, cognitive and motor abilities. The second part compared the effectiveness of different methods of presenting landmark information. We show that a pedestrian navigation aid based around landmarks is particularly useful for older people and demonstrate that text, speech and photographs are all effective ways of presenting landmark information, although speech on its own has some drawbacks. We found that different people prefer information to be presented in different modalities, indicating a need for personalisation, although multi-modality may also help to address this issue.

This paper presents a review of the research surrounding the paper “The Design and Evaluation of a Sonically Enhanced Tool Palette” by Brewster and Clarke from ICAD 1997. A historical perspective is given followed by a discussion of how this work has fed into current developments in the area.

This paper describes an experiment to investigate the effectiveness of adding sound to tool palettes. Palettes have usability problems because users need to see the information they present, but they are often outside the area of visual focus. We used nonspeech sounds called earcons to indicate the current tool and when tool changes occurred so that users could tell what tool they were in wherever they were looking. Results showed a significant reduction in the number of tasks performed with the wrong tool. Therefore, users knew what the current tool was and did not try to perform tasks with the wrong one. All of this was not at the expense of making the tool palettes any more annoying to use.

Two investigations into the identification of concurrently presented, structured sounds, called earcons were carried out. One of the experiments investigated how varying the number of concurrently presented earcons affected their identification. It was found that varying the number had a significant effect on the proportion of earcons identified. Reducing the number of concurrently presented earcons lead to a general increase in the proportion of presented earcons successfully identified. The second experiment investigated how modifying the earcons and their presentation, using techniques influenced by auditory scene analysis, affected earcon identification. It was found that both modifying the earcons such that each was presented with a unique timbre, and altering their presentation such that there was a 300 ms onset-to-onset time delay between each earcon were found to significantly increase identification. Guidelines were drawn from this work to assist future interface designers when incorporating concurrently presented earcons.

Mobile and wearable computers present input/output prob-lems due to limited screen space and interaction techniques. When mobile, users typically focus their visual attention on navigating their environment - making visually demanding interface designs hard to operate. This paper presents two multimodal interaction techniques designed to overcome these problems and allow truly mobile, 'eyes-free' device use. The first is a 3D audio radial pie menu that uses head gestures for selecting items. An evaluation of a range of different audio designs showed that egocentric sounds re-duced task completion time, perceived annoyance, and al-lowed users to walk closer to their preferred walking speed. The second is a sonically enhanced 2D gesture recognition system for use on a belt-mounted PDA. An evaluation of the system with and without audio feedback showed users' ges-tures were more accurate when dynamically guided by au-dio-feedback. These novel interaction techniques demon-strate effective alternatives to visual-centric interface de-signs on mobile devices.

In this paper, we describe a study comparing the strengths of a multimodal Virtual Reality (VR) interface against traditional tactile diagrams in conveying information to visually impaired and blind people. The multimodal VR interface consists of a force feedback device (SensAble PHANTOM), synthesized speech and non-speech audio. Potential advantages of the VR technology are well known however its real usability in comparison with the conventional paper-based medium is seldom investigated. We have addressed this issue in our evaluation. The experimental results show benefits from using the multimodal approach in terms of more accurate information about the graphs obtained by users.

This paper discusses the use of gesture and non-speech audio as ways to improve the user interface of a mobile music player. Their key advantages mean that users could use a player without having to look at its controls when on the move. Two very different evaluations of the player took place: one based on a standard usability experiment (comparing the new player to a standard design) and the other a video analysis of the player in use. Both of these showed significant usability improvements for the gesture/audio-based interface over a standard visual/pen-based display. The similarities and differences in the results produced by the two studies are discussed

We describe AudioCave, an environment for exploring the impact of spatialising sonified graphs on a set of numerical data comprehension tasks. Its design builds on findings regarding the effectiveness of sonified graphs for numerical data overview and discovery by visually impaired and blind students. We demonstrate its use as a test bed for comparing the approach of accessing a single sonified numerical datum at a time to one where multiple sonified numerical data can be accessed concurrently. Results from this experiment show that concurrent access facilitates the tackling of our set multivariate data comprehension tasks. AudioCave also demonstrates how the spatialisation of the sonified graphs provides opportunities for sharing the representation. We present two experiments investigating users solving set data comprehension tasks collaboratively by sharing the data representation.

Line graphs stand as an established information visualisation and analysis technique taught at various levels of difficulty according to standard Mathematics curricula. It has been argued that blind individuals cannot use line graphs as a visualisation and analytic tool because they currently primarily exist in the visual medium. The research described in this paper aims at making line graphs accessible to blind students through auditory and haptic media. We describe (1) our design space for representing line graphs, (2) the technology we use to develop our prototypes and (3) the insights from our preliminary work.

Haptic devices are now commercially available and thus touch has become a potentially realistic solution to a variety of interaction design challenges. We report on an investigation of the use of touch as a way of reducing visual overload in the conventional desktop. In a two-phase study, we investigated the use of the PHANToM haptic device as a means of interacting with a conventional graphical user interface. The first experiment compared the effects of four different haptic augmentations on usability in a simple targeting task. The second experiment involved a more ecologically-oriented searching and scrolling task. Results indicated that the haptic effects did not improve users performance in terms of task completion time. However, the number of errors made was significantly reduced. Subjective workload measures showed that participants perceived many aspects of workload as significantly less with haptics. The results are described and the implications for the use of haptics in user interface design are discussed.

Future human-computer interfaces will use more than just graphical output to display information. In this paper we suggest that sound and graphics together can be used to improve interaction. We describe an experiment to improve the usability of standard graphical menus by the addition of sound. One common difficulty is slipping off a menu item by mistake when trying to select it. One of the causes of this is insufficient feedback. We designed and experimentally evaluated a new set of menus with much more salient audio feedback to solve this problem. The results from the experiment showed a significant reduction in the subjective effort required to use the new sonically-enhanced menus along with significantly reduced error recovery times. A significantly larger number of errors were also corrected with sound.

This paper describes the design of user-interface widgets that include non-speech sound. Previous research has shown that the addition of sound can improve the usability of human–computer interfaces. However, there is little research to show where the best places are to add sound to improve usability. The approach described here is to integrate sound into widgets, the basic components of the human–computer interface. An overall structure for the integration of sound is presented. There are many problems with current graphical widgets and many of these are difficult to correct by using more graphics. This paper presents many of the standard graphical widgets and describes how sound can be added. It describes in detail usability problems with the widgets and then the non-speech sounds to overcome them. The non-speech sounds used are earcons. These sonically-enhanced widgets allow designers who are not sound experts to create interfaces that effectively improve usability and have coherent and consistent sounds.

This article describes 3 experiments that investigate the possibiity of using structured nonspeech audio messages called earcons to provide navigational cues in a menu hierarchy. A hierarchy of 27 nodes and 4 levels was created with an earcon for each node. Rules were defined for the creation of hierarchical earcons at each node. Participants had to identify their location in the hierarchy by listening to an earcon. Results of the first experiment showed that participants could identify their location with 81.5% accuracy, indicating that earcons were a powerful method of communicating hierarchy information. One proposed use for such navigation cues is in telephone-based interfaces (TBIs) where navigation is a problem. The first experiment did not address the particular problems of earcons in TBIs such as “does the lower quality of sound over the telephone lower recall rates,” “can users remember earcons over a period of time.” and “what effect does training type have on recall?” An experiment was conducted and results showed that sound quality did lower the recall of earcons. However; redesign of the earcons overcame this problem with 73% recalled correctly. Participants could still recall earcons at this level after a week had passed. Training type also affected recall. With personal training participants recalled 73% of the earcons, but with purely textual training results were significantly lower. These results show that earcons can provide good navigation cues for TBIs. The final experiment used compound, rather than hierarchical earcons to represent the hierarchy from the first experiment. Results showed that with sounds constructed in this way participants could recall 97% of the earcons. These experiments have developed our general understanding of earcons. A hierarchy three times larger than any previously created was tested, and this was also the first test of the recall of earcons over time.

A structured method is described for the analysis of interactions to identify situations where hidden information may exist and where non-speech sound might be used to overcome the associated problems. Interactions are considered in terms of events, status and modes to find any hidden information. This is then categorised in terms of the feedback needed to present it. An auditory-enhanced scrollbar, based on the method described, was then experimentally tested. Timing and error rates were used along with subjective measures of workload. Results from the experiment show a significant reduction in time to complete one task, a decrease in the mental effort required and an overall preference for the auditory-enhanced scrollbar.