Interactive systems have long been dominated by visual paradigms, shaping how we design, evaluate, and experience technology. Yet human perception is inherently multisensory, grounded in the integration of sound, touch, movement, and internal bodily signals. In this talk, I argue for a shift from visual interfaces toward multisensory experiences, drawing on research in sonic interaction design, virtual reality, and hearing science.

Computational models of dialogue often struggle to capture the nuanced structures of spontaneous conversation - specifically in polyadic, real-world settings. We introduce a multilayered annotation protocol designed for the GaMMA corpus, a Danish dataset of four-person conversations recorded in both quiet and noisy environments. The protocol targets key interactional phenomena: Turn Construction Units, backchannels, floor transfer attempts, and repair sequences. Each annotation layer is grounded in Conversation Analysis while remaining machine-actionable, enabling alignment with multimodal data such as gaze and motion. We report inter-annotator agreement metrics across annotation tiers and discuss how the protocol supports both fine-grained interaction analysis and the training of context-aware dialogue models.

The interest and dissemination of Virtual Reality (VR) is still expanding across multiple domains. While VR has the capacity to revolutionize many different industries and fields, the recent Covid-19 pandemic has also increased awareness of hygiene and safety associated with VR usage. Despite the growing commercial availability of both VR headsets and preventive and disinfection solutions, confirmatory studies required to validate both the efficacy and safety of the different solutions are severely lacking. This paper presents the findings of a survey aimed at gathering information about current hygiene practices in various domains, along with the perception of research availability. Cleaning methods varied among respondents (n=42), but most popular methods consisted of several consecutive solutions. Respondents primarily used anti-bacterial or alcohol disinfection wipes (81%), permanent face covers (leather/silicone) (43%), disposable cover/mask (26%), and UVC light disinfection (26%). 65% of the respondents stated that the Covid-19 pandemic made them change their practices. A majority of respondents remarked that there was a scarcity of research, yet, most respondents were fairly or completely confident that their cleaning protocols were sufficient, despite remarking that it was sometimes not adhered to. The efficacy of VR hygiene solutions and practices remains largely understudied despite the urgent need to establish validated and efficacious cleaning protocols and practices. Current solutions and practices primarily focuses on the inside of the headset, although the outside of the headset may be far more exposed to contaminants through e.g. hand-contact. Further research is needed to define and evaluate context-dependent risk-assessments as well as suitable cleaning protocols for VR-headsets.

According to the World Health Organisation’s World Report on Hearing, there is a strong need to provide better education on hearing loss from a young age. This project aims to educate the Danish young population (13 to 17-year-olds) about the hearing sense through an educational multiplayer virtual reality-based escape room with the benefits of educational escape rooms. In collaboration with relevant audiologist stakeholders, this project follows an iterative process of design, implementation, and evaluation of the application. The developed solution will undergo several user studies in the following months.

Abstract In the growing field of sonic interaction design, increasing emphasis is being placed on walking-based interactions within an array of applications, including virtual reality, interactive media, and rehabilitation. Our study focuses on recreating the aural experience of wading through water, specifically the challenge of eliciting accurate and natural movement-sound associations for wading, both in interactive and listening-only contexts. We engineered a real-time wading simulation using a digital Foley technique that maps lower limb angular velocity contours to the intensity of steady-state flowing sounds. Our first iteration was evaluated in a perceptual experiment involving 16 participants, as well as an interactive test with 9 participants, which revealed the need for additional sensors along with time-domain preprocessing to ensure a consistently natural sound envelope across walking cadences (step tempi). We then refined the mapping function and incorporated more sound layers. In our subsequent listening test, 55 participants compared the realism of the initial and refined versions with real-life wading sounds at various step cadences. While the refined version demonstrated a notable improvement over the initial one and was deemed fairly realistic overall, it fell just short of the authenticity of the real-life recordings at faster cadences, suggesting room for further improvement of our simulation. Nevertheless, this research marks a stride in the evolution of walking-based sonic interactions, instigating wider acceptance and application of such systems in the future.

We present DuoRhythmo, a collaborative accessible digital musical interface (CADMI) that gives people living with Amyotrophic Lateral Sclerosis (PALS) the experience of remotely and collaboratively creating music in real-time. We designed DuoRhythmo specifically to be utilized for eye tracking and optimized it for head- and computer mouse interaction, as well using a user-centered design approach. Together with five PALS, we completed a mixed-methods evaluation to assess the accessibility of DuoRhythmo. Participants described the CADMI using the Microsoft Desirability Toolkit (MDT) as fun, empowering, accessible, easy to use, engaging, and stimulating and gave an average System Usability Scale (SUS) score of 79.5. We suggest further research on remote collaboration within the field of accessible digital musical instruments (ADMIs) using the term CADMI to explore the positive effects of collaborative music-making on the quality of life of PALS.

Music Therapy (MT) has shown many benefits in helping autistic children, but some challenges remain due to children’s social anxiety and sensory issues. Yet, very few studies have investigated how Virtual Reality (VR) could help to increase the accessibility of MT approaches. This paper presents an exploratory study investigating the use of VR to perform MT sessions for autistic children with severe learning disabilities and complex needs. The study is performed in terms of acceptability, usability, and social communication. A collaborative MT approach was designed in close collaboration with music therapists from Denmark and psychologists from France, using head-mounted display-based VR. Testing were conducted with thirteen children with various neurodevelopmental conditions and intellectual disabilities at a children’s day hospital in Paris. The results indicate positive acceptability and usability for these children, and suggest a positive effect of MT in VR regarding communication.

We present a demo of Earnormous, a virtual reality game to teach about the human ear.

Although hearing loss is treated with hearing technology and rehabilitation, children with hearing loss still face challenges. Factors such as distance to the sound source and noise from the surroundings are the children’s biggest enemies. In the "Listen Again" project, a listening- and spatial awareness training application was co-designed together with deaf and hard-of-hearing children who use cochlear implants. This paper presents quantitative and qualitative results from a two-month evaluation where 22 children were asked to play with the VR solution for two months, three times a week.

This paper details preliminary explorations of how cutaneous rabbit illusions and virtual reality can be combined to produce experiences of cutaneous saltation along paths that do not match the actuator configuration, and to reduce the number of needed actuators. We present two studies exploring the effects of actuator configuration, vibration properties, and varying visual presentations of the virtual object touching the user’s forearm. The results indicate that the illusion can occur along a curvilinear path, and congruent visual movement can enhance the illusion and potentially reduce the number of needed actuators. Incongruent visual movement does not break the illusion but appears to modulate the perceived path of the virtual object.

Physical modelling sound synthesis is a powerful method for constructing virtual instruments aiming to mimic the sound of real-world counterparts, while allowing for the possibility of engaging with these instruments in ways which may be impossible in person. Such a case is explored in this paper: particularly the simulation of a friction drum inspired instrument. It is an instrument played by causing the membrane of a drum head to vibrate via friction. This involves rubbing the membrane via a stick or a cord attached to its center, with the induced vibrations being transferred to the air inside a sound box. This paper describes the development of a real-time audio application which models such an instrument as a bowed membrane connected to an acoustic tube. This is done by means of a numerical simulation using finite-difference time-domain (FDTD) methods in which the excitation, whose position is free to change in real-time, is modelled by a highly non-linear elasto-plastic friction model. Additionally, the virtual instrument allows for dynamically modifying physical parameters of the model, thereby allowing the user to generate new and interesting sounds that go beyond a real-world friction drum.

For physical modelling sound synthesis, many techniques are available; time-stepping methods (e.g., finite-difference time-domain (FDTD) methods) have an advantage of flexibility and generality in terms of the type of systems they can model. These methods do, however, lack the capability of easily handling smooth parameter changes while retaining optimal simulation quality and stability, something other techniques are better suited for. In this paper, we propose an efficient method to smoothly add and remove grid points from a FDTD simulation under sub-audio rate parameter variations. This allows for dynamic parameter changes in physical models of musical instruments. An instrument such as the trombone can now be modelled using FDTD methods, as well as physically impossible instruments where parameters such as e.g. material density or its geometry can be made time-varying. Results show that the method does not produce (visible) artifacts and stability analysis is ongoing.

In this paper, a complete simulation of a trombone using finite-difference time-domain (FDTD) methods is proposed. In particular, we propose the use of a novel method to dynamically vary the number of grid points associated to the FDTD method, to simulate the fact that the physical dimension of the trombone's resonator dynamically varies over time. We describe the different elements of the model and present the results of a real-time simulation.

While previous research on spatial sound perception investigated the physical mechanisms producing the most relevant elevation cues, how spectral notches are generated and related to the individual morphology of the human pinna is still a topic of debate. Correctly modeling these important elevation cues, and in particular the lowest frequency notches, is an essential step for individualizing Head-Related Transfer Functions (HRTFs). In this paper we propose a simple computational model able to predict the center frequencies of pinna notches from ear meshes. We apply such a model to a highly controlled HRTF dataset built with the specific purpose of understanding the contribution of the pinna to the HRTF. Results show that the computational model is able to approximate the lowest frequency notch with improved accuracy with respect to other state-of-the-art methods. By contrast, the model fails to predict higher-order pinna notches correctly. The proposed approximation supplements understanding of the morphology involved in generating spectral notches in experimental HRTFs.

When physical props serve as proxies for virtual tools used to manipulate the virtual environment, it is challenging to provide appropriate haptic feedback. Redirected tool-mediated manipulation addresses this challenge by distorting the mapping between physical and virtual tools to provide a sensation of manipulating the virtual environment, when the physical tool comes into contact with another physical prop. For example, a virtual hammer's position can be offset to ensure that physical impacts accompany each strike of a virtual nail. We demonstrate the idea by showing that it can be used to create sensations of impact and resistance when driving a virtual nail into a surface, when tightening a virtual screw, and when sawing through a virtual plank. The results of a user study indicate that the proposed approach is perceived as more realistic than interaction with a single physical prop or controller and no notable detriments to precision were observed.

This paper provides a small set of essential parameters for a personalized and effective real-time auralization with headphones. An image-guided procedure with two 2D images of the user's head guides the mixed structural modeling of head-related transfer function (HRTF), combining a spherical head model with ear displacement with the HRTF high-frequency magnitude selected from a database according to ear anthropometry. Room acoustics phenomena are simplified following the scattering delay network (SDN) approach which allows an accurate spatialization of first order reflections. Finally, statically significant improvements in localization performances within a virtual reality (VR) test allow to identify some benefits of the proposed customized auralization model compared to the widely used higher-order ambisonics (HOA) rendering with generic HRTFs.

Directivity and gain in microphone array systems for hearing aids or hearable devices allow users to acoustically enhance the information of a source of interest. This source is usually positioned directly in front. This feature is called acoustic beamforming. The current study aimed to improve users' interactions with beamforming via a virtual prototyping approach in immersive virtual environments (VEs). Eighteen participants took part in experimental sessions composed of a calibration procedure and a selective auditory attention voice-pairing task. Eight concurrent speakers were placed in an anechoic environment in two virtual reality (VR) scenarios. The scenarios were a purely virtual scenario and a realistic 360° audio-visual recording. Participants were asked to find an individual optimal parameterization for three different virtual beamformers: (i) head-guided, (ii) eye gaze-guided, and (iii) a novel interaction technique called dual beamformer, where head-guided is combined with an additional hand-guided beamformer. None of the participants were able to complete the task without a virtual beamformer (i.e., in normal hearing condition) due to the high complexity introduced by the experimental design. However, participants were able to correctly pair all speakers using all three proposed interaction metaphors. Providing superhuman hearing abilities in the form of a dual acoustic beamformer guided by head and hand movements resulted in statistically significant improvements in terms of pairing time, suggesting the task-relevance of interacting with multiple points of interests.

Virtual Reality (VR) technologies are becoming more and more adopted for treatment of mental health, especially due to the availability of low cost and usable hardware and software solutions. In this paper we present the views of three teachers on how to use VR to instruct children and adolescents diagnosed with Autism Spectrum Disorder (ASD). All three teachers work with students diagnosed with ASD on a daily basis. Three categories of skills that would benefit from being taught using VR emerged from the discussion with the teachers: everyday living skills, social skills, and academic skills. Skills within these categories as well as some of the main advantages of VR according to the teachers are discussed in this paper.

The poster describes a multisensory simulation of plucking guitar strings in virtual reality and a user study evaluating the simulation. Auditory feedback is generated by a physics-based simulation of guitar strings, and haptic feedback is provided by a combination of high fidelity vibrotactile actuators and a Phantom Omni. The study compared four conditions: no haptic feedback, vibrotactile feedback, force feedback, and a combination of force and vibrotactile feedback. The results indicate that the combination of vibrotactile and force feedback elicits the most realistic experience, and during this condition, participants were less likely to inadvertently hit strings. Notably, no significant differences were found between the conditions involving either vibrotactile or force feedback.

Being able to hear objects in an environment, for example using echolocation, is a challenging task. The main goal of the current work is to use virtual environments (VEs) to train novice users to navigate using echolocation. Previous studies have shown that musicians are able to differentiate sound pulses from reflections. This paper presents design patterns for VE simulators for both training and testing procedures, while classifying users' navigation strategies in the VE. Moreover, the paper presents features that increase users' performance in VEs. We report the findings of two user studies: a pilot test that helped improve the sonic interaction design, and a primary study exposing participants to a spatial orientation task during four conditions which were early reflections (RF), late reverberation (RV), early reflections-reverberation (RR) and visual stimuli (V). The latter study allowed us to identify navigation strategies among the users. Some users (10/26) reported an ability to create spatial cognitive maps during the test with auditory echoes, which may explain why this group performed better than the remaining participants in the RR condition.

The process of listening to an audiobook is usually a rather passive act that does not require an active interaction. If spatial interaction is incorporated into a storytelling scenario, can open. Possibilities of a novel experience which allows an active participation might affect the user-experience. The aim of this paper is to create a portable prototype system based on an embedded hardware platform, allowing listeners to get immersed in an interactive audio storytelling experience enhanced by dynamic binaural audio rendering. For the evaluation of the experience, a short story based on the horror narrative of Stephen King’s Strawberry Springs is adapted and designed in virtual environments. A comparison among three different listening experiences, namely, (i) monophonic (traditional audio story), (ii) static binaural rendering (state-of-the-art audio story), and (iii) our prototype, is conducted. We discuss the quality of the experience based on usability testing, physiological data, emotional assessments, and questionnaires for immersion and spatial presence. Results identify a clear trend for an increase in immersion with our prototype compared to traditional audiobooks, showing also an emphasis on story-specific emotions, i.e., terror and fear.

Achieving a full 3D auditory experience with head-related transfer functions (HRTFs) is still one of the main challenges of spatial audio rendering. HRTFs capture the listener's acoustic effects and personal perception, allowing immersion in virtual reality (VR) applications. This paper aims to investigate the connection between listener sensitivity in vertical localization cues and experienced presence, spatial audio quality, and attention. Two VR experiments with head-mounted display (HMD) and animated visual avatar are proposed: (i) a screening test aiming to evaluate the participants' localization performance with HRTFs for a non-visible spatialized audio source, and (ii) a 2 minute free exploration of a VR scene with five audiovisual sources in a both non-spatialized (2D stereo panning) and spatialized (free-field HRTF rendering) listening conditions. The screening test allows a distinction between good and bad localizers. The second one shows that no biases are introduced in the quality of the experience (QoE) due to different audio rendering methods; more interestingly, good localizers perceive a lower audio latency and they are less involved in the visual aspects.

This poster describes a within-subject study of the virtual body ownership (VBO) illusion using anatomically similar but morphologically different body of a virtual bat. Participants experienced visuo-tactile stimulation of their arms while seeing an object touching the wing of the bat. The mapping between the real and the virtual touch points varied across three conditions: no spatial deviation between visual and tactile input, 50% deviation, and 70% deviation. The results suggest that the degree of experienced VBO varies across the conditions. The illusion was broken in the absence of visuo-tactile stimuli.

This poster describes a within-subject study of agency's influence on virtual body ownership (VBO) using anatomically similar but morphologically different body of a virtual bat. Paricipants were exposed to flight under four conditions: voluntary movement through virtual environment (VE) with avatar present, voluntary movement through virtual environment (VE) with avatar absent, voluntary limbs movement without movements through VE, and finally involuntary movement of the avatar through VE. The results suggest that agency enhances VBO illusion the most under participants' full control during flight locomotion.

This poster details the design and evaluation of Locomotion3- a framework that allows users to freely alternate between real walking, walking-in-place (WIP), and a skateboard metaphor depending on whether navigation requires efficiency, precision, or both. The user study compared the framework to WIP locomotion and the skateboard metaphor and found that Locomotion3achieves a balance between the efficiency of the skateboard metaphor and the precision of WIP locomotion.

We have previously presented an augmented reality (AR) audio application, where scattering delay networks efficiently generate and organize a reverberator, based on room geometry scanned by an AR device. The application allowed for real-time processing and updating of reflection path geometry and provided a proof-of-concept for plausible audio-spatial registration of a virtual object in real environments. Here we present our ongoing work that aims to extend the simulation to outdoor scenes by using the Waveguide Web, instead of the original formulation with the Scattering Delay Networks. The current implementation is computationally more demanding, but has a potential to provide more accurate second-order reflections, and therefore, better registering of audio-visual AR scenes.

Walking-in-place (WIP) techniques provide users with a relatively natural way of walking in virtual reality. However, previous research has primarily focused on WIP during forward movement and tasks involving turning. Thus, little is known about what gestures to use in combination with WIP in order to enable sidestepping. This paper presents two user studies comparing three different types of gestures based on movement of the hip, leaning of the torso, and actual sidesteps. The first study focuses on purely lateral movement while the second involves both forward and lateral movement. The results of both studies suggest that leaning yielded significantly more natural walking experiences and this gesture also produced significantly less positional drift.

This poster describes a between-groups study (n=24) exploring the use of virtual reality (VR) for facilitating focused meditation. Half of the participants were exposed to a meditation session combing the sound of a guiding voice and a visual environment including virtual objects for the participants to focus on. The other half of the participants were only exposed to the auditory guide. The participants' experience of the sessions was assessed using self-reported measures of perceived concentration, temporal duration, stress reduction, and comfort. Interestingly, no statistically significant differences were found between the two conditions. This finding may be revealing in regards to the usefulness of VR-based meditation.

This poster describes a within-groups study (n=17) comparing participants' experience of three different input conditions for instigating virtual teleportation (button clicking, physical jumping, and fist clenching). The results indicated that teleportation by clicking a button generally required less explicit attention and was perceived as more enjoyable, less disorienting, and less physically demanding.

Previous research on visual realism and presence has not involved scenarios, graphics, and hardware representative of commercially available VR games. This poster details a between-subjects study (n=50) exploring if polygon count and texture resolution influence presence during exposure to a VR game. The results suggest that a higher polygon count and texture resolution increased presence as assessed by means of self-reports and physiological measures.

In this paper we propose an experiment that evaluates the influence of audience noise on the feeling of presence and the perceived quality in a virtual reality concert experience delivered using Wave Field Synthesis. A 360 degree video of a live rock concert from a local band was recorded. Single sound sources from the stage and the PA system were recorded, as well as the audience noise, and impulse responses of the concert venue. The audience noise was implemented in the production phase. A comparative study compared an experience with and without audience noise. In a between subject experiment with 30 participants we found that audience noise does not have a significant impact on presence. However, qualitative evaluations show that the naturalness of the sonic experience delivered through wavefield synthesis had a positive impact on the participants.

In this paper we present a within-subjects study (n=26) comparing participants' experience of three kinds of haptic feedback (no haptic feedback, low fidelity haptic feedback and high fidelity haptic feedback) simulating the impact between a virtual baseball bat and ball. We noticed some minor effect on high fidelity versus low fidelity haptic feedback, but haptic feedback generally enhanced realism and quality of experience.

We present an augmented reality (AR) audio application where scattering delay networks efficiently generate and organize a reverberator, based on room geometry scanned by an AR device. The application allows for real-time processing and updating of reflection path geometry. It provides a proof-of-concept for plausible audio-spatial registration of a virtual object in a real environment, but further tests are needed in perceptual evaluation.

The advent of affordable virtual reality (VR) displays and 360° video cameras has sparked an interest in bringing cinematic experiences from the screen and into VR. However, it remains uncertain whether traditional approaches to filmmaking can be directly applied to cinematic VR. Historically editing has provided filmmakers with a powerful tool for shaping stories and guiding the attention of audiences. However, will an immersed viewer, experiencing the story from inside the fictional world, find cuts disorienting? This paper details two studies exploring how cut frequency influences viewers' sense of disorientation and their ability to follow the story, during exposure to fictional 360° films experienced using a head-mounted display. The results revealed no effects of increased cut frequency which leads us to conclude that editing need not pose a problem in relation to cinematic VR, as long as the participants' attention is appropriately guided at the point of the cut.

This poster details a within-subjects study (n=17) investigating the effects of vibrotactile stimulation on illusory self-motion, presence and perceived realism during an interactive sandboarding simulation. The study compared three conditions: no vibration, constant vibration and dynamic vibration. The results suggest that constant vibrotactile feedback led to significantly more compelling self-motion illusions and a higher degree of perceived realism.

This study investigates sliding as a walking-in-place (WIP) method for virtual reality navigation using the Wizdish. The Wizdish is a novel WIP device built for home usage. Two WIP methods, sliding and marching, were compared for naturalness, presence, and surface difference. The sliding technique used on the Wizdish was found to be significantly more disruptive during the experience compared to marching. This could be due to the size of the Wizdish, restricting the users stride, or due to a longer acclimatization time.

Redirection techniques allow users to explore large virtual environments on foot while remaining within a limited physical space. However, research has primarily focused on redirection through manipulation of the visuals used to represent the virtual environment. We describe a within-subjects study (n=31) exploring if participants' ability to detect differences between real and virtual rotations is influenced by the addition of sound that is spatially aligned with its virtual source. The results revealed similar detection thresholds for conditions involving moving audio, static audio, and no audio. This may be viewed as an indication of visual dominance during scenarios such as the one used for the current study.

In virtual reality environments that allow users to walk freely, the area of the virtual environment (VE) is constrained to the size of the tracking area. By using redirection techniques, this problem can be partially circumvented; one of the techniques involves rotating the user more or less in the virtual world than in the physical world; this technique is referred to as rotational gain. Experiments conducted by Frank Steinicke et al. [7] sought to uncover users' ability to detect the usage of rotational gain. This paper seeks to further investigate this area, examining the effect of visual density in the VE. Testing visual density using 0, 4 and 16 objects did not seem to reveal the point at which the test participants become noticeably affected.

This paper presents two within-subjects studies (n=23) exploring how different combinations of visual and auditory feedback influence perceived realism, virtual self-perception and the experience of safety during walks on a virtual platform suspended over a canyon. In the first study, the frequency factor of the footstep sounds was altered and the visual appearance was changed between a newly built wooden bridge and an old bridge with a weaker structure and broken planks. In the second study, the sounds of creaking wood were added to the footstep sounds in half of the trails and compared against footsteps without creaking sounds. Moreover, the frequency factor of the frequency controls for footsteps was also manipulated between trails, but the visual appearance of the bridge was limited to the model of the old broken bridge.

Recent technological advances have brought Virtual Reality (VR) into the homes of consumers, and there is a growing interest in bringing cinematic experiences from the screen and into VR. However, cinematic VR limits filmmakers' ability to effectively guide the audience's attention. In this paper we present a taxonomy of approaches to guiding users' attention, and present a study comparing two such approaches with a control condition devoid of guidance. One approach guides users by controlling their body's orientation, and the other implicitly directs their attention by encouraging them to follow a firefly with their gaze. The results revealed interesting, albeit statistically insignificant, indications that assuming control of the user's action may negatively influence presence, whereas the firefly was perceived as significantly more helpful.

In this paper we describe an experiment aiming to investigate the impact of auditory feedback in a virtual reality supermarket scenario. The participants were asked to read a shopping list and collect items one by one and place them into a shopping cart. Three conditions were presented randomly, where audio feedback was (1) absent, (2) had impact sounds for collisions including when grasping, (3) had impact sounds as well as continuous sounds when moving the products. The subjects experience of the simulation during the three experimental conditions were studied using a questionnaire where ratings on presence, body ownership, awareness of own movements, usability and enjoyment were collected. The results are presented and discussed.

Individuals tend to find realistic walking speeds too slow when relying on treadmill walking or Walking-In-Place (WIP) techniques for virtual travel. This paper details three studies investigating the effects of visual display properties and gain presentation mode on the perceived naturalness of virtual walking speeds: The first study compared three different degrees of peripheral occlusion; the second study compared three different degrees of perceptual distortion produced by varying the geometric field of view (GFOV); and the third study compared three different ways of presenting visual gains. All three studies compared treadmill walking and WIP locomotion. The first study revealed no significant main effects of peripheral occlusion. The second study revealed a significant main effect of GFOV, suggesting that the GFOV size may be inversely proportional to the degree of underestimation of the visual speed. The third study found a significant main effect of gain presentation mode. Allowing participants to interactively adjust the gain led to a smaller range of perceptually natural gains and this approach was significantly faster. However, the efficiency may come at the expense of confidence. Generally the lower and upper bounds of the perceptually natural speeds were higher for treadmill walking than WIP. However, not all differences were statistically significant.

This poster details a study investigating the effect of Head Mounted Display (HMD) weight and locomotion method (Walking-In-Place and treadmill walking) on the perceived naturalness of virtual walking speeds. The results revealed significant main effects of movement type, but no significant effects of HMD weight were identified.

An experiment investigated the subjective experiences of ownership and agency of a pair of virtual wings attached to a motion controlled avatar in an immersive virtual reality setup. A between groups comparison of two ways of controlling the movement of the wings and flight ability. One where the subjects achieved the wing motion and flight ability by using a hand-held video game controller and the other by moving the shoulder. Through four repetitions of a flight task with varying amounts of self-produced audio feedback (from the movement of the virtual limbs), the subjects evaluated their experienced embodiment of the wings on a body ownership and agency questionnaire. The results shows significant differences between the controllers in some of the questionnaire items and that adding self-produced sounds to the avatar, slightly changed the subjects evaluations.

This experiment aimed to investigate whether a user controlling a full body avatar via real time motion tracking in an immersive virtual reality setup, would estimate the weight of the virtual avatar differently if the footstep sounds are manipulated using three different audio filter settings. The visual appearance of the avatar was available in two sizes. The subjects performed six walks with each audio configuration active once over two ground types. After completing each walk, the participants were asked to estimate the weight of the virtual avatar and the suitability of the audio feedback. The results indicate that the filters amplifying the two lower center frequencies altered the subjects estimates of the weight of the avatar body towards being heavier than when compared with the filter with the higher center frequency. There were no significant differences between the weight estimates of the two groups using the different avatar bodies.

In this paper we describe the design and evaluation of a novel, tangible user interface for interaction with sound, to be implemented in a museum setting. Our workinprogress is part of a larger concept for an installation prioritizing a collaborative, explorative, multimodal experience. Focus has been centered on novice children, in order to accommodate all potential users of the museum, and to minimize the risk of excluding users based on skill or previous musical knowhow. We have developed four instances of a multimodal device for interacting with sounds via a tangible interface, and called them Interactive Musical Fruits (IMFs). The IMF consists of an embedded processing system, which can detect its orientation. Qualitative testing with children has been performed, to better evaluate the current design state. Positive feedback from the test subjects upholds the validity and the potential of the IMF as an interface in a museum context. However, further research is required to improve the interactive and collaborative aspects of the device, as well as the aural and visual properties of the IMF.

We describe an experiment whose goal is to investigate the usage of different audio rendering techniques delivered through headphones while walking inside a wide four-side CAVE environment. In our experiment, participants had to physically walked along a virtual path exposed to different auditory stimuli. Each subject was exposed to three conditions: Stereo, Binaural sound spatially congruent with visual and binaural sound spatially incongruent with visuals and had to rate subjectively each. The results of the experiment showed increased preference ratings for the binaural audio rendering, followed by stereo rendering. As expected incongruent spatial cues were ranked significantly lower. Binaural rendering can deliver an increased immersive experience and do no require specialized hardware.

In four-sided CAVE-like VR systems, the absence of the rear wall has been shown to decrease the level of immersion and can introduce breaks in presence. With this experiment we analyse how user attention is diverted while physically walking in a virtual environment, when audio and/or visual attractors are present. Key features of the experiment are the fact that auditory feedback was delivered through binaural audio rendering dependent on the user's head position and orientation and that the four-sided CAVE used for the experiment allowed users to walk up to 9m in straight line. In the experiment we analysed how different “attractors” (audio and/or visual, static or dynamic) modify the user's attention. Results of the experiment show that audio-visual attractors are the most efficient attractors in order to keep the user's attention toward the inside of the CAVE.

Walking-In-Place (WIP) techniques make relatively natural walking experiences within immersive virtual environments possible when the physical interaction space is limited in size. In order to facilitate such experiences it is necessary to establish a natural connection between steps in place and virtual walking speeds. This paper details a study investigating the effects of movement type (treadmill walking and WIP) and step frequency (1.4, 1.8 and 2.2 steps per second) on the range of perceptually natural visual walking speeds. The results suggests statistically significant main effects of both movement type and step frequency but no significant interaction between the two variables.

Walking-In-Place (WIP) techniques make it possible to facilitate relatively natural locomotion within immersive virtual environments that are larger than the physical interaction space. However, in order to facilitate natural walking experiences one needs to know how to map steps in place to virtual motion. This paper describes two within-subjects studies performed with the intention of establishing the range of perceptually natural walking speeds for WIP locomotion. In both studies, subjects performed a series of virtual walks while exposed to visual gains (optic flow multipliers) ranging from 1.0 to 3.0. Thus, the slowest speed was equal to an estimate of the subjects normal walking speed, while the highest speed was three times greater. The perceived naturalness of the visual speed was assessed using self-reports. The first study compared four different types of movement, namely, no leg movement, walking on a treadmill, and two forms of gestural input for WIP locomotion. The results suggest that WIP locomotion is accompanied by a perceptual distortion of the speed of optic flow. The second study was performed using a 4×2 factorial design and compared four different display field-of-views (FOVs) and two types of movement, walking on a treadmill and WIP locomotion. The results revealed significant main effects of both movement type and field of view, but no significant interaction between the two variables. Particularly, they suggest that the size of the display FOV is inversely proportional to the degree of underestimation of the virtual speeds for both treadmill-mediated virtual walking and WIP locomotion. Combined, the results constitute a first attempt at establishing a set of guidelines specifying what virtual walking speeds WIP gestures should produce in order to facilitate a natural walking experience.

The SiMPE workshop series started in 2006 with the goal of enabling speech processing on mobile and embedded devices. The SiMPE 2012 workshop extended the notion of audio to non-speech "Sounds" and thus the expansion became "Speech and Sound". SiMPE 2010 and 2011 brought together researchers from the speech and the HCI communities. Speech User interaction in cars was a focus area in 2009. Multimodality got more attention in SiMPE 2008. In SiMPE 2007, the focus was on developing regions.

Walking-In-Place (WIP) techniques have potential in terms of solving the problem arising when an immersive virtual environment offers a larger freedom of movement than the physical environment. Such techniques are particularly useful when the spatial constraints are very prominent, as they are likely to be in relation to immersive gaming systems located in the homes of consumers. However, most existing WIP techniques rely on movement of the legs which may cause users, wearing a head mounted display, to unintentionally move. This paper details a within-subjects study performed with the intention of investigating how two alternative types of gestural input relying on arm and hip movements compare to the traditional WIP gesture and keyboard input. Visual feedback was delivered through a head-mounted display and auditory feedback was provided by means of a 16-channel surround sound system. The gestures were evaluated in terms of perceived naturalness, presence and real world positional drift. The results suggest that both WIP and arm swinging are perceived as significantly more natural than hip movement and the keyboard configuration. However, arm swinging better matched real walking in terms of energy expenditure and led to significantly less positional drift.

Walking-In-Place interaction techniques seem particularly useful in relation to immersive virtual environments where the user's movement is greatly constrained by a limited physical space. However, current techniques may not be particularly useful in combination with head-mounted displays since many users unintentionally move forward while walking in place. We refer to this phenomenon accidental movement as Unintended Positional Drift. The poster presents evidence of the phenomenon's existence and subsequently discusses different design solutions which potentially could circumvent the problem.

We describe two psychophysical experiments where we have quantified how much humans can unknowingly be redirected by auditory stimuli. The experiments use a two-alternative-forced-choice task. We tested 19 subjects in two different experiments: (E1) discrimination between virtual and physical rotation, and (E2) discrimination of path curvature. In experiment E1 subjects performed rotations with different gains, and then had to choose whether the auditory perceived rotation was smaller or greater than the physical rotation. In experiment E2 subjects estimate the path curvature when walking a curved path in the real world while the visual display shows a straight path in the virtual world. Our results show that users can be turned physically about 20% percent more or 12% less than the perceived virtual rotation, and users can be redirected on a circular arc while they believe they are walking straight.

This article investigates whether auditory feedback affects natural locomotion patterns. Individuals were provided with footstep sounds simulating different surface materials. The sounds were interactively generated using shoes with pressure sensors. Results showed that subjects' walking speed changed as a function of the type of simulated ground material. This effect may arise due to the presence of conflicting information between the auditory and foot-haptic modality, or because of an adjustment of locomotion to the physical properties evoked by the sounds simulating the ground materials. The results reported in this study suggest that auditory feedback may be more important in the regulation of walking in natural environments than has been acknowledged. Furthermore, auditory feedback could be used to develop novel approaches to the design of therapeutic and rehabilitation procedures for locomotion.

In this paper we describe an experiment whose goal is to investigate the role of physics-based auditory and haptic feedback provided at feet level to enhance realism in a virtual environment. To achieve this goal, we designed a multimodal virtual environment where subjects could walk on a platform overlooking a canyon. Subjects were asked to visit the environment wearing an head-mounted display and a custom made pair of sandals enhanced with sensors and actuators. A 12-channels surround sound system delivered a soundscape which was consistent with the visual environment. Passive haptics was provided by having a physical wooden platform present in the laboratory. Subjects reported of having a more realistic experience while auditory and haptic feedback are present. However, measured physiological data and post-experimental presence questionnaire do not show significant differences when audio-haptic feedback is provided.

We propose a system that affords real-time sound synthesis of footsteps on different materials. The system is based on microphones, which detect real footstep sounds from subjects, from which the ground reaction force (GRF) is estimated. Such GRF is used to control a sound synthesis engine based on physical models. Two experiments were conducted. In the first experiment, the ability of subjects to recognize the surface they were exposed to was assessed. In the second experiment, the sound synthesis engine was enhanced with environmental sounds. Results show that, in some conditions, adding a soundscape significantly improves the recognition of the simulated environment.

We describe a system which simulates in realtime the auditory and haptic sensations of walking on different surfaces. The system is based on a pair of sandals enhanced with pressure sensors and actuators. The pressure sensors detect the interaction force during walking, and control several physically based synthesis algorithms, which drive both the auditory and haptic feedback. The different hardware and software components of the system are described, together with possible uses and possibilities for improvements in future design iterations.

A system to synthesize in real-time the sound of footsteps on different materials is presented. The system is based on microphones which allow the user to interact with his own footwear. This solution distinguishes our system from previous efforts that require specific shoes enhanced with sensors. The microphones detect real footsteps sounds from users, from which the ground reaction force (GRF) is estimated. Such GRF is used to control a sound synthesis engine based on physical models. Evaluations of the system in terms of sound validity and fidelity of interaction are described.

In this paper, we present an experiment whose goal is to investigate the role of sound and vision in the recognition of different surface profiles in a walking scenario. Fifteen subjects participated to two within-subjects experiments where they were asked to interact with a desktop system simulating bumps, holes and flat surfaces by means of audio, visual and audio-visual cues. Results of the first experiment show that participants are able to successfully identify the surface profiles provided through the proposed audio-visual techniques. Results of a second experiment in which conflictual audiovisual stimuli were presented, reveal that for some of the proposed visual effects the visual feedback is dominant on the auditory one, while for the others the role of dominance is inverted.

Acoustic simulation of friction is a particularly challenging task, because continuous (strong) contact conditions require a tight and veridical integration of the synthesis layer with the control input. This paper presents an algorithmic realization that combines recently proposed physical models of friction with the lumped modal description of resonating bodies. It is shown that the resulting nonlinear dynamical system can be discretized using a numerical technique that allows efficient and accurate simulation. Applications in the context of interactive audio-visual animation on low-cost general-purpose computers are demonstrated, and an approach to joint audio-visual synthesis is proposed that provides fine-scale synchronization and high coherence between the two modalities. The interactive animations show that the model is successful in reproducing several salient everyday sound phenomena, such as rubbing, braking, and squeaky doors.