Creating sound by manipulating the location of sound-producing objects using gestures is an interesting interaction paradigm. To understand it better, we analyzed videos of users interacting with ‘Random Access Lattice’. In this sound installation, users move a loudspeaker to explore sound laid out in space using a time to space mapping. We performed an inductive analysis of user movement in relation to the visitor intention and the sonic outcome. We identified several body, hand, and grip gestures which were performed with different movement qualities to manipulate the loudspeaker at variable speeds, orientations, and body areas. These were used to search and trace the sonic material in a goal-oriented fashion but also to interact creatively with sound by looping and modulating sounds. We provide a visual index of our findings which can be used when designing gesture interactions with sound.

Abstract Location monitoring is a common task that is typically performed using visual displays that may constrain user location and visual attention. Using spatial audio to present the location of the monitored target could help relax such constraints. To evaluate this hypothesis, we conducted three experiments in which the location monitoring display modality, location, cognitive load and the task spatial resolution were varied. Visual and audiovisual location monitoring resulted in a higher location monitoring accuracy and speed but it induced a significantly higher dual-task cost compared to auditory when both displays were not within peripheral vision. Furthermore, auditory location monitoring accuracy approximated visual when the spatial resolution required by the task was adapted to auditory localization accuracy. The results show that using spatial audio to create multimodal location monitoring can reduce visual attention load and increase the flexibility of user placement relative to the monitoring display without incurring an additional location monitoring cost.

Interactive systems are traditionally evaluated against an 'intended' use by involving external participants. This approach has been challenged recently because of difficulties in addressing applications without an `intended' use or an 'intended' interpretation, but also because the propositionality of the evaluation medium may not address the aesthetics of interactive systems sufficiently. We turn our attention to the evaluation of interactive art, in which, although both difficulties emerge, traditional evaluation methods are commonly used. In trying to stay open to interpretation and address aesthetic thinking and knowledge, we introduce and apply creative evaluation. Ten artists were asked to both direct themselves enacting their interaction experience and to express it using artistic media. Inspiration was obtained in two interactive installations. The results of this experiment demonstrate the ability of artistic practice to maintain interpretation variability and its capacity to address aesthetic thinking and knowledge in the evaluation of interactive systems.

We present a study that investigates whether multimodal audiovisual presentation improves the recall of visuospatial sequences. In a modified Corsi block-tapping experiment, participants were asked to serially recall a spatial stimulus sequence in conditions that manipulated stimulus modality and sequence length. Adding spatial sound to the visuospatial sequences did not improve serial spatial recall performance. The results support the hypothesis that no modality-specific components in spatial working memory exist.

We investigated the detection of sound displacement in a four-voice musical piece under conditions that manipulated the attentional setting (selective or divided attention), the sound source numerosity, the spatial dispersion of the voices, and the tonal complexity of the piece. Detection was easiest when each voice was played in isolation and performance deteriorated when source numerosity increased and uncertainty with respect to the voice in which displacement would occur was introduced. Restricting the area occupied by the voices improved performance in agreement with the auditory spotlight hypothesis as did reducing the tonal complexity of the piece. Performance under increased numerosity conditions depended on the voice in which displacement occurred. The results highlight the importance of top-down processes in the context of the detection of spatial displacement in a musical scene.

Listening and interacting with audio commonly relies on using earphones which limit the ability of users to perceive their auditory environment. Earphone sets that integrate miniature microphones on their exterior can, however, be used to hear-through the auditory environment. We present an evaluation study in which sound localization when wearing such a hear-through system is compared to normal earphones, open headphones and unblocked ears. Although localization performance is improved compared to open headphones, we find that it is compromised in comparison to listening without earphones because confusions of sound direction increase and localization judgment distributions are more dispersed and show a weaker correlation to the test directions. The implications of the results to human computer interaction and possible improvements to hear-through system design are discussed.

In two experiments, visual cues from gesture control of spatialization were found to affect auditory movement perception depending on the identifiability of auditory motion trajectories, the congruency of audiovisual stimulation, the sensory focus of attention, and the attentional process involved. Visibility of the performer’s gestures improved spatial audio trajectory identification, but it shifted the listeners’ attention to vision, impairing auditory motion encoding in the case of incongruent stimulation. On the other hand, selectively directing attention to audition resulted in interference from the visual cues for acoustically ambiguous trajectories. Auditory motion information was poorly preserved when dividing attention between auditory and visual movement feedback from performance gestures. An auditory focus of attention is a listener strategy that maximizes performance, due to the improvement caused by congruent visual stimulation and its robustness to interference from incongruent stimulation for acoustically unambiguous trajectories. Attentional strategy and auditory motion calibration are two aspects that need to be considered when employing gesture control of spatialization.

We present the results of an empirical study investigating the effect of feedback, mobility and index of difficulty on a deictic spatial audio target acquisition task in the horizontal plane in front of a user. With audio feedback, spatial audio display elements are found to enable usable deictic interac-tion that can be described using Fitts law. Feedback does not affect perceived workload or preferred walking speed compared to interaction without feedback. Mobility is found to degrade interaction speed and accuracy by 20%. Participants were able to perform deictic spatial audio target acquisition when mobile while walking at 73% of their pre-ferred walking speed. The proposed feedback design is ex-amined in detail and the effects of variable target widths are quantified. Deictic interaction with a spatial audio display is found to be a feasible solution for future interface designs.

An empirical study that compared six different feedback cue types to enhance pointing efficiency in deictic spatial audio displays is presented. Participants were asked to select a sound using a physical pointing gesture, with the help of a loudness cue, a timbre cue and an orientation update cue as well as with combinations of these cues. Display content was varied systematically to investigate the effect of increasing display population. Speed, accuracy and throughput ratings are provided as well as effective target widths that allow for minimal error rates. The results showed direct pointing to be the most efficient interaction technique; however large effective target widths reduce the applicability of this technique. Movement-coupled cues were found to significantly reduce display element size, but resulted in slower interaction and were affected by display content due to the requirement of continuous target attainment. The results show that, with appropriate design, it is possible to overcome interaction uncertainty and provide solutions that are effective in mobile human computer interaction.