This paper explores several ways to drive a music-oriented computer system by push-button controls, with a particular focus on music education for young children and individuals with disabilities.The research investigates a range of interaction paradigms where heterogeneous push-button actions can be mapped onto musical functions, such as triggering Note-On/Note-Off events, dynamically controlling other musical parameters, or playing and stopping pre-recorded sequences.The ultimate goal is to propose a hardware/software ecosystem that utilizes button-based human-computer interfaces that are not specialized for music (e.g., joypads or colored computer keyboards).These paradigms are designed to lower the barrier to entry for engaging with music, making it accessible even to those with limited motor skills or no prior musical training.To this end, we propose an implementation where multiple push-button devices can be connected to a hub that communicates with a computer, and the role of the latter is to associate a customizable musical meaning to button events in the framework of inclusive music education.

Navigation assistance services for people with visual impairment pose the challenge of providing accurate guidance through non-visual navigation instructions. This paper proposes two techniques to guide the user during navigation assistance, and in particular during turns and straight paths. Both techniques adopt a combination of speech and non-speech audio with the aim of providing continuous, accurate, and unobtrusive guidance. The two solutions differ in the sonification technique used to generate the non-speech audio. The techniques were evaluated in a real-world environment with nine participants having severe visual impairment. Experimental results show that one of the proposed techniques is significantly more effective than a baseline approach adopted in existing solutions, in terms of navigation accuracy during turns and straight paths. Thus, it is a practical and effective solution to the problem of non-visual navigation assistance for people with visual impairment.

People with blindness or severe low vision access mobile devices using screen readers. However, noisy environments can impair screen reader intelligibility. During mobility, this could disorient or even endanger the user. To address this issue, we propose three screen reader speech compensation techniques based on environmental noise: speech rate slowing, adaptive volume increase, and adaptive equalization. Through a study with 12 participants in three simulated noise scenarios, we evaluate screen reader intelligibility and the perceived distraction from the soundscape, with and without compensations. Four of the proposed compensations, in particular those that pair speech rate reduction with volume or equalization adaptation, significantly improve screen reader's speech intelligibility in all the considered scenarios, and the compensations do not have a significant impact on the distraction from the soundscape.

In immersive Audio Augmented Reality, a virtual sound source should be indistinguishable from the existing real ones. This property can be evaluated with the co-immersion criterion, which encompasses scenes constituted by arbitrary configurations of real and virtual objects. Thus, we introduce the term Audio Augmented Virtuality (AAV) to describe a fully virtual environment consisting of auditory content captured from the real world, augmented by synthetic sound generation. We propose an experimental design in AAV investigating how simplified late reverberation (LR) affects the co-immersion of a sound source. Participants listened to simultaneous virtual speakers dynamically rendered through spatial Room Impulse Responses, and were asked to detect the presence of an impostor, i.e., a speaker rendered with one of two simplified LR conditions. Detection rates were found to be close to chance level, especially for one condition, suggesting a limited influence on co-immersion of the simplified LR in the evaluated AAV scenes. This methodology can be straightforwardly extended and applied to different acoustics scenes, complexities, i.e., the number of simultaneous speakers, and rendering parameters in order to further investigate the requirements for immersive audio technologies in AAR and AAV applications.

Motivated by the state-of-art psychological research, we note that a piano performance transcribed with existing Automatic Music Transcription (AMT) methods cannot be successfully resynthesized without affecting the artistic content of the performance. This is due to 1) the different mappings between MIDI parameters used by different instruments, and 2) the fact that musicians adapt their way of playing to the surrounding acoustic environment. To face this issue, we propose a methodology to build acoustics-specific AMT systems that are able to model the adaptations that musicians apply to convey their interpretation. Specifically, we train models tailored for virtual instruments in a modular architecture that takes as input an audio recording and the relative aligned music score and outputs the acoustics-specific velocities of each note. We test different model shapes and show that the proposed methodology generally outperforms the usual AMT pipeline which does not consider the specificities of the instrument and of the acoustic environment. Interestingly, such a methodology is extensible in a straightforward way since only slight efforts are required to train models for the inference of other piano parameters, such as pedaling.

Abstract This study focuses on the perception of music performances when contextual factors, such as room acoustics and instrument, change. We propose to distinguish the concept of “performance” from the one of “interpretation”, which expresses the “artistic intention”. Towards assessing this distinction, we carried out an experimental evaluation where 91 subjects were invited to listen to various audio recordings created by resynthesizing MIDI data obtained through Automatic Music Transcription (AMT) systems and a sensorized acoustic piano. During the resynthesis, we simulated different contexts and asked listeners to evaluate how much the interpretation changes when the context changes. Results show that: (1) MIDI format alone is not able to completely grasp the artistic intention of a music performance; (2) usual objective evaluation measures based on MIDI data present low correlations with the average subjective evaluation. To bridge this gap, we propose a novel measure which is meaningfully correlated with the outcome of the tests. In addition, we investigate multimodal machine learning by providing a new score-informed AMT method and propose an approximation algorithm for thep-dispersion problem.

A Digital Audio Workstation (DAW) is a hardware and/or software device aiming to ease those operations required for music production, such as arranging, recording, editing, mixing, and, more in general, modifying sounds creatively. A peculiarity of a DAW environment is that most of the work is highly parallelizable, since the basic architecture of a DAW consists in the simultaneous processing of different audio tracks, mainly independent from each other. In order to exploit such a feature, this paper proposes an interface that lets the DAW interact with the Graphics Processing Unit (GPU) in a standardized way. Despite some academic research and experimentation, the professional audio software industry almost never exploited GPUs when implementing entire DAWs, but only when realising very specific tools or third party extensions (plugins). This work also presents and discusses the outcomes of a number of tests conducted in order to choose the optimal architecture. As a result, a GPU-based approach turned to be a valid alternative to the use of CPUs in the computation of audio effects, such as the rendering of audio tracks after mixing and mastering operations, both in real time and offline.

This paper deals with Harmonic Touch, a Web platform designed to foster the practice of tonal harmony also in young children.The work focuses on one of the experiences provided by the framework, namely the gamification of harmonic change recognition in songs.The platform, specifically equipped with new features to accommodate the needs of teachers during the COVID-19 pandemic, has been tested in two Italian schools in February 2021.Early experimental results about the main difficulties encountered by the children during the games are presented and discussed.

Audio-to-score alignment (A2SA) is a multimodal task consisting in the alignment of audio signals to music scores. Recent literature confirms the benefits of Automatic Music Transcription (AMT) for A2SA at the frame-level. In this work, we aim to elaborate on the exploitation of AMT Deep Learning (DL) models for achieving alignment at the note-level. We propose a method which benefits from HMM-based score-to-score alignment and AMT, showing a remarkable advancement beyond the state-of-the-art. We design a systematic procedure to take advantage of large datasets which do not offer an aligned score. Finally, we perform a thorough comparison and extensive tests on multiple datasets.

This paper presents AREmbody, an augmented-reality mobile application for the development of tonal harmony awareness.Continuing from previous prototypes based on full body and desktop interaction, AREmbody benefits from a very simple portable setup which allows physical interaction and supports the activity of one or more users.The application combines in a single mobile device a video processor, a media player and a movement tracker, opening the way to the design of harmonic games with challenges and recordable scores.Thus the application not only fosters music education activities in the classroom, but also extends them outside the school times and places, promoting educational practices in informal and private contexts.

Accessible Digital Musical Instruments (ADMI) are increasingly raising interest within the scientific community, especially in the contexts of Sound and Music Computing and Human-Computer Interaction.In the past, Netytar has been proposed among these.Netytar is a software ADMI operated through the eyes using an eye tracker and an additional switch or sensor (e.g., a breath sensor).The instrument is dedicated to quadriplegic users: it belongs to the niche of gaze operated musical instruments, and has been proven effective and functional through testing.Although there are several other gaze operated ADMIs available in market and literature, a formal method for studying music with them has not yet been proposed.The present work introduces a simple study method based on a set of exercises.This can be useful for approaching musical performance with Netytar, but it's also potentially generalizable for learning other similar instruments.The exercises are illustrated, discussed and explained in view of an improvement.A simple musical notation is introduced.At the end of a learning cycle, a user is expected to be able to perform simple melodies, and have a basis with which to learn other new ones.In the future, the method will be tested with the target users.

Independent mobility is one of the main challenges for blind or visually impaired (BVI) people. In particular, BVI people often need to identify and avoid nearby obstacles, for example a bicycle parked on the sidewalk. This is generally achieved with a combination of residual vision, hearing and haptic sensing using the white cane. However, in many cases, BVI people can only perceive obstacles at short distance (typically about 1m, i.e., the white cane detection range), in other situations obstacles are hard to detect (e.g., those elevated from the ground), while others should not be hit by the white cane (e.g., a standing person). Thus, some time and effort are required to identify the object in order to understand how to avoid it. A solution to these problems can be found in recent computer vision techniques that can run on mobile and wearable devices to detect obstacles at a distance. However, in addition to detecting obstacles, it is also necessary to convey information about them to a BVI user. This contribution presents WatchOut, a sonification technique for conveying real-time information about the main characteristics of an obstacle to a BVI person, who can then use this additional feedback to safely navigate in the environment. WatchOut was designed with a user-centric approach, involving two iterations of online questionnaires with BVI participants in order to define, improve and evaluate the sonification technique. WatchOut was implemented and tested as a module of a mobile app that detects obstacles using state-of-the-art computer vision technology. Results show that the system is considered usable, and can guide the users to avoid more than 85% of the obstacles.

Indoor navigation services for people with visual impairment are being researched in academia, and working systems have already been deployed in public places. While previous research mainly focuses on computing the user's position with high accuracy, providing non-visual navigation instructions is also a challenge and naive approaches can fail in helping users reach their target destination or even expose them to hazards.In this paper we investigate the problem of guiding users to rotate towards a target direction. We propose three different sonification techniques that provide continuous guidance during rotation, and we compare them to a single-impulse baseline, used in previous works. We also explore three variations that reinforce the proposed techniques by combining them with the baseline. A preliminary study with 10 blind participants highlights two dominant techniques, which we analyze through a follow-up study with 18 participants, from 2 groups with very distant cultural backgrounds. While stark differences emerge in the performance from the two groups, we highlight two clear results common to both: 1) one of the proposed techniques significantly outperforms the baseline, reducing the average rotation error by a factor of 3.5 (from 11° to 3°); 2) the interaction speed of this technique, generally slower than the baseline, significantly improves when combined with the baseline technique.

Indoor navigation is an important service, currently investigated both in industry and academia. While the main focus of research is the computation of users' position, the additional challenge of conveying guidance instructions arises when the target user is blind or visually impaired (BVI). This contribution presents our ongoing research aimed at adopting sonification techniques to guide a BVI person. In particular we introduce three sonification techniques to guide the user during rotations. Preliminary results, conducted with 7 BVI people, show that some of the proposed sonification technique outperform a benchmark solution adopted in previous contributions.

Achieving a full 3D auditory experience with head-related transfer functions (HRTFs) is still one of the main challenges of spatial audio rendering. HRTFs capture the listener's acoustic effects and personal perception, allowing immersion in virtual reality (VR) applications. This paper aims to investigate the connection between listener sensitivity in vertical localization cues and experienced presence, spatial audio quality, and attention. Two VR experiments with head-mounted display (HMD) and animated visual avatar are proposed: (i) a screening test aiming to evaluate the participants' localization performance with HRTFs for a non-visible spatialized audio source, and (ii) a 2 minute free exploration of a VR scene with five audiovisual sources in a both non-spatialized (2D stereo panning) and spatialized (free-field HRTF rendering) listening conditions. The screening test allows a distinction between good and bad localizers. The second one shows that no biases are introduced in the quality of the experience (QoE) due to different audio rendering methods; more interestingly, good localizers perceive a lower audio latency and they are less involved in the visual aspects.

This paper deals with the issue of individualizing the head-related transfer function (HRTF) rendering process for auditory elevation perception. Is it possible to find a nonindividual, personalized HRTF set that allows a listener to have an equally accurate localization performance than with his/her individual HRTFs? We propose a psychoacoustically motivated, anthropometry based mismatch function between HRTF pairs that exploits the close relation between the listener's pinna geometry and localization cues. This is evaluated using an auditory model that computes a mapping between HRTF spectra and perceived spatial locations. Results on a large number of subjects in the center for image processing and integrated computing (CIPIC) and acoustics research institute (ARI) HRTF databases suggest that there exists a nonindividual HRTF set, which allows a listener to have an equally accurate vertical localization than with individual HRTFs. Furthermore, we find the optimal parameterization of the proposed mismatch function, i.e., the one that best reflects the information given by the auditory model. Our findings show that the selection procedure yields statistically significant improvements with respect to dummy-head HRTFs or random HRTF selection, with potentially high impact from an applicative point of view.

Virtual and augmented realities are expected to become more and more important in everyday life in the next future; the role of spatial audio technologies over headphones will be pivotal for application scenarios which involve mobility. This paper introduces the SelfEar project, aimed at low-cost acquisition and personalization of Head-Related Transfer Functions (HRTFs) on mobile devices. This first version focuses on capturing individual spectral features which characterize external ear acoustics, through a self-adjustable procedure which guides users in collecting such information: their mobile device must be held with the stretched arm and positioned at several specific elevation points; acoustic data are acquired by an audio augmented reality headset which embeds a pair of microphones at listener ear-canals. A preliminary measurement session assesses the ability of the system to capture spectral features which are crucial for elevation perception. Moreover, a virtual experiment using a computational auditory model predicts clear vertical localization cues in the measured features.

A novel approach to the selection of generic head-related transfer functions (HRTFs) for binaural audio rendering through headphones is formalized and described in this paper. A reflection model applied to the user's ear picture facilitates extraction of the relevant anthropometric cues that are used for selecting two HRTF sets in a database fitting that user, whose localization performances are evaluated in a complete psychoacoustic experiment. The proposed selection increases the average elevation performances of 17% (with a peak of 34%) with respect to generic HRTFs from an anthropomorphic mannequin. It also significantly enhances externalization and reduces the number of up/down reversals.

The contribution of the external ear to the head-related transfer function (HRTF) heavily depends on the listener's unique anthropometry. In particular, the shape of the most prominent contours of the pinna defines the frequency location of the HRTF spectral notches along the elevation of the sound source. This paper addresses the issue of automatically estimating the location of pinna edges starting from a set of pictures produced by a multi-flash imaging device. A basic image processing algorithm designed to obtain the principal edges and their distance from the ear canal entrance is described. The effectiveness of the developed hardware and software is preliminarily evaluated on a small number of test subjects.

The paper presents a system for customized binaural audio delivery based on the extraction of the relevant features from a 2-D representation of the listener's pinna. A procedure based on multi-flash imaging for recognizing the main contours of the pinna and their position with respect to the ear canal entrance is detailed. The resulting contours drive the parametrization of a structural head-related transfer function model that performs in real time the spatialization of a desired sound file according to the listener's position with respect to the virtual sound source, tracked by sensor-equipped headphones. The low complexity of the model allows smooth implementation and delivery on any mobile device. The purpose of the desired system is to provide low-tech custom binaural audio to any user without the need of tedious and cumbersome subjective measurements.

This paper studies the relationship between head-related transfer functions (HRTFs) and pinna reflection patterns in the frontal hemispace. A pre-processed database of HRTFs allows extraction of up to three spectral notches from each response taken in the median sagittal plane. Ray-tracing analysis performed on the obtained notches' central frequencies is compared with a set of possible reflection surfaces directly recognizeable from the corresponding pinna picture. Results of such analysis are discussed in terms of the reflection coefficient sign, which is found to be most likely negative. Based on this finding, a model for real-time HRTF synthesis that allows to control separately the evolution of different acoustic phenomena such as head diffraction, ear resonances, and reflections is proposed through the design of distinct filter blocks. Parameters to be fed to the model are derived either from analysis or from specific anthropometric features of the subject. Finally, objective evaluations of reconstructed HRTFs in the chosen spatial range are performed through spectral distortion measurements.

A physically based impact model-already known and exploited in the field of sound synthesis-is studied using both analytical tools and numerical simulations. It is shown that the Hamiltonian of a physical system composed of a mass impacting on a wall can be expressed analytically as a function of the mass velocity during contact. Moreover, an efficient and accurate approximation for the mass outbound velocity is presented, which allows to estimate the Hamiltonian at the end of the contact. Analytical results are then compared to numerical simulations obtained by discretizing the system with several numerical methods. It is shown that, for some regions of the parameter space, the trajectories of the discretized systems may significantly drift from the analytically derived curves. Two approaches, based on enforcing numerical energy consistency, are then proposed to improve the accuracy of numerical simulations.

This paper faces the general problem of modeling pinna-related transfer functions (PRTFs) for 3-D sound rendering. Following a structural approach, we aim at constructing a model for PRTF synthesis which allows to control separately the evolution of ear resonances and spectral notches through the design of two distinct filter blocks. Taking such model as endpoint, we propose a method based on the McAulay-Quatieri partial tracking algorithm to extract the frequencies of the most important spectral notches. Ray-tracing analysis performed on the so obtained tracks reveals a convincing correspondence between extracted frequencies and pinna geometry of a bunch of subjects.

This paper presents a set of novel physical models for sound synthesis of membrane percussion instruments. First, a model for tension modulation in a struck circular membrane is discussed, which simulates the dynamic variations of partial frequencies occurring at large amplitude vibrations of the membrane. It is shown that the model, which is derived from a more general theory of nonlinear elastic plates, can be efficiently integrated into a modal synthesis engine. Novel models for two relevant sound production mechanisms in membrane percussions are then proposed, i.e., coupling between two membranes through enclosed air in two-headed percussions, and string-membrane coupling. Both are based on a lumped modeling approach and can be straightforwardly connected to the nonlinear membrane model. By virtue of this modular approach, individual elements (circular linear/nonlinear membranes, impact force, membrane coupling through air, string-membrane coupling) can be combined to form different instruments. The acoustic results of the proposed models are demonstrated by means of analysis of numerical simulations.

This paper reports on a study on the perception and rendering of distance in multimodal virtual environments. A model for binaural sound synthesis is discussed, and its integration in a real-time system with motion tracking and visual rendering is presented. Results from a validation experiment show that the model effectively simulates relevant auditory cues for distance perception in dynamic conditions. The model is then used in a subsequent experiment on the perception of egocentric distance. The design and preliminary result from this experiment are discussed.

Abstract This paper presents a multimodal rendering architecture that integrates physically based sound models with haptic and visual rendering. The proposed sound modeling approach is compared to other existing techniques. An example of implementation of the architecture is presented, that realizes bimodal (auditory and haptic) rendering of contact stiffness. It is shown that the proposed rendering scheme allows tight synchronization of the two modalities, as well as a high degree of interactivity and responsiveness of the sound models to gestures and actions of a user. Finally, an experiment on the relative contributions of haptic and auditory information to bimodal judgments of contact stiffness is presented. Experimental results support the effectiveness of auditory feedback in modulating haptic perception of stiffness. Copyright © 2006 John Wiley & Sons, Ltd.

Acoustic simulation of friction is a particularly challenging task, because continuous (strong) contact conditions require a tight and veridical integration of the synthesis layer with the control input. This paper presents an algorithmic realization that combines recently proposed physical models of friction with the lumped modal description of resonating bodies. It is shown that the resulting nonlinear dynamical system can be discretized using a numerical technique that allows efficient and accurate simulation. Applications in the context of interactive audio-visual animation on low-cost general-purpose computers are demonstrated, and an approach to joint audio-visual synthesis is proposed that provides fine-scale synchronization and high coherence between the two modalities. The interactive animations show that the model is successful in reproducing several salient everyday sound phenomena, such as rubbing, braking, and squeaky doors.

This paper describes an algorithm for real-time synthesis of contact sounds for interactive simulations and animation. The algorithm is derived from a physically-based impact model, and the acoustic characteristics of colliding objects can be realistically simulated by properly adjusting the physical parameters of the model. A technique for describing the spatial dynamics of a resonating object is proposed, which allows simulation of position-dependent interaction. It is shown that the numerical implementation leads to an efficient sound synthesis module, that runs in real-time on low cost platforms. The effectiveness of the model is demonstrated, and its applications are discussed.

A lumped physical model of the glottal source is presented. Vocal folds are described as single masses, but vertical phase differences between upper and lower margins of the folds are taken into account by appropriately describing the non-linear interaction of the mechanical model with aerodynamics. This results in a modified one-mass model, or a “one-delayed-mass model”. Analysis on numerical simulations shows that the system behaves qualitatively as higher-dimensional models (such as the two-mass model by Ishizaka and Flanagan); in particular, control over flow skewness is guaranteed, allowing for synthesis of realistic glottal flow waveforms. As only one degree of freedom (one mass) is needed in the model, structure and number of parameters are drastically reduced, thus making it suitable for real-time synthesis applications.