Mixed reality (MR) technologies hold promise for distributed musical collaboration, yet the needs of professional musicians remain underexplored. We conducted a set of co-design workshops with 24 professional musicians, organized into six four-piece bands, to envision future MR systems for networked music performance. Using a three-phase process (hands-on MR experience, individual brainwriting, and collaborative consensus-building), we identified core requirements that challenge current MR paradigms. Professional musicians operate from an “Audio-First” framework where sonic experience constitutes the primary reality, contradicting visual-first MR approaches. Five unanimous requirements emerged: audio-first workflow, support for non-verbal communication, personalized audio control, spatial layout customization, and plug-and-play simplicity. Other high-consensus themes included real-time voice communication and realistic avatar representation. Taken together, our findings reveal that professional creative collaboration demands fundamentally different design priorities than traditional office-centered collaborative MR applications, requiring systems that support established workflows and nonverbal communication channels while prioritizing workflow preservation over visual immersion.

Music is increasingly performed and experienced in Social Virtual Reality (Social VR), from VRChat raves to high-production concerts on bespoke platforms. Yet Human-Computer-Interaction (HCI) research still focuses mainly on building new VR systems rather than examining the communities that already create and sustain these practices. We present a cultural mapping of the Social VR music scene based on 84 survey responses, 27 interviews, and 17 event observations with diverse stakeholders, including audience members, musicians, developers, platform owners, and event organisers. We found that the scene operates as a fragmented cross-platform ecosystem sustained by user-generated infrastructure and continuous community labour. The bottom-up organisation produces role fluidity with individuals dynamically shifting between roles as performers, world builders, organisers, and audience members. However, the openness that enables this creativity also creates tensions between expectations of free access and the financial and emotional labour required to keep events running. Taken together, our findings reveal the vibrant cultural practices that continue to flourish in Social VR, even as corporate narratives declare the “metaverse” dead.

Networked Music Performance (NMP) systems have traditionally focused on optimizing audio transmission for remote collaboration, relying on basic 2D video feeds that fail to provide the spatial awareness and visual cues crucial for coordinated musical performance. While Mixed Reality (MR) technology offers promising solutions for enhanced musical collaboration, its integration into NMP systems remains understudied, particularly regarding the balance between immersion and usability. This study investigates this gap by comparing traditional 2D video conferencing setups with MR environments using point cloud representations and spatial audio. Through an experimental study with professional electric guitar players (N=16) and a simulation-based approach with pre-recorded setup, we examined how both 2D video and MR setups affect musicians’ performance, movement, and sense of presence using quantitative movement analysis and qualitative feedback. Head movement analysis revealed significantly reduced motion in MR compared to 2D conditions (p < 0.001), while detrended fluctuation analysis showed more structured movement patterns in MR (p = 0.0028). Post-task evaluations demonstrated that while MR enhanced immersion and presence (p < 0.001), participants reported significantly better performance coherence and reduced technical interference with 2D videos (p < 0.001). Spatial audio features showed minimal impact on user experience, with visual elements dominating attention. The findings highlight a fundamental tension between immersion and physical expression in MR music performance, suggesting that future systems should prioritize ergonomic considerations alongside technological capabilities. This research contributes to understanding the potential and the limitations of MR technologies in musical collaboration within the emerging Musical Metaverse.

Our work explores how audiences perceive and engage with remote music events. The shift to digital platforms has transformed the live music experience, leading to the rise of remote music performances. Following the COVID-19 pandemic, platforms such as Twitch have been repurposed and surged in popularity as meaningful outlets for live music dissemination. Concurrently, novel platforms specifically crafted for music events have begun to emerge. Understanding the preferences and challenges of remote music audiences is crucial because it guides the development of platforms that effectively answer varied user needs, which could enhance overall engagement and shape the future of music consumption. For this, we employed a qualitative multi-method approach, including a survey, interviews, and event observations, to start mapping the design space of remote music performances and capture a wide spectrum of audience perspectives. Our findings reveal audience engagement patterns in remote music events across different digital platforms, highlighting unmet needs, desires as well as opportunities for future HCI work in this area.

Previous research has shown that the vibrotactile enhancement of music signals can be effective in creating intense musical experiences. Virtual Reality is emerging as a new medium with which experiencing music and inducing emotions, but the study of the effects of tactile enrichments of music in virtual environments (VEs) has been largely overlooked thus far. It is unknown whether the same results found for real environments hold for VEs too, and what is the role played by vibrations in altering the emotion, perception and behavior of listeners in such spaces. To bridge these gaps, we created 12 affective musical VEs accompanying as many musical pieces and assessed the impact of adding synchronous vibrations. Results showed that the VEs involving the vibrations received significantly higher ratings of valence, arousal and presence compared to their audio-visual counterparts. The majority of participants preferred the condition with the vibrations. Cybersickness was not reduced in presence of vibrations, but it was found to be significantly higher for females than males. Although most participants reported that they were induced to move to a greater extent in presence of vibrations, the recorded motion of their head and hands did not increase when vibrations were provided.

The paradigm of context-aware computing allows storing situational and environmental information in such a way that its interpretation can be done easily and more meaningfully. In turn, this understanding is used to anticipate users’ needs, and proactively provide them with situation-aware content and experiences. Whereas context-awareness has been investigated extensively in the computer science and IoT disciplines, it has been largely overlooked by the research community dealing with musical interfaces design. Existing musical instruments are not equipped with the ability to understand the context around them, namely who is the musician playing them, what musical activity is being conducted, as well as where and when. Enhancing musical instruments with context-awareness has the concrete potential to enable novel kinds of interactions between musicians and musical content in a large variety of situations, from playing alone to playing in a group, from music learning to music composition. To accomplish such a vision of intelligence embedded in musical instruments it is necessary to model the context around their users. In this paper, we present an ontology devised to represent the knowledge related to musicians and musical activities, the “Musician’s Context Ontology” (MUSICO) to facilitate the development of context-aware musical applications. There was no previous comprehensive data model for the domain of musicians’ context, nevertheless, the new ontology relates to several existing ontologies, including the Internet of Musical Things Ontology to represent Internet of Musical Things ecosystems and the Music Ontology that deals with the description of the music value-chain from production to consumption. This paper documents the design of the ontology and its evaluation with respect to specific requirements gathered from an extensive literature review and interviews with musicians. The utility of the ontology is demonstrated by a smartphone application that enables to search for musicians based on both textual and content-based musical queries. MUSICO can be accessed at: https://w3id.org/musico# .

Networked Music Performances (NMPs), where geographically displaced musicians play together over a data network, represent a challenging application for today’s wireless communications. This is due to the stringent constraints on latency, throughput, and reliability that need to be obeyed in order to achieve a satisfactory quality of experience for the musicians. Slicing is a promising feature of 5G networks in the context of NMP applications, as it makes it possible to isolate the networking and computing resources allocated to NMP devices. However, the use of slicing has been scarcely investigated for the NMP context so far. Moreover, previous works focusing on NMPs over 5G involved up to 4 nodes. To bridge these gaps, we study 5G performance in support of NMPs involving an architecture with 10 nodes, both with and without slicing. Specifically, we focused on the assessment of the sole wireless link, as the measurements can be easily transferred to a realistic NMP architecture involving a wide area network. Our results show that, in the slicing condition, latency slightly increased due to the realistically different computing specifications of the MEC servers compared to those of the Core Network servers, whereas reliability slightly improved as expected.

While numerous studies have examined 3D interaction techniques for Virtual Reality (VR) across various tasks and scenarios, limited research has focused on music-related applications. However, the most common input techniques in consumer VR systems have been developed outside of the musical domain. Therefore they have not been tested in tasks where synchronization with auditory stimuli and timing plays a crucial role. There is a lack of empirical knowledge about performance and user experience. This paper presents a comparison of five selection input techniques for VR employing the tapping paradigm commonly utilized in the research on sensorimotor synchronization. We assess asynchrony and timing variance as well as user experience, encompassing factors such as ease of use, workload, and cybersickness of such techniques. The study involved 30 participants, both with and without musical expertise, and encompassed the examination of all techniques using one and two hands. Our analysis yielded several key findings: (1) different input techniques yielded distinct outcomes regarding timing asynchrony and variance; (2) the choice of interaction metaphor significantly influenced the user experience; (3) tracking stability emerged as a critical factor. Building upon these insights, we identified essential considerations for selecting the most suitable technique for music creation in VR and proposed design guidelines and future research directions in this domain.

The integration of emotion recognition capabilities within musical instruments can spur the emergence of novel art formats and services for musicians. This paper proposes the concept of emotionally-aware smart musical instruments, a class of musical devices embedding an artificial intelligence agent able to recognize the emotion contained in the musical signal. This spurs the emergence of novel services for musicians. Two prototypes of emotionally-aware smart piano and smart electric guitar were created, which embedded a recognition method for happiness, sadness, relaxation, aggressiveness and combination thereof. A user study, conducted with eleven pianists and eleven electric guitarists, revealed the strengths and limitations of the developed technology. On average musicians appreciated the proposed concept, who found its value in various musical activities. Most of participants tended to justify the system with respect to erroneous or partially erroneous classifications of the emotions they expressed, reporting to understand the reasons why a given output was produced. Some participants even seemed to trust more the system than their own judgments. Conversely, other participants requested to improve the accuracy, reliability and explainability of the system in order to achieve a higher degree of partnership with it. Our results suggest that, while desirable, perfect prediction of the intended emotion is not an absolute requirement for music emotion recognition to be useful in the construction of smart musical instruments.

The availability of high-performance embedded audio systems, along with high-bandwidth and low-latency connectivity options provided by 5G networks, is enabling the Internet of Musical Things (IoMusT) paradigm. A central component of this paradigm is represented by Networked Music Performances (NMPs), where geographically displaced musicians play together over the network in real time. However, to date, IoMusT deployments over 5G networks remain scarce, and very limited statistical results are available on the actual latency and reliability of 5G networks for IoMusT and NMP scenarios. In this paper, we present a private 5G IoMusT deployment and analyze its performance when supporting NMPs. Our IoMusT system is composed of up to four nodes and includes different background traffic conditions. We focused on the assessment of the sole wireless link, as the measurements can be easily transferred to a realistic NMP architecture involving a WAN by compounding them with those of the WAN. Our results show that latency increases with the number of nodes and with the presence of background traffic, whereas the reliability did not vary with the complexity of the conditions. For all tested scenarios, the average measured latency was below 24 ms (including a jitter buffer of 10.66 ms), whereas packet losses occurred with a probability of less than 0.01. However, irregular spikes were found for all latency and reliability metrics, which can significantly reduce the quality of service perceived by the users of NMP applications. Finally, packet loss and latency resulted to be uncorrelated, which suggests that they have different root causes.

Considering the specific quality-of-service requirements of Internet of Musical Things (IoMusT) deployments, 5G networks are deemed as a key enabling technology to support IoMusT applications as needed, e.g., in networked music performance (NMP) settings. Slicing, in particular, is a promising feature of 5G networks, as it could help dedicate specific resources to NMP traffic, so as to schedule NMP communications with higher priority and thus guarantee swift delivery delays and higher reliability. Moreover, handling NMP traffic via a slice residing on a geographically closer multi-access edge computing (MEC) server would help significantly reduce WAN transit times for NMP traffic. In this paper, we conduct the first tests with slicing in 5G networks supporting IoMusT applications (to the best of our knowledge). We conduct our tests by decoupling WAN transit times from the radio access performance (evaluated in terms of latency, error rate, number of lost packets, and maximum length of a packet loss burst). Our results show that slicing, even when introducing slight extra delays due to UPF implementation or hardware specifications, is a potentially excellent choice to save on WAN transit times for NMP traffic while delivering the same reliability as the operator’s core network equipment.

This article investigated the automatic recognition of felt and musically communicated emotions using electroencephalogram (EEG), electrocardiogram (ECG), and acoustic signals, which were recorded from eleven musicians instructed to perform music in order to communicate happiness, sadness, relaxation, and anger. Musicians' self-reports indicated that the emotions they musically expressed were highly consistent with those they actually felt. Results showed that the best classification performances, in a subject-dependent classification using a KNN classifier were achieved by using features derived from both the EEG and ECG (with an accuracy of 98.11%). Which was significantly more accurate than using ECG features alone, but was not significantly more accurate than using EEG features alone. The use of acoustic features alone or in combination with EEG and/or ECG features did not lead to better performances than those achieved with EEG plus ECG or EEG alone. Our results suggest that emotion detection of playing musicians, both felt and musically communicated, when coherent, can be classified in a more reliable way using physiological features than involving acoustic features. The reported machine learning results are a step toward the development of affective brain–computer interfaces capable of automatically inferring the emotions of a playing musician in real-time.

This paper presents an Internet of Musical Things system devised to support recreational music-making, improvisation, composition, and music learning via vocal queries to an online music repository. The system involves a commercial voice-based interface and the Jamendo cloud-based repository of Creative Commons music content. Thanks to the system the user can query the Jamendo music repository by six content-based features and each combination thereof: mood, genre, tempo, chords, key and tuning. Such queries differ from the conventional methods for music retrieval, which are based on the piece’s title and the artist’s name. These features were identified following a survey with 112 musicians, which preliminary validated the concept underlying the proposed system. A user study with 20 musicians showed that the system was deemed usable, able to provide a satisfactory user experience, and useful in a variety of musical activities. Differences in the participants’ needs were identified, which highlighted the need for personalization mechanisms based on the expertise level of the user. Importantly, the system was seen as a concrete solution to physical encumbrances that arise from the concurrent use of the instrument and devices providing interactive media resources. Finally, the system offers benefits to visually-impaired musicians.

Current sound-based practices and systems developed in both academia and industry point to convergent research trends that bring together the field of Sound and Music Computing with that of the Internet of Things. This paper proposes a vision for the emerging field of the Internet of Sounds (IoS), which stems from such disciplines. The IoS relates to the network of Sound Things, i.e., devices capable of sensing, acquiring, processing, actuating, and exchanging data serving the purpose of communicating sound-related information. In the IoS paradigm, which merges under a unique umbrella the emerging fields of the Internet of Musical Things and the Internet of Audio Things, heterogeneous devices dedicated to musical and non-musical tasks can interact and cooperate with one another and with other things connected to the Internet to facilitate sound-based services and applications that are globally available to the users. We survey the state of the art in this space, discuss the technological and non-technological challenges ahead of us and propose a comprehensive research agenda for the field.

The so-called metaverse relates to a vision of a virtual, digital world which is parallel to the real, physical world, where each user owns and interact through his/her own avatar. Music is one of the possible activities that can be conducted in such a space. The "Musical Metaverse" (MM), the metaverse part which is dedicated to musical activities, is currently in its infancy, although is a concept that is constantly evolving and is progressing at a steady pace. However, to the best of the author's knowledge, as of today an investigation about the opportunities and challenges posed by the MM has not been conducted yet. In this paper, we provide a vision for the MM and discuss what are the opportunities for musical stakeholders offered by current implementations of the MM, as well as we envision those that are likely to occur as the metaverse emerges. We also identify the technical, artistic, ethical, sustainability, and regulatory issues that need to be addressed so for the MM to be created and utilized in efficient, creative, and responsible ways. Given the importance and timeliness of the MM, we believe that a discussion on the related opportunities and concerns is useful to provide developers with guidelines for creating better virtual environments and musical interactions between stakeholders.

Abstract This paper presents an Internet of Musical Things system designed to enhance the singing practices of conventional vocal ensembles with electronic sounds generated by smartphones. The system comprises a small loudspeaker connected to a smartphone running a web-based app that generates sounds locally to the chorister, who uses it while singing. An evaluation of the user experience was conducted through three experiments involving a small choir of 9 choristers and a conductor. In the first experiment, the system was utilized in a co-located setting, where choristers shared the same physical space. In the second experiment, the system was employed in a remote setting, where geographically displaced choristers were connected through a networked music performance system. In the third experiment, a hybrid condition was implemented where part of the choir was co-located and part was remotely connected. Overall, results show that the application can be successfully utilized to augment the practice and experience of choir singing, leading to novel forms of musical expression. We provide a critical reflection where we discuss the lessons learned, strengths, limitations, and possible future developments.

In the past two decades, we have witnessed the diffusion of an increasing number of technologies, products, and applications at the intersection of music and networking. As a result of the growing attention devoted by academy and industry to this area, three main research fields have emerged and progressively consolidated: the Networked Music Performances, Ubiquitous Music, and the Internet of Musical Things. Based on the review of the most relevant works in these fields, this paper attempts to delineate their differences and commonalities. The aim of this inquiry is helping avoid confusion between such fields and achieve a correct use of the terminology. A trend towards the convergence between such fields has already been identified, and it is plausible to expect that in the future their evolution will lead to a progressive blurring of the boundaries identified today.

Immersive listening systems have grown significantly over the past decade and are now an established area of scientific, artistic, and industrial research. However, scarce research has been conducted on musicians' preferences for playing through headphones over binaural spatialization systems with the addition of head tracking, as opposed to classical stereophonic systems. This comparison is essential to optimally support the playing experience with others for cases of remote collaborative playing, individual instrumental practice, individual recreational music-making using backing tracks, and studio recording sessions. In this article, we study the preferences of playing musicians for a stereophonic system versus a binaural head-tracking system composed of Ambisonics technology and binaural synthesis with generalized head-related transfer functions. We conducted two experiments, each with 30 expert musicians, where participants were asked to rate and compare the 2 listening conditions while playing their instrument either seated or standing. Overall, the quantitative and qualitative results indicated a generalized preference for the binaural system with head tracking over the stereophonic system, with higher ratings for localization, immersion, social presence, realism, and connection with other musicians. Moreover, participants moved their heads significantly more in the binaural conditions. This phenomenon may be explained by the higher engagement and arousal due to the improved auditory experience, or alternatively by the presence of embodied music cognition mechanisms that cause a higher degree of exploration to better understand the action–perception loop. These findings highlight the need for progressing current commercial hardware and software systems used by musicians while playing over headphones.

Blockchain technology is impacting several industries, including the creative industries and those operating in the Internet of Things (IoT). Lately, researchers' attention has been devoted to the application of blockchain in the recorded music industry. However, thus far, no research has investigated the use of such technology in the Internet of Musical Things (IoMusT). The IoMusT is a new area emerging in industry and academy as an extension of the IoT to the musical domain. The IoMusT itself, as the IoT, is a distributed network of musical things, which are objects augmented with information and communication technologies serving a musical purpose. The IoMusT vision requires, above all, IoT features such as decentralization, seamless authentication, transparency, data integrity and privacy, and self-maintenance, as well as the musical domain features such as efficient handling of copyrights and speed of royalties payment. Such features can be brought by blockchain. In this paper, we investigate the integration of blockchain technology with the IoMusT, and we name such synthesis “Blockchain-based IoMusT”. We present a vision for this new paradigm in terms of the novel opportunities that are enabled, and we propose a set of application scenarios enabled by technological integration. Finally, we outline the open research directions in this promising area.

This paper investigates to which extent state of the art machine learning methods are effective in classifying emotions in the context of individual musical instruments, and how their performances compare with musically trained and untrained listeners. To address these questions we created a novel dataset of 391 classical and acoustic guitar excerpts annotated along four emotions (aggressiveness, relaxation, happiness and sadness) with three emotion intensity levels (low, medium, high), according to the intended emotion of 30 professional guitarists acting as both composers and performers. A first experiment investigated listeners’ perception involving 8 professional guitarists and 8 non-musicians. Results showed that the emotions intended by a composer-performer are not always well recognized by listeners, and in general not with the same intensity. Listeners’ identification accuracy was proportional to the intensity with which an emotion was expressed. Emotions were better recognized by musicians than by listeners without musical background. Such differences between the two groups were found for different intensity levels of the intended emotions. A second experiment investigated machine listening performance based on a transfer learning method. To compare machine and human identification accuracies fairly, we derived a fifth, “ambivalent” category from the machine listening output categories (i.e., excerpts rated with more than one predominant emotion). Results showed that the machine perception of emotions matched or even exceeded musicians’ performance for all emotions except “relaxation”. The differences between the intended and human-perceived emotions, as well as those due to musical training, suggest that a device or application involving a music emotion recognition system should take into account the characteristics of the users (in particular their musical expertise) as well as their roles (e.g., composers, performers, listeners). For developers this translates into the use of datasets annotated by different categories of annotators, whose role and musical expertise will match the characteristics of the end users. Such results are particularly relevant to the creation of emotionally-aware smart musical instruments.

Cybersickness is a type of motion sickness that may occur during a virtual reality (VR) experience. Many studies have proposed solutions to mitigate cybersickness, while navigating a virtual environment with controllers or walking over a floor. However, reducing the levels of cybersickness while physically walking on an omnidirectional treadmill has been largely overlooked. In this article, we performed a within-subject study, where 34 novice participants underwent four visual conditions while walking in a virtual maze over an omnidirectional treadmill. In the control condition, the movement speed was reduced of the half compared to a standard navigation speed, a movement speed smoothing was added, and the user’s virtual body was represented. The other three conditions changed one of the visual parameters of the control condition: in the standard speed condition, the speed reduction was not performed; for the no smoothing condition, the smoothing was not performed; and for the no avatar condition, the user’s avatar was removed. Results showed that the standard speed condition was reported to induce a significant level of cybersickness compared to the control and no avatar conditions. Nevertheless, standard speed was also the condition most preferred to navigate a virtual environment. This suggests the need to find a tradeoff between the easiness to move quickly in a virtual environment and the cybersickness that can be induced. We provide a discussion of the obtained results and their implications for the design of VR experiences while users walk upon an omnidirectional treadmill.

Onset detectors are used to recognize the beginning of musical events in audio signals. Manual parameter tuning for onset detectors is a time consuming task, while existing automated approaches often maximize only a single performance metric. These automated approaches cannot be used to optimize detector algorithms for complex scenarios, such as real-time onset detection where an optimization process must consider both detection accuracy and latency. For this reason, a flexible optimization algorithm should account for more than one performance metric in a multi-objective manner. This paper presents a generalized procedure for automated optimization of parametric onset detectors. Our procedure employs a bio-inspired evolutionary computation algorithm to replace manual parameter tuning, followed by the computation of the Pareto frontier for multi-objective optimization. The proposed approach was evaluated on all the onset detection methods of the Aubio library, using a dataset of monophonic acoustic guitar recordings. Results show that the proposed solution is effective in reducing the human effort required in the optimization process: it replaced more than two days of manual parameter tuning with 13 hours and 34 minutes of automated computation. Moreover, the resulting performance was comparable to that obtained by manual optimization.

The emergence of new technologies is providing opportunities to develop novel solutions that facilitate the integration of visually-impaired people in different activities of our daily life, including collective music making. This paper presents a study conducted with visually-impaired music performers, which involved a participatory approach to the design of accessible technologies for musical communication in group playing. We report on three workshops that were conducted together with members of an established ensemble of solely visually-impaired musicians. The first workshop focused on the identification of the participants’ needs during the activity of playing in groups and how technology could satisfy such needs. The second and third workshops investigated, respectively, the activities of choir singing and instrument playing in ensemble, focusing on the key issue of synchronisation that was identified in the first workshop. The workshops involved prototypes of musical haptic wearables, which were co-designed and evaluated by the participants. Overall, results indicate that wireless tactile communication represents a promising avenue to cater effectively to the needs of visually-impaired performers.

As the Internet of Musical Things (IoMusT) emerges, audio-specific operating systems (OSs) are required on embedded hardware to ease development and portability of IoMusT applications. Despite the increasing importance of IoMusT applications, in this article, we show that there is no OS able to fulfill the diverse requirements of IoMusT systems. To address such a gap, we propose the Elk Audio OS as a novel and open source OS in this space. It is a Linux-based OS optimized for ultra-low-latency and high-performance audio and sensor processing on embedded hardware, as well as for handling wireless connectivity to local and remote networks. Elk Audio OS uses the Xenomai real-time kernel extension, which makes it suitable for the most demanding of low-latency audio tasks. We provide the first comprehensive overview of Elk Audio OS, describing its architecture and the key components of interest to potential developers and users. We explain operational aspects like the configuration of the architecture and the control mechanisms of the internal sound engine, as well as the tools that enable an easier and faster development of connected musical devices. Finally, we discuss the implications of Elk Audio OS, including the development of an open source community around it.

The Internet of Audio Things (IoAuT) is an emerging research field positioned at the intersection of the Internet of Things, sound and music computing, artificial intelligence, and human-computer interaction. The IoAuT refers to the networks of computing devices embedded in physical objects (Audio Things) dedicated to the production, reception, analysis, and understanding of audio in distributed environments. Audio Things, such as nodes of wireless acoustic sensor networks, are connected by an infrastructure that enables multidirectional communication, both locally and remotely. In this article, we first review the state of the art of this field, then we present a vision for the IoAuT and its motivations. In the proposed vision, the IoAuT enables the connection of digital and physical domains by means of appropriate information and communication technologies, fostering novel applications and services based on auditory information. The ecosystems associated with the IoAuT include interoperable devices and services that connect humans and machines to support human-human and human-machines interactions. We discuss the challenges and implications of this field, which lead to future research directions on the topics of privacy, security, design of Audio Things, and methods for the analysis and representation of audio-related information.

Large online music databases under Creative Commons licenses are rarely recorded by well-known artists, therefore conventional metadata-based search is insufficient in their adaptation to instrument players’ needs. The emerging class of smart musical instruments (SMIs) can address this challenge. Thanks to direct internet connectivity and embedded processing, SMIs can send requests to repositories and reproduce the response for improvisation, composition, or learning purposes. We present a smart guitar prototype that allows retrieving songs from large online music databases using criteria different from conventional music search, which were derived from interviewing 30 guitar players. We investigate three interaction methods coupled with four search criteria (tempo, chords, key and tuning) exploiting intelligent capabilities in the instrument: (i) keywords-based retrieval using an embedded touchscreen; (ii) cloud-computing where recorded content is transmitted to a server that extracts relevant audio features; (iii) edge-computing where the guitar detects audio features and sends the request directly. Overall, the evaluation of these methods with beginner, intermediate, and expert players showed a strong appreciation for the direct connectivity of the instrument with an online database and the approach to the search based on the actual musical content rather than conventional textual criteria, such as song title or artist name.

Communication among performers is a fundamental aspect in music performance. A large number of electronic music instruments based on tangible and screen-based interfaces require a focused visual attention from performers while they are controlled. In certain stage and artistic configurations, this may be an obstacle to face-to-face creative interactions between coperformers and their collaborators. To address these issues, we adopted a user-centered design methodology to develop a novel class of IoT devices that we term musical haptic wearables for performers. We conducted a co-design workshop with 10 electronic musicians using focus-group discussions and the bootlegging technique. This workshop identified numerous creative communication issues among performers in electronic music practice and resulted in mock-up prototypes. We then developed three chest-, foot-, and arm-worn haptic wearables respectively for coperformer, performer-conductor, and performer-sound-engineer interactions. The wearables were assessed with 25 participants using a mixed-methods approach. High accuracies (70%-100%) were obtained for musical actions expected after instructions wirelessly communicated via tactile signals. The results provide evidence that musical haptic wearables can be an effective medium of communication in the context of electronic music performances. More challenges were identified regarding size and placement of the devices on the body, interferences with concurrent vibrations generated by music signals, limitations on the range of creative controls, and a required training curve.

This paper presents initial steps towards the design of an embedded system for body-centric sonic performance. The proposed prototyping system allows performers to manipulate sounds through gestural interactions captured by textile wearable sensors. The e-textile sensor data control, in real-time, audio synthesis algorithms working with content from Audio Commons, a novel web-based ecosystem for re-purposing crowd-sourced audio. The system enables creative embodied music interactions by combining seamless physical e-textiles with web-based digital audio technologies.

This paper investigated how different emotional states of a walker can be rendered and recognized by means of footstep sounds synthesis algorithms. In a first experiment, participants were asked to render, according to imagined walking scenarios, five emotions (aggressive, happy, neutral, sad, and tender) by manipulating the parameters of synthetic footstep sounds simulating various combinations of surface materials and shoes types. Results allowed to identify, for the involved emotions and sound conditions, the mean values and ranges of variation of two parameters, sound level and temporal distance between consecutive steps. Results were in accordance with those reported in previous studies on real walking, suggesting that expression of emotions in walking is independent from the real or imagined motor activity. In a second experiment participants were asked to identify the emotions portrayed by walking sounds synthesized by setting the synthesis engine parameters to the mean values found in the first experiment. Results showed that the involved algorithms were successful in conveying the emotional information at a level comparable with previous studies. Both experiments involved musicians and non-musicians. In both experiments, a similar general trend was found between the two groups.

This paper investigates the production and identification of emotional states of a walker using plantar vibro-tactile simulations. In a first experiment, participants were asked to render, according to imagined walking scenarios, five emotions (aggressive, happy, neutral, sad, and tender) by manipulating the parameters of synthetic footstep vibrations simulating various combinations of surface materials and shoes. Results allowed to identify, for the involved emotions and vibration conditions, the mean values and ranges of variation of two parameters, vibration amplitude and temporal distance between consecutive steps. Results were in accordance with those reported in previous studies on real walking, suggesting that the plantar vibro-tactile expression of emotions in walking is independent of the real or imagined motor activity. In a second experiment, participants were asked to identify the emotions portrayed by walking vibrations synthesized by setting the synthesis engine parameters to the mean values found in the first experiment. Results showed that the involved algorithms were successful in conveying the emotional information at a level comparable with previous studies. Results of both experiments revealed strong similarities with those of an analogous study on footstep sounds suggesting that emotionally expressive walking styles are consistently produced and recognized at auditory and plantar vibro-tactile level.

This paper describes two experiments conducted to investigate the role of sonically simulated ground materials in modulating both production and recognition of walks performed with emotional intentions. The results of the first experiment showed that the involved auditory feedbacks affected the pattern of emotional walking in different ways, although such an influence manifested itself in more than one direction. The results of the second experiment showed the absence of an influence of the sound conditions on the recognition of the emotions from acoustic information alone. Similar results were found in both experiments for musically-trained and untrained participants. Our results suggest that tempo and sound level are two acoustical features important in both production and recognition of emotions in walking. In addition, the similarities of the presented results with those reported in the music performance domain, as well as the absence of an influence of musical expertise lend support to the “motor origin hypothesis of emotional expression in music” according to which a motor origin for the expression of emotions is common in all those domains of human activity that result in the generation of an acoustical signal.

This article investigates whether auditory feedback affects natural locomotion patterns. Individuals were provided with footstep sounds simulating different surface materials. The sounds were interactively generated using shoes with pressure sensors. Results showed that subjects' walking speed changed as a function of the type of simulated ground material. This effect may arise due to the presence of conflicting information between the auditory and foot-haptic modality, or because of an adjustment of locomotion to the physical properties evoked by the sounds simulating the ground materials. The results reported in this study suggest that auditory feedback may be more important in the regulation of walking in natural environments than has been acknowledged. Furthermore, auditory feedback could be used to develop novel approaches to the design of therapeutic and rehabilitation procedures for locomotion.

In this paper we describe an experiment whose goal is to investigate the role of physics-based auditory and haptic feedback provided at feet level to enhance realism in a virtual environment. To achieve this goal, we designed a multimodal virtual environment where subjects could walk on a platform overlooking a canyon. Subjects were asked to visit the environment wearing an head-mounted display and a custom made pair of sandals enhanced with sensors and actuators. A 12-channels surround sound system delivered a soundscape which was consistent with the visual environment. Passive haptics was provided by having a physical wooden platform present in the laboratory. Subjects reported of having a more realistic experience while auditory and haptic feedback are present. However, measured physiological data and post-experimental presence questionnaire do not show significant differences when audio-haptic feedback is provided.

We propose a system that affords real-time sound synthesis of footsteps on different materials. The system is based on microphones, which detect real footstep sounds from subjects, from which the ground reaction force (GRF) is estimated. Such GRF is used to control a sound synthesis engine based on physical models. Two experiments were conducted. In the first experiment, the ability of subjects to recognize the surface they were exposed to was assessed. In the second experiment, the sound synthesis engine was enhanced with environmental sounds. Results show that, in some conditions, adding a soundscape significantly improves the recognition of the simulated environment.

We describe a system which simulates in realtime the auditory and haptic sensations of walking on different surfaces. The system is based on a pair of sandals enhanced with pressure sensors and actuators. The pressure sensors detect the interaction force during walking, and control several physically based synthesis algorithms, which drive both the auditory and haptic feedback. The different hardware and software components of the system are described, together with possible uses and possibilities for improvements in future design iterations.

A system to synthesize in real-time the sound of footsteps on different materials is presented. The system is based on microphones which allow the user to interact with his own footwear. This solution distinguishes our system from previous efforts that require specific shoes enhanced with sensors. The microphones detect real footsteps sounds from users, from which the ground reaction force (GRF) is estimated. Such GRF is used to control a sound synthesis engine based on physical models. Evaluations of the system in terms of sound validity and fidelity of interaction are described.

In this paper, we present an experiment whose goal is to investigate the role of sound and vision in the recognition of different surface profiles in a walking scenario. Fifteen subjects participated to two within-subjects experiments where they were asked to interact with a desktop system simulating bumps, holes and flat surfaces by means of audio, visual and audio-visual cues. Results of the first experiment show that participants are able to successfully identify the surface profiles provided through the proposed audio-visual techniques. Results of a second experiment in which conflictual audiovisual stimuli were presented, reveal that for some of the proposed visual effects the visual feedback is dominant on the auditory one, while for the others the role of dominance is inverted.