Mixed reality (MR) technologies hold promise for distributed musical collaboration, yet the needs of professional musicians remain underexplored. We conducted a set of co-design workshops with 24 professional musicians, organized into six four-piece bands, to envision future MR systems for networked music performance. Using a three-phase process (hands-on MR experience, individual brainwriting, and collaborative consensus-building), we identified core requirements that challenge current MR paradigms. Professional musicians operate from an “Audio-First” framework where sonic experience constitutes the primary reality, contradicting visual-first MR approaches. Five unanimous requirements emerged: audio-first workflow, support for non-verbal communication, personalized audio control, spatial layout customization, and plug-and-play simplicity. Other high-consensus themes included real-time voice communication and realistic avatar representation. Taken together, our findings reveal that professional creative collaboration demands fundamentally different design priorities than traditional office-centered collaborative MR applications, requiring systems that support established workflows and nonverbal communication channels while prioritizing workflow preservation over visual immersion.

Music is increasingly performed and experienced in Social Virtual Reality (Social VR), from VRChat raves to high-production concerts on bespoke platforms. Yet Human-Computer-Interaction (HCI) research still focuses mainly on building new VR systems rather than examining the communities that already create and sustain these practices. We present a cultural mapping of the Social VR music scene based on 84 survey responses, 27 interviews, and 17 event observations with diverse stakeholders, including audience members, musicians, developers, platform owners, and event organisers. We found that the scene operates as a fragmented cross-platform ecosystem sustained by user-generated infrastructure and continuous community labour. The bottom-up organisation produces role fluidity with individuals dynamically shifting between roles as performers, world builders, organisers, and audience members. However, the openness that enables this creativity also creates tensions between expectations of free access and the financial and emotional labour required to keep events running. Taken together, our findings reveal the vibrant cultural practices that continue to flourish in Social VR, even as corporate narratives declare the “metaverse” dead.

Networked Music Performance (NMP) systems have traditionally focused on optimizing audio transmission for remote collaboration, relying on basic 2D video feeds that fail to provide the spatial awareness and visual cues crucial for coordinated musical performance. While Mixed Reality (MR) technology offers promising solutions for enhanced musical collaboration, its integration into NMP systems remains understudied, particularly regarding the balance between immersion and usability. This study investigates this gap by comparing traditional 2D video conferencing setups with MR environments using point cloud representations and spatial audio. Through an experimental study with professional electric guitar players (N=16) and a simulation-based approach with pre-recorded setup, we examined how both 2D video and MR setups affect musicians’ performance, movement, and sense of presence using quantitative movement analysis and qualitative feedback. Head movement analysis revealed significantly reduced motion in MR compared to 2D conditions (p < 0.001), while detrended fluctuation analysis showed more structured movement patterns in MR (p = 0.0028). Post-task evaluations demonstrated that while MR enhanced immersion and presence (p < 0.001), participants reported significantly better performance coherence and reduced technical interference with 2D videos (p < 0.001). Spatial audio features showed minimal impact on user experience, with visual elements dominating attention. The findings highlight a fundamental tension between immersion and physical expression in MR music performance, suggesting that future systems should prioritize ergonomic considerations alongside technological capabilities. This research contributes to understanding the potential and the limitations of MR technologies in musical collaboration within the emerging Musical Metaverse.

Our work explores how audiences perceive and engage with remote music events. The shift to digital platforms has transformed the live music experience, leading to the rise of remote music performances. Following the COVID-19 pandemic, platforms such as Twitch have been repurposed and surged in popularity as meaningful outlets for live music dissemination. Concurrently, novel platforms specifically crafted for music events have begun to emerge. Understanding the preferences and challenges of remote music audiences is crucial because it guides the development of platforms that effectively answer varied user needs, which could enhance overall engagement and shape the future of music consumption. For this, we employed a qualitative multi-method approach, including a survey, interviews, and event observations, to start mapping the design space of remote music performances and capture a wide spectrum of audience perspectives. Our findings reveal audience engagement patterns in remote music events across different digital platforms, highlighting unmet needs, desires as well as opportunities for future HCI work in this area.

Previous research has shown that the vibrotactile enhancement of music signals can be effective in creating intense musical experiences. Virtual Reality is emerging as a new medium with which experiencing music and inducing emotions, but the study of the effects of tactile enrichments of music in virtual environments (VEs) has been largely overlooked thus far. It is unknown whether the same results found for real environments hold for VEs too, and what is the role played by vibrations in altering the emotion, perception and behavior of listeners in such spaces. To bridge these gaps, we created 12 affective musical VEs accompanying as many musical pieces and assessed the impact of adding synchronous vibrations. Results showed that the VEs involving the vibrations received significantly higher ratings of valence, arousal and presence compared to their audio-visual counterparts. The majority of participants preferred the condition with the vibrations. Cybersickness was not reduced in presence of vibrations, but it was found to be significantly higher for females than males. Although most participants reported that they were induced to move to a greater extent in presence of vibrations, the recorded motion of their head and hands did not increase when vibrations were provided.

While numerous studies have examined 3D interaction techniques for Virtual Reality (VR) across various tasks and scenarios, limited research has focused on music-related applications. However, the most common input techniques in consumer VR systems have been developed outside of the musical domain. Therefore they have not been tested in tasks where synchronization with auditory stimuli and timing plays a crucial role. There is a lack of empirical knowledge about performance and user experience. This paper presents a comparison of five selection input techniques for VR employing the tapping paradigm commonly utilized in the research on sensorimotor synchronization. We assess asynchrony and timing variance as well as user experience, encompassing factors such as ease of use, workload, and cybersickness of such techniques. The study involved 30 participants, both with and without musical expertise, and encompassed the examination of all techniques using one and two hands. Our analysis yielded several key findings: (1) different input techniques yielded distinct outcomes regarding timing asynchrony and variance; (2) the choice of interaction metaphor significantly influenced the user experience; (3) tracking stability emerged as a critical factor. Building upon these insights, we identified essential considerations for selecting the most suitable technique for music creation in VR and proposed design guidelines and future research directions in this domain.

Deformable interfaces are emerging in HCI and prototypes show potential for non-rigid interactions. Previous reviews looked at deformation as a material property of shape-changing interfaces and concentrated on output. As such,deformable input was under-discussed. We distinguish deformable from shape-changing interfaces to concentrate on input. We survey 131 papers on deformable interfaces and review their key design elements (e.g., shape, material) based on how they support input. Our survey shows that deformable input was often used to augment or replace rigid input, particularly on elastic and flexible displays. However, when shapes and materials guide interactions, deformable input was used to explore new HCI paradigms, where gestures are potentially endless, and input become analogy to sculpting, metaphor to non-verbal communication, and expressive controls are enhanced. Our review provides designers and practitioners with a baseline for designing deformable interfaces and input methodically. We conclude by highlighting under-explored areas and identify research goals to tackle in future work with deformable interfaces.

This paper studies the characteristics of the human perception of a haptic shape-changing interface, capable of altering its size and rigidity simultaneously for presenting characteristics of virtual objects physically. The haptic interface is composed of an array of computer-controlled balloons, with two mechanisms, one for changing size and one for changing rigidity. We manufactured two balloons and conducted psychophysical experiments with twenty subjects to measure perceived sensory thresholds and haptic perception of the change of size and rigidity. The results show that subjects can correctly discriminate different conditions with an acceptable level of accuracy. Our results also suggest that the proposed system can present an ample range of rigidities and variations of the size in a way that is compatible with the human haptic perception of physical materials. Currently, shape-changing interfaces do not hold a defined position in the current VR / AR research. Our results provide basic knowledge for developing novel types of haptic interfaces that can enhance the haptic perception of virtual objects, allowing rich embodied interactions, and synchronize the virtual and the physical world through computationally-controlled materiality.

This paper summarizes the author's interest in haptic interfaces for Virtual Reality Musical Instruments. The current research focuses on finding interfaces that can improve physical interaction and presence with such instruments. Musical expression is a topic rarely addressed in the field of Virtual Reality. During the years, the author has explored different systems and concepts while finding the thesis topic for the Ph.D. research. They include the development and evaluation of deformable input surfaces and Shape-Changing interfaces. The results from these implementations led us to investigate Encounter-type haptics, a method that has never received a proper consideration in the design of virtual musical instruments. This represents the current stage of our research. However, the exact direction towards the Ph.D. thesis topic is still in search. Through this paper, we will describe the background and motivations behind this research together with the research hypothesis developed until now.

In this paper we introduce Vital+Morph, a prototype of a shape-changing interface intended to be used as a media for remote biometric monitoring. Signals measured by a Vital Signs Monitoring Station are physicalized into a series of reactive tangible objects with a life-like behavior. Through this interface a person can feel the internal state of an hospitalized patient, as a new form of communication and awareness over distance. We propose a perceptual equivalent of an imaginative material capable of displaying digital information through shape-changing, and start to explore the social impact in complex contexts such as health care. Ultimately, Vital+Morph proposes an unusual viewpoint of the relations between bodies, clinical data and future materials.