Recent advances in generative AI have enabled new forms of audio synthesis for musical creation, yet model training remains underexplored as an artistic and collaborative practice. While prior work has focused on composition tools and live performance interfaces, how artists train their models have received less attention. This paper presents a qualitative study of collaborative training practices around audio synthesis models. We conducted semi-structured interviews with 13 artists across 8 musical projects relying on custom-trained models for musical performance or installations. Using thematic analysis, our findings show that artists approach training as a situated, craft-like practice rather than a purely technical task. Collectives develop shared languages and embodied practices to communicate and guide training, using iterative feedback and dataset sculpting to shape model behavior, with musical engagement as a primary method for steering models. We discuss implications to better support model training as key site of interaction and collaboration.

Hip-hop dance battles are events where dancers improvise to an unfamiliar DJ’s mix. The dancers’ technicality in responding to the music, along with the collective dimensions of these encounters, represents a largely unexplored area of investigation in HCI. Using an autobiographical design approach grounded in the hip-hop practice of the first author, we developed five interaction scenarios on a hip-hop–specific movement vocabulary. Our interactive sonification enables both individual and collective improvisation through synchronized motion sensing and interactive sound loops. This approach allowed us to design three interconnected workshops to investigate how dancers use sonification in improvisation. Our findings show that sonification was perceived either as a validation of movement or as a medium for open-ended exploration. We contribute with: (1) the formalization of a movement vocabulary and the design of interactive scenarios for hip-hop movement sonification, (2) insights into how dancers experience sonification through exploration and improvisation, and (3) guidelines for the design of movement sonification in hip-hop contexts.

We present a retrospective autoethnography grounded in data-driven design. The first author collected her movement data and subjective experience of learning the dance repertoire of modern dance pioneer Isadora Duncan, which together were encoded into the design of a set of plaster artefacts physicalising her embodied dance learning progression. The artefacts reflect the first author’s bodily transformation, mirroring her transition from discomfort to ease, and changes in her expressive capabilities. Our method offers an alternative to documentation of embodied learning through design. Throughout our design process we leverage on the movement data, the field notes and the first author’s memory of her journey, all of which constitute entangled and complementary input into her experience of dance learning. We show that the data physicalisations provided a gateway into the intangible experience and allowed for a deep and reflexive understanding of our dataset.

Wearables integrating movement sonification can support body-perception changes and related physical activity; yet, we lack design principles for such sonifications. Through two mixed-methods studies, we investigate sound pitch and movement direction interaction effects on self-perception during squat exercises. We measured effects on body perception, affective quality of the experience, and actual and perceived movement, and compared them with two control conditions: no-sound and vibrotactile feedback. Results show that regardless of movement direction, ascending pitch enhances several body feelings and overall experience quality, while descending pitch increases movement acceleration. These effects were moderated by exercise physical demand. Sound and vibrotactile feedback enhanced flexibility and strength feelings, respectively, and contributed to exercise completion in different ways. Sound was perceived as an internal-to-body force while vibrotactile feedback was perceived as an external-to-body force. Feedback effects were stronger in people with lower fitness levels. We discuss results in terms of malleability of body perceptions and highlight opportunities to support demanding physical activity through wearable devices.

With the increasing interest in movement sonification and expressive gesture-based interaction, it is important to understand which factors contribute to movement learning and how. We explore the effects of movement sonification and users’ musical background on motor variability in complex gesture learning. We contribute an empirical study in which musicians and non-musicians learn two gesture sequences over three days, with and without movement sonification. Results show the interlaced interaction effects of these factors and how they unfold in the three-day learning process. For gesture 1, which is fast and dynamic with a direct “action-sound” sonification, movement sonification induces higher variability for both musicians and non-musicians on day 1. While musicians reduce this variability to a similar level as no auditory feedback condition on day 2 and day 3, non-musicians remain to have significantly higher variability. Across three days, musicians also have significantly lower variability than non-musicians. For gesture 2, which is slow and smooth with an “action-music” metaphor, there are virtually no effects. Based on these findings, we recommend future studies to take into account participants’ musical background, consider longitudinal study to examine these effects on complex gestures, and use awareness when interpreting the results given a specific design of gesture and sound.

In this paper, we outline a diffractive practice of machine learning (ML) in the frame of material-centered interaction design. To this aim, we review related work in ML, HCI, design, new interfaces for musical expression, and computational art, and introduce two practice-based studies of music performance and robotic art based on interactive machine learning tools, with the hope of revealing the computational materiality of ML, and the potential of embodiment to craft prototypes of ML that reconfigure conceptual or technical approaches to ML. We derive five interference conditions for such art-based ML prototypes—situational whole, small data, shallow model, learnable algorithm, and somaesthetic behaviour—and describe their widening of design and engineering practices of ML prototyping. Finally, we sketch how a process of intra-active machine learning could complement that of interactive machine learning to take materiality as an entry point for ML design within HCI.

Aerial hoops are circular, hanging devices for both acrobatic exercise and artistic performance that let us explore the role of interactive sonification in physical activity. We present SonicHoop, an augmented aerial hoop that generates auditory feedback via capacitive touch sensing, thus becoming a digital musical instrument that performers can play with their bodies. We compare three sonification strategies through a structured observation study with two professional aerial hoop performers. Results show that SonicHoop fundamentally changes their perception and choreographic processes: instead of translating music into movement, they search for bodily expressions that compose music. Different sound designs affect their movement differently, and auditory feedback, regardless of type of sound, improves movement quality. We discuss opportunities for using SonicHoop as an aerial hoop training tool, as a digital musical instrument, and as a creative object; as well as using interactive sonification in other acrobatic practices to explore full-body vertical interaction.

Negative body perceptions are a major predictor of physical inactivity, a serious health concern. Sensory feedback can be used to alter such body perceptions; movement sonification, in particular, has been suggested to affect body perception and levels of physical activity (PA) in inactive people. We investigated how metaphorical sounds impact body perception and PA. We report two qualitative studies centered on performing different strengthening/flexibility exercises using SoniBand, a wearable that augments movement through different sounds. The first study involved physically active participants and served to obtain a nuanced understanding of the sonifications’ impact. The second, in the home of physically inactive participants, served to identify which effects could support PA adherence. Our findings show that movement sonification based on metaphors led to changes in body perception (e.g., feeling strong) and PA (e.g., repetitions) in both populations, but effects could differ according to the existing PA-level. We discuss principles for metaphor-based sonification design to foster PA.

Machine learning systems became pervasive in modern interactive technology but provide users with little, if any, agency with respect to how their models are trained from data. In this paper, we are interested in the way novices handle learning algorithms, what they understand from their behavior and what strategy they may use to "make it work". We developed a web-based sketch recognition algorithm based on Deep Neural Network (DNN), called Marcelle-Sketch, that end-users can train incrementally. We present an experimental study that investigate people's strategies and (mis)understandings in a realistic algorithm-teaching task. Our study involved 12 participants who performed individual teaching sessions using a think-aloud protocol. Our results show that participants adopted heterogeneous strategies in which variability affected the model performances. We highlighted the importance of sketch sequencing, particularly at the early stage of the teaching task. We also found that users' understanding is facilitated by simple operations on drawings, while confusions are caused by certain inherent properties of DNN. From these findings, we propose implications for design of IML systems dedicated to novices and discuss the socio-cultural aspect of this research.

Software tools for generating digital sound often present users with high-dimensional, parametric interfaces, that may not facilitate exploration of diverse sound designs. In this article, we propose to investigate artificial agents using deep reinforcement learning to explore parameter spaces in partnership with users for sound design. We describe a series of user-centred studies to probe the creative benefits of these agents and adapting their design to exploration. Preliminary studies observing users’ exploration strategies with parametric interfaces and testing different agent exploration behaviours led to the design of a fully-functioning prototype, called Co-Explorer, that we evaluated in a workshop with professional sound designers. We found that the Co-Explorer enables a novel creative workflow centred on human–machine partnership, which has been positively received by practitioners. We also highlight varied user exploration behaviours throughout partnering with our system. Finally, we frame design guidelines for enabling such co-exploration workflow in creative digital applications.

Physical inactivity is an increasing problem. It has been linked to psychological and emotional barriers related to the perception of one's body, such as physical capabilities. It remains a challenge to design technologies to increase physical activity in inactive people. We propose the use of a sound interactive system where inputs from movement sensors integrated in shoes are transformed into sounds that evoke body sensations at a metaphorical level. Our user study investigates the effects of various gesture-sound mappings on the perception of one's body and its movement qualities (e.g. being flexible or agile), the related emotional state and movement patterns, when people performed two exercises, walking and thigh stretch. The results confirm the effect of the “metaphor” conditions vs. the control conditions in feelings of body weight; feeling less tired and more in control; or being more comfortable, motivated, and happier. These changes linked to changes in affective state and body movement. We discuss the results in terms of how acting upon body perception and affective states through sensory feedback may in turn enhance physical activity, and the opportunities opened by our findings for the design of wearable technologies and interventions in inactive populations.

This PhD project is multidisciplinary, at the intersection of Early Childhood Education, Design, and Human-Computer Interaction. The PhD goal aim at developing new approaches for early childhood education focusing on the role of body in the learning process. We leverage digital tangible and interactive technologies to develop specific scenarios that are evaluated in preschools. Our work proposes novel paradigms and uses of digital technologies currently not found in typical preschool environments.

In this paper, we present a complete framework, both technical and conceptual, aimed at developing and analysing Networked Music Systems. After a short description of our technical framework called soundworks, a JavaScript library especially designed for collective music interaction using web browser of mobile phones, we introduce a new conceptual framework, we named interaction topologies, that aims at providing a generic tool for the description of interaction in such systems. Our proposition differs from the theoretical approaches generally proposed in the literature by decoupling the description of interaction topologies from the low level technical implementation of the network. We then report on a set of scenarios and prototypes, illustrating and assessing our framework, which were successfully deployed in several public installations and performances. We particularly show that our concept of interaction topologies succeeds at describing and analysing global aspects of interaction from multiple point of views (e.g., social, human-computer) by allowing for composing simple abstract figures. We finally introduce a discussion on agencies and perception of users engaged in such systems that could later complete our framework to conceive and analyse Networked Music Systems.

Technologies for sensing movement are expanding toward everyday use in virtual reality, gaming, and artistic practices. In this context, there is a need for methodologies to help designers and users create meaningful movement experiences. This article discusses a user-centered approach for the design of interactive auditory feedback using interactive machine learning. We discuss Mapping through Interaction, a method for crafting sonic interactions from corporeal demonstrations of embodied associations between motion and sound. It uses an interactive machine learning approach to build the mapping from user demonstrations, emphasizing an iterative design process that integrates acted and interactive experiences of the relationships between movement and sound. We examine Gaussian Mixture Regression and Hidden Markov Regression for continuous movement recognition and real-time sound parameter generation. We illustrate and evaluate this approach through an application in which novice users can create interactive sound feedback based on coproduced gestures and vocalizations. Results indicate that Gaussian Mixture Regression and Hidden Markov Regression can efficiently learn complex motion-sound mappings from few examples.

Human movement has historically been approached as a functional component of interaction within human computer interaction. Yet movement is not only functional, it is also highly expressive. In our research, we explore how movement expertise as articulated in Laban Movement Analysis (LMA) can contribute to the design of computational models of movement's expressive qualities as defined in the framework of Laban Efforts. We include experts in LMA in our design process, in order to select a set of suitable multimodal sensors as well as to compute features that closely correlate to the definitions of Efforts in LMA. Evaluation of our model shows that multimodal data combining positional, dynamic and physiological information allows for a better characterization of Laban Efforts. We conclude with implications for design that illustrate how our methodology and our approach to multimodal capture and recognition of Effort qualities can be integrated to design interactive applications.

We present the Collective Sound Checks, an exploration of user scenarios based on mobile web applications featuring motion-controlled sound that enable groups of people to engage in spontaneous collaborative sound and music performances. These new forms of musical expression strongly shift the focus of design from human-computer interactions towards the emergence of computer mediated interactions between players based on sonic and social affordances of ubiquitous technologies. At this early stage, our work focuses on experimenting with different user scenarios while observing the relationships between different interactions and affordances.

In this study, we investigated the ability of blindfolded adults to discriminate between concave and convex auditory virtual surfaces. We used a Leap MotionTMdevice to measure the movements of the hand and fingers. Participants were asked to explore the space above the device with the palm of one hand and an auditory feedback was produced only when the palm was moving into the boundaries of the surface. In order to demonstrate that curvature direction was correctly perceived by our participants, we estimated their discrimination thresholds with a psychophysical staircase procedure. Two groups of participants were fed with two different sonification of the surface. Results showed that most of the participants were able to learn the task. The best results were obtained with an auditory feedback related to the component of the hand velocity tangential to the virtual surface. This work proposes a contribution to the introduction in virtual reality of auditory virtual objects.

The notion of “movement qualities” is central in contemporary dance; it describes the manner in which a movement is executed. Movement qualities convey information revealing movement expressiveness; their use has strong potential for movement-based interaction with applications in arts, entertainment, education, or rehabilitation. The purpose of our research is to design and evaluate interactive reflexive visuals for movement qualities. The theoretical basis for this research is drawn from a collaboration with the members of the international dance company Emio Greco|PC to study their formalization of movement qualities. We designed a pedagogical interactive installation called Double Skin/Double Mind (DS/DM) for the analysis and visualization of movement qualities through physical model-based interactive renderings. In this article, we first evaluate dancers’ perception of the visuals as metaphors for movement qualities. This evaluation shows that, depending on the physical model parameterization, the visuals are capable of generating dynamic behaviors that the dancers associate with DS/DM movement qualities. Moreover, we evaluate dance students’ and professionals’ experience of the interactive visuals in the context of a dance pedagogical workshop and a professional dance training. The results of these evaluations show that the dancers consider the interactive visuals to be a reflexive system that encourages them to perform, improves their experience, and contributes to a better understanding of movement qualities. Our findings support research on interactive systems for real-time analysis and visualization of movement qualities, which open new perspectives in movement-based interaction design.

This article presents a gesture recognition/adaptation system for human--computer interaction applications that goes beyond activity classification and that, as a complement to gesture labeling, characterizes the movement execution. We describe a template-based recognition method that simultaneously aligns the input gesture to the templates using a Sequential Monte Carlo inference technique. Contrary to standard template-based methods based on dynamic programming, such as Dynamic Time Warping, the algorithm has an adaptation process that tracks gesture variation in real time. The method continuously updates, during execution of the gesture, the estimated parameters and recognition results, which offers key advantages for continuous human--machine interaction. The technique is evaluated in several different ways: Recognition and early recognition are evaluated on 2D onscreen pen gestures; adaptation is assessed on synthetic data; and both early recognition and adaptation are evaluated in a user study involving 3D free-space gestures. The method is robust to noise, and successfully adapts to parameter variation. Moreover, it performs recognition as well as or better than nonadapting offline template-based methods.

We investigate the use of interactive sound feedback for dance pedagogy based on the practice of vocalizing while moving. Our goal is to allow dancers to access a greater range of expressive movement qualities through vocalization. We propose a methodology for the sonification of Effort Factors, as defined in Laban Movement Analysis, based on vocalizations performed by movement experts. Based on the experiential outcomes of an exploratory workshop, we propose a set of design guidelines that can be applied to interactive sonification systems for learning to perform Laban Effort Factors in a dance pedagogy context.

We investigated gesture description of sound stimuli performed during a listening task. Our hypothesis is that the strategies in gestural responses depend on the level of identification of the sound source and specifically on the identification of the action causing the sound. To validate our hypothesis, we conducted two experiments. In the first experiment, we built two corpora of sounds. The first corpus contains sounds with identifiable causal actions. The second contains sounds for which no causal actions could be identified. These corpora properties were validated through a listening test. In the second experiment, participants performed arm and hand gestures synchronously while listening to sounds taken from these corpora. Afterward, we conducted interviews asking participants to verbalize their experience while watching their own video recordings. They were questioned on their perception of the listened sounds and on their gestural strategies. We showed that for the sounds where causal action can be identified, participants mainly mimic the action that has produced the sound. In the other case, when no action can be associated with the sound, participants trace contours related to sound acoustic features. We also found that the interparticipants’ gesture variability is higher for causal sounds compared to noncausal sounds. Variability demonstrates that, in the first case, participants have several ways of producing the same action, whereas in the second case, the sound features tend to make the gesture responses consistent.

This article presents Gesture Interaction DEsigner (GIDE), an innovative application for gesture recognition. Instead of recognizing gestures only after they have been entirely completed, as happens in classic gesture recognition systems, GIDE exploits the full potential of gestural interaction by tracking gestures continuously and synchronously, allowing users to both control the target application moment to moment and also receive immediate and synchronous feedback about system recognition states. By this means, they quickly learn how to interact with the system in order to develop better performances. Furthermore, rather than learning the predefined gestures of others, GIDE allows users to design their own gestures, making interaction more natural and also allowing the applications to be tailored by users' specific needs. We describe our system that demonstrates these new qualities—that combine to provide fluid gesture interaction design—through evaluations with a range of performers and artists.

We address the issue of mapping between gesture and sound for gesture-based control of physical modeling sound synthesis. We propose an approach called mapping by demonstration, allowing users to design the mapping by performing gestures while listening to sound examples. The system is based on a multimodal model able to learn the relationships between gestures and sounds.

In this paper, we propose a multimodal approach to create the mapping between gesture and sound in interactive music systems. Specifically, we propose to use a multimodal HMM to conjointly model the gesture and sound parameters. Our approach is compatible with a learning method that allows users to define the gesture--sound relationships interactively. We describe an implementation of this method for the control of physical modeling sound synthesis. Our model is promising to capture expressive gesture variations while guaranteeing a consistent relationship between gesture and sound.

In this paper, we explore the use of movement qualities as interaction modality. The notion of movement qualities is widely used in dance practice and can be understood as how the movement is performed, independently of its specific trajectory in space. We implemented our approach in the context of an artistic installation called A light touch. This installation invites the participant to interact with a moving light spot reacting to the hand movement qualities. We conducted a user experiment that showed that such an interaction based on movement qualities tends to enhance the user experience favouring explorative and expressive usage.

We present an ensemble of tangible objects and software modules designed for musical interaction and performance. The tangible interfaces form an ensemble of connected objects communicating wirelessly. A central concept is to let users determine the final musical function of the objects, favoring customization, assembling, repurposing. This might imply assembling the wireless interfaces with existing everyday objects or musical instruments. Moreover, gesture analysis and recognition modules allow the users to define their own action/motion for the control of sound parameters. Various sound engines and interaction scenarios were built and experimented. Some examples that were developed in a music pedagogy context are described.