We present the design and evaluation of several digital musical instruments (DMIs) inspired by practices of guitar playing using open-source technology. The DMIs were designed using a participatory design process involving four children who expressed an interest in playing the guitar, and two music teachers at a German special educational needs (SEN) school. The instruments were integrated into various educational activities and evaluated through observations and qualitative interviews with the music teachers. One of the primary contributions of our study, in addition to the instruments themselves, is an understanding of the intricate and hitherto unexplored domain of designing in SEN schools. Our findings indicate that the DMIs facilitate motivating access to music making and contribute to the development of musical and non-musical competencies. Furthermore, we discovered that adequate educational activities and emotional support are equally crucial factors for the successful implementation of DMIs in the SEN school context. • Participatory design with children and teachers shaped guitar-inspired DMIs. • We observed the development of both musical and non-musical competencies. • Educational activities and support are crucial for DMI use in SEN schools. • Differing DMI perceptions highlight need for multiple stakeholder involvement.

In this paper, we present a complete framework, both technical and conceptual, aimed at developing and analysing Networked Music Systems. After a short description of our technical framework called soundworks, a JavaScript library especially designed for collective music interaction using web browser of mobile phones, we introduce a new conceptual framework, we named interaction topologies, that aims at providing a generic tool for the description of interaction in such systems. Our proposition differs from the theoretical approaches generally proposed in the literature by decoupling the description of interaction topologies from the low level technical implementation of the network. We then report on a set of scenarios and prototypes, illustrating and assessing our framework, which were successfully deployed in several public installations and performances. We particularly show that our concept of interaction topologies succeeds at describing and analysing global aspects of interaction from multiple point of views (e.g., social, human-computer) by allowing for composing simple abstract figures. We finally introduce a discussion on agencies and perception of users engaged in such systems that could later complete our framework to conceive and analyse Networked Music Systems.

As an interdisciplinary group of developers, researchers, and designers, we are currently exploring collaborative co-located applications that invite users to engage with interactive audiovisual media using the mobile devices they carry with them. This work simultaneously investigates basic concepts of collaborative interactions and user experience, together with technical concepts and features. Allowing for rapid prototyping, easy deployment, and spontaneous participation, these explorations are entirely based on established and upcoming web standards such as WebSockets, WebGL, and Web Audio API. After an overview of basic concepts common to these applications, this paper describes Collective Loops, a prototypical interactive audiovisual installation that instantiates these concepts and allows for discussing important details in the design and implementation of applications of its kind. The final part of the article summarizes the incremental design and validation of two versions of the Collective Loops installation presented at public events.

We present the Collective Sound Checks, an exploration of user scenarios based on mobile web applications featuring motion-controlled sound that enable groups of people to engage in spontaneous collaborative sound and music performances. These new forms of musical expression strongly shift the focus of design from human-computer interactions towards the emergence of computer mediated interactions between players based on sonic and social affordances of ubiquitous technologies. At this early stage, our work focuses on experimenting with different user scenarios while observing the relationships between different interactions and affordances.

We investigated gesture description of sound stimuli performed during a listening task. Our hypothesis is that the strategies in gestural responses depend on the level of identification of the sound source and specifically on the identification of the action causing the sound. To validate our hypothesis, we conducted two experiments. In the first experiment, we built two corpora of sounds. The first corpus contains sounds with identifiable causal actions. The second contains sounds for which no causal actions could be identified. These corpora properties were validated through a listening test. In the second experiment, participants performed arm and hand gestures synchronously while listening to sounds taken from these corpora. Afterward, we conducted interviews asking participants to verbalize their experience while watching their own video recordings. They were questioned on their perception of the listened sounds and on their gestural strategies. We showed that for the sounds where causal action can be identified, participants mainly mimic the action that has produced the sound. In the other case, when no action can be associated with the sound, participants trace contours related to sound acoustic features. We also found that the interparticipants’ gesture variability is higher for causal sounds compared to noncausal sounds. Variability demonstrates that, in the first case, participants have several ways of producing the same action, whereas in the second case, the sound features tend to make the gesture responses consistent.

We address the issue of mapping between gesture and sound for gesture-based control of physical modeling sound synthesis. We propose an approach called mapping by demonstration, allowing users to design the mapping by performing gestures while listening to sound examples. The system is based on a multimodal model able to learn the relationships between gestures and sounds.

In this paper, we propose a multimodal approach to create the mapping between gesture and sound in interactive music systems. Specifically, we propose to use a multimodal HMM to conjointly model the gesture and sound parameters. Our approach is compatible with a learning method that allows users to define the gesture--sound relationships interactively. We describe an implementation of this method for the control of physical modeling sound synthesis. Our model is promising to capture expressive gesture variations while guaranteeing a consistent relationship between gesture and sound.

We present an ensemble of tangible objects and software modules designed for musical interaction and performance. The tangible interfaces form an ensemble of connected objects communicating wirelessly. A central concept is to let users determine the final musical function of the objects, favoring customization, assembling, repurposing. This might imply assembling the wireless interfaces with existing everyday objects or musical instruments. Moreover, gesture analysis and recognition modules allow the users to define their own action/motion for the control of sound parameters. Various sound engines and interaction scenarios were built and experimented. Some examples that were developed in a music pedagogy context are described.

This paper describes our attempt to make the hidden Markov model (HMM) score following system, developed at Ircam, sensible to past experiences in order to obtain better audio to score real-time alignment for musical applications. A new observation modeling based on Gaussian mixture models is developed which is trainable using a learning algorithm we would call automatic discriminative training. The novelty of this system lies in the fact that this method, unlike classical methods for HMM training, is not concerned with modeling the music signal but with correctly choosing the sequence of music events that was performed. Besides obtaining better alignment, the new system's parameters are controllable in a physical manner and the training algorithm learns different styles of music performance as discussed.

This article presents ESCHER, a sound synthesis environment based on Ircam's real-time audio environment jMax. ESCHER is a modular system providing synthesis-independent prototyping of gesturally-controlled instruments by means of parameter interpolation. The system divides into two components: gestural controller and synthesis engine. Mapping between components takes place on two independent levels, coupled by an intermediate abstract parameter layer. This separation allows a flexible choice of controllers and/or sound synthesis methods.