Direct interaction with digital synthesisers using audio signals can offer opportunities for intimate and nuanced interaction in digital musical instrument designs. Unlike acoustic instruments, these hybrid instruments tend to follow a unidirectional interaction structure: tactile gestures generate audio signals that are fed into a synthesiser, but there is no vibrotactile feedback from the instrument back to the musician. This paper presents the HaptiCoupler system that enables bidirectional tactile interaction with digital musical instruments using a single voice coil transducer. A study is undertaken with experienced digital musical instrument designers to explore the design implications of introducing closely coupled, collocated haptic feedback in musical systems. The potential creative implications for designers are discussed.

This paper investigates sound and music interactions arising from the use of electromyography (EMG) to instrumentalise signals from muscle exertion of the human body. We situate EMG within a family of embodied interaction modalities, where it occupies a middle ground, considered as a “signal from the inside” compared with external observations of the body (e.g., motion capture), but also seen as more volitional than neurological states recorded by brain electroencephalogram (EEG). To understand the messiness of gestural interaction afforded by EMG, we revisit the phenomenological turn in HCI, reading Paul Dourish’s work on the transparency of “ready-to-hand” technologies against the grain of recent posthumanist theories, which offer a performative interpretation of musical entanglements between bodies, signals, and representations. We take music performance as a use case, reporting on the opportunities and constraints posed by EMG in workshop-based studies of vocal, instrumental, and electronic practices. We observe that across our diverse range of musical subjects, they consistently challenged notions of EMG as a transparent tool that directly registered the state of the body, reporting instead that it took on “present-at-hand” qualities, defamiliarising the performer’s own sense of themselves and reconfiguring their embodied practice.

If, as several recent papers claim, we have entered a new wave of “Entanglement HCI,” then we are still at a liminal stage prior to consensus around which sources underpin this paradigm shift or how they might inform actionable approaches to design practice. Now is the time to interpret technosocial mediation from a range of disciplinary perspectives, rather than settling on a narrow canon of literature. To this end, our paper enacts a diffractive dialogue between researchers from different disciplines, focusing on digital musical instruments to examine how technical knowledge from design and engineering can be read against the grain of critical theories from music, media, and cultural studies. Drawing on two object lessons—keyboards and step sequencers, plus their remediations in recent musical interaction research—we highlight interdependencies of theory, design, and practice, and we show how the idea of entanglement is itself entangled in a cross-disciplinary web.

Entanglement theories are well established in HCI discourse. These involve a commitment to view human experience in encounters with technology as relational and contingent, and research apparatuses as co-producers rather than passive observers of phenomena. In this paper, we argue that sound is the sensory modality best suited to the investigation of entanglements. Materialist theories of sound and listening guide both the design of a novel interactive sound installation and the methodological approach of a participant study exploring the experience of listening. We present a diffractive analysis whereby micro-phenomenological interview data is read with sonic theories, generating accounts that might otherwise remain mute: the temporal fluctuation and physical feeling of proximity in listener entanglements with sound, somatic intention setting, and plural interpretations of interactivity. Finally, we offer a series of provocations for HCI to embrace qualities of the sonic and consider epistemological positions grounded in other sense modalities.

This article examines how digital systems designers distil the messiness and ambiguity of the world into concrete data that can be processed by computing systems. Using Karen Barad's agential realism as a guide, we explore how data is fundamentally entangled with the tools and theories of its measurement. We examine data-enabled artefacts acting as Baradian apparatuses: they do not exist independently of the phenomenon they seek to measure but rather collect and co-produce observations from within their entangled state: the phenomenon and the apparatus co-constitute one another. Connecting Barad's quantum view of indeterminacy to the prevailing HCI discourse on the opportunities and challenges of ambiguity, we suggest that the very act of trying to stabilise a conceptual interpretation of data within an artefact has the paradoxical effect of amplifying and shifting ambiguity in interaction. We illustrate these ideas through three case studies from our own practices of designing digital musical instruments (DMIs). DMIs necessarily encode symbolic and music-theoretical knowledge as part of their internal operation, even as conceptual knowledge is not their intended outcome. In each case, we explore the nature of the apparatus, what phenomena it co-produces, and where the ambiguity lies to suggest approaches for design using these abstract theoretical frameworks.

From an engineering perspective, sensors provide measurements of phenomena in the world. Sensor data might be noisy, biased or otherwise subject to error, and these labels presuppose the existence of an objective ground truth the sensor is intended to approximate. This paper explores an alternative look at sensor signals as situated observations entangled with the systems they seek to measure, where meaning can be carried in qualitative particulars rather than quantitative and statistical analyses. Karen Barad’s ‘agential realism’ [3] and Graham Harman’s ‘tool-being’ [29] inform our approach to understanding together with the sensors, which become co-investigators and co-creators of the subsequent knowledge. We illustrate our collaborative effort with the sensor through a case study where we examine sensor signals from a device designed to query the experience of woodcarving by making the experience unfamiliar. We seek a qualitative approach to knowledge (co-)creation with sensor data from a more-than-human perspective.

Research on Digital Musical Instruments (DMIs) design highlights that materiality plays an important role in DMI design and musical interaction. However, DMI design research often focuses on technology-oriented factors, with less exploration of the meaning of materials in design practice. In this paper, we explore how DMI designers understand deformable sensor materials and how they use these as a resource for creative aesthetic design. Eleven DMI designers were invited to use a selection of deformable sensor materials to create prototype DMIs with them in a design activity. Three design approaches emerged, determined by how designers perceived and explored sensor materials. We discuss the potential of the methodology for exploring strongly entangled elements, such as material, gesture, and sound, in DMI design. The results contribute to the design practice for DMI designers and to further exploration of material-based design research in Human-Computer Interaction.

An implicit assumption in metaphor use is that it requires grounding in a familiar concept, prominently seen in the popular Desktop Metaphor. In human-to-human communication, however, abstract metaphors, without such grounding, are often used with great success. To understand when and why metaphors work, we present a case study of metaphor use in voice teaching. Voice educators must teach about subjective, sensory experiences and rely on abstract metaphor to express information about unseen and intangible processes inside the body. We present a thematic analysis of metaphor use by 12 voice teachers. We found that metaphor works not because of strong grounding in the familiar, but because of its ambiguity and flexibility, allowing shared understanding between individual lived experiences. We summarise our findings in a model of metaphor-based communication. This model can be used as an analysis tool within the existing taxonomies of metaphor in user interaction for better understanding why metaphor works in HCI. It can also be used as a design resource for thinking about metaphor use and abstracting metaphor strategies from both novel and existing designs.

Multi-sensory experiences underpin embodiment, whether with the body itself or technological extensions of it. Vocalists experience intensely personal embodiment, as vocalisation has few outwardly visible effects and kinaesthetic sensations occur largely within the body, rather than through external touch. We explored this embodiment using a probe which sonified laryngeal muscular movements and provided novel auditory feedback to two vocalists over a month-long period. Somatic and micro-phenomenological approaches revealed that the vocalists understand their physiology through its sound, rather than awareness of the muscular actions themselves. The feedback shaped the vocalists’ perceptions of their practice and revealed a desire for reassurance about exploration of one’s body when the body-as-sound understanding was disrupted. Vocalists experienced uncertainty and doubt without affirmation of perceived correctness. This research also suggests that technology is viewed as infallible and highlights expectations that exist about its ability to dictate success, even when we desire or intend to explore.

A shift in perspective is underway in design research and human-computer interaction (HCI) from humans as the centre of attention to considering complex assemblages of human and non-human stakeholders. While this shift is often approached from a broad ecological level, there is opportunity for a more local shift in understanding our day to day meeting with the material world. Drawing on the posthuman theories of Karen Barad, we explore the creation of a digital interactive system as a material-discursive practice in which matter and culture are inseparably entangled. We seek a fresh look at the process rather than the outcome of interactive system design through a diffractive reading of four traditional woodworking practices and an auto-ethnographic account of the development of a digital sensor and actuator apparatus as a way to find alternative ways of attending to materials in HCI.

Research on the design of Digital Musical Instruments (DMIs) has highlighted the importance of musical gestures and embodied interaction in DMI design. However, this research often focuses on technical and sonic factors of design, with less attention on how materials influence the design process and DMI design idea generation. Thus, this paper explores materiality in DMIs design through a material probe approach with deformable materials. This paper reports on a study with fifteen DMI designers investigating the evoked meaning of material properties in a musical context beyond their digital interactivity. Results suggest that material properties inspired participants’ design thinking, and there was a strong connection between tactility and imagined sound production. We also reported the patterns of gestural interaction of deformable materials in DMIs. We reflect on these results to report lessons learned that could inform interactive systems’ material design within and beyond the musical domain.

Digital interfaces are shrinking, driven by pressures of mass production and consumer culture, and often accompanied by a discourse of control, precision or convenience. Meanwhile, human bodies remain the same size, and the changing size of interfaces has implications for the formation of user identities. Drawing on embodied cognition, effort and entanglement theories of HCI, we explored the impact of interface size on the co-constitution of humans and technology. We designed an oversized digital musical instrument and invited musicians to use the instrument to create original performances. We found that both the performances and the musicians’ self-perception were influenced by the large size of the instrument, shining new light on the ways in which designing technology is designing humans and in turn culture.

Through experience, the techniques used by professional vocalists become highly ingrained and much of the fine muscular control needed for healthy singing is executed using well-refined mental imagery. In this paper, we provide a method for observing intention and embodied practice using surface electromyography (sEMG) to detect muscular activation, in particular with the laryngeal muscles. Through sensing the electrical neural impulses causing muscular contraction, sEMG provides a unique measurement of user intention, where other sensors reflect the results of movement. In this way, we are able to measure movement in preparation, vocalised singing, and in the use of imagery during mental rehearsal where no sound is produced. We present a circuit developed for use with the low voltage activations of the laryngeal muscles; in sonification of these activations, we further provide feedback for vocalists to investigate and experiment with their own intuitive movements and intentions for creative vocal practice.

We report on the outcomes of a hackathon organised around the themes of absurd musical interfaces, questionable sonic interactions and unworkable music designs. At the core of the project is the intention to explore absurd making as a way to support critical and disruptive design practices. We reflect on how surreal, nonsensical and fragile artefacts can be helpful to stretch and critique conventional ideas of what is useful and appropriate in technology research and development. After introducing both concepts and methods that shaped the event we present a selection of useless interfaces designed by the hackathon's attendees. These musical artefacts, and the considerations around them, are then discussed as a viable means for communicating both design concerns and future visions. We also consider two features identified as playing a crucial role within the event: the discovery of contradictions and the importance of context-based ingredients.

The design of a new technology entails the materialisation of values emerging from the specific community, culture and context in which that technology is created. Within the domain of musical interaction, HCI research often examines new digital tools and technologies which can carry unstated cultural assumptions. This paper takes a step back to present a value discovery exercise exploring the breadth of perspectives different communities might have in relation to the values inscribed in fictional technologies for musical interaction. We conducted a hands-on activity in which musicians active in different contexts were invited to envision not-yet-existent musical instruments. The activity revealed several sources of influence on participants' artefacts, including cultural background, instrumental training, and prior experience with music technology. Our discussion highlights the importance of cultural awareness and value rationality for the design of interactive systems within and beyond the musical domain.

This paper explores the process by which designers come to terms with an unfamiliar and ambiguous sensor material. Drawing on craft practice and material-driven interaction design, we developed a simple yet flexible sensor technology based on the movement of conductive elements within a magnetic field. Variations in materials and structure give rise to objects which produce a complex time-varying signal in response to physical interaction. Sonifying the signal yields nuanced and intuitive action-sound correspondences which nonetheless defy easy categorisation in terms of conventional types of sensors. We reflect on a craft-based exploration of the material by one of the authors, then report on two workshops with groups of designers of varying background. Through examining the objects produced and the experience of the participants, we explore the tension between tacit and explicit understanding of unfamiliar materials and the ways that material thinking can create new design opportunities.

Failure is a common artefact of challenging experiences, a fact of life for interactive systems but also a resource for aesthetic and improvisational performance. We present a study of how three professional pianists performed an interactive piano composition that included playing hidden codes within the music so as to control their path through the piece and trigger system actions. We reveal how apparent failures to play the codes occurred for diverse reasons including mistakes in their playing, limitations of the system, but also deliberate failures as a way of controlling the system, and how these failures provoked aesthetic and improvised responses from the performers. We propose that creative and performative interfaces should be designed to enable aesthetic failures and introduce a taxonomy that compares human approaches to failure with approaches to capable systems, revealing new creative design strategies of gaming, taming, riding and serving the system.

This paper reflects on the dynamics and practices of building a maker community around a new hardware platform. We examine the factors promoting the successful uptake of a maker platform from two perspectives: first, we investigate the technical and user experience considerations that users identify as the most important. Second, we explore the specific activities that help attract a community and encourage sustained participation. We present an inductive approach based on the case study of Bela, an embedded platform for creating interactive audio systems. The technical design and community building processes are detailed, culminating in a successful crowdfunding campaign. To further understand the community dynamics, the paper also presents an intensive three-day workshop with eight digital musical instrument designers. From observations and interviews, we reflect on the relationship between the platform and the community and offer suggestions for HCI researchers and practitioners interested in establishing their own maker communities.

Expression in musical practice is inextricably tied to the touch of the performer. In digital musical instruments (DMIs) the relationship of touch to sound is indirect: the nuances and fine detail of performer control can be flattened and limited during the translation of physical gesture to physical sound. The locus of this research is in the contact made between performer and DMI: focusing on this area can grant insight on fundamental issues of human computer interaction, particularly regarding intimate and expressive control of tangible interfaces. In this paper I present my research on this topic so far, which includes empirical studies that focus on specific parameters of performance where touch plays an integral role. The first study investigates how dynamic vibrations in an instrument's body can guide the hand of a performer and assist with intonation. The second study looks at asynchrony between action and sound and the influence this latency has on the perceived quality of an instrument.

This paper explores the design of digital musical instruments (DMIs) for exploratory play. Based on Gaver's principles of ludic design, we examine the ways in which people come to terms with an unfamiliar musical interface. We describe two workshops with the D-Box, a DMI designed to be modified and hacked by the user. The operation of the D-Box is deliberately left ambiguous to encourage users to develop their own meanings and interaction techniques. During the workshops we observed emergent patterns of exploration which revealed a rich process of exploratory play. We discuss our observations in relation to previous literature on appropriation, ambiguity and ludic engagement, and we provide recommendations for the design of playful and exploratory interfaces.

Stylized temporal behaviour is difficult to incorporate into computational models of environmental sound, requiring either a specialised algorithm or a potentially large amount of scripted automation. For decades Foley artists have relied on embodied performance to rapidly generate expressive sound effects for the moving image. This research project explores performance-led techniques in the design of computational audio for interactive applications.

We present a study investigating the impact of dynamic tactile feedback on performer navigation of a continuous pitch space on a digital musical instrument. Ten musicians performed a series of blind pitch selection and melodic tasks on a self-contained digital musical instrument with audio-frequency tactile feedback that was generated in response to their interaction. Results from the study show that tactile feedback can positively impact a performer's ability to play in tune when the instrument is hidden from sight, however with a temporal impact on performance. Furthermore, several playing techniques were observed that emerged from the performer's engagement with the tactile feedback conditions. We discuss the implications of our findings in the context of tangible interface design and non-visual interface navigation. We also discuss how our implementation suggests guidelines for future instruments and interfaces incorporating dynamic tactile feedback and present a novel tactile feedback technique that uses tactile 'beating'.

This paper addresses the question of how to extend the capabilities of a well-established interface in a way that respects users' existing expertise. The piano-style keyboard is among the most widely used and versatile of digital musical interfaces. However, it lacks the ability to alter the pitch of a note after it has been played, a limitation which prevents the performer from executing common expressive techniques including vibrato and pitch bending. We present a system for controlling pitch from the keyboard surface using capacitive touch sensors to measure the locations of the player's fingers on the keys. The large community of trained pianists makes the keyboard a compelling target for augmentation, but it also poses a challenge: how can a musical interface be extended while making use of the existing techniques performers have spent thousands of hours learning? In this paper, user studies with conservatory pianists explore the constraints of traditional keyboard technique and evaluate the usability of the continuous pitch control system. The paper also discusses implications for the extension of other established interfaces in musical and non-musical contexts.

This project aims to simulate the effect of Fishtank virtual reality in a telepresence system by using head-tracking and real-time control of a robotic camera. Despite the use of the term telepresence, the focus is here not on two-way communication between persons in separate spaces, but in creating a strong sense of physical presence by making the observed scene seem to be an extension of the room. The observer can thus interact with a 3D stage as though the screen were a window into the remote space. The research carried out here is unprecedented in that, to the authors' knowledge, nobody has employed one of the fundamental aspects of the Fishtank concept, namely the alignment of viewer-screen distance to camera-stage distance, in conjunction with a head-coupled camera system. The implementation is described followed by a brief discussion of its limitations and suggestions for future developments.

In this paper we present a new keyboard interface for computer music applications. Where traditional keyboard controllers report the velocity of each key-press, our interface senses up to five separate dimensions: velocity, percussiveness, rigidity, weight, and depth. These dimensions, which we identified based on the pedagogical piano literature and pilot studies with professional pianists, together present a rich picture of physical gestures at the keyboard, including information on the performer's motion before, during, and after a note is played. User studies confirm that the sensed dimensions are intuitive and controllable and that mappings between gesture and sound produce novel, playable musical instruments, even for users without prior keyboard experience. The multidimensional sensing capability demonstrated in this paper is also potentially applicable to button interfaces outside the musical domain.