We present a modular, sensor-based violin bow interface designed to augment musical performance and audiovisual composition. The system integrates real-time audio processing, visual feedback, and machine learning models for gesture recognition. Through an expert user study with seven violinists, we identified themes of ergonomic adaptation, co-agency, and creative potential. While results highlight the toolkit’s promise as both instrument and collaborator, limitations include the absence of non-expert perspectives, training protocols, and audience evaluation. This work contributes to human–agent interaction by positioning augmented instruments as sites of improvisational dialogue and outlines future directions in unifying system components, quantitative evaluation, and broader user studies.

We propose a novel, cost-effective soft sensor capable of detecting contact force, multiple touch points, and reflecting sensor interaction in real-time with a 3D virtual surface representation. Our fabrication process has been optimized for cost efficiency through careful material selection, utilization of automated machinery, and low-cost hardware. The sensor can be easily replicated without the need for complex laboratory equipment. The sensor employs trained neural network models for real-time signal translation into localization, force measurement, and deformation mapping. We have also developed an efficient data collection system that captures accurate 2D localization, force measurement, and 3D surface data to generate a high-quality pre-validated data set. This data set is filtered using prior knowledge before being fed to two neural network models. Our interactive prototype demonstrates the stability and accuracy of the low-cost soft sensor, delivering reliable results in both single-point and multi-point contact scenarios.

In human-computer interaction, particularly in multimedia delivery, information is communicated to users sequentially, whereas users are capable of receiving information from multiple sources concurrently. This mismatch indicates that a sequential mode of communication does not utilise human perception capabilities as efficiently as possible. This article reports an experiment that investigated various speech-based (audio) concurrent designs and evaluated the comprehension depth of information by comparing comprehension performance across several different formats of questions (main/detailed, implied/stated). The results showed that users, besides answering the main questions, were also successful in answering the implied questions, as well as the questions that required detailed information, and that the pattern of comprehension depth remained similar to that seen to a baseline condition, where only one speech source was presented. However, the participants answered more questions correctly that were drawn from the main information, and performance remained low where the questions were drawn from detailed information. The results are encouraging to explore the concurrent methods further for communicating multiple information streams efficiently in human-computer interaction, including multimedia.

In Music Emotion Recognition (MER) research, most existing research uses human engineered audio features as learning model inputs, which require domain knowledge and much effort for feature extraction. We propose a novel end-to-end deep learning approach to address music emotion recognition as a regression problem, using the raw audio signal as input. We adopt multi-view convolutional neural networks as feature extractors to learn feature representations automatically. Then the extracted feature vectors are merged and fed into two layers of Bidirectional Long Short-Term Memory to capture temporal context sufficiently. In this way, our model is capable of recognizing dynamic music emotion without requiring too much workload on domain knowledge learning and audio feature processing. Combined with data augmentation strategies, the experimental results show that our model outperforms the state-of-the-art baseline with a significant margin in terms of R2score (approximately 16%) on the Emotion in Music Database.

Media multiplicities are media artworks that employ multiple networked digital devices to create holistic aesthetic effects. Examples include the networked light artworks of Squidsoup, the Spaxels drone-mounted light performances, DrawBots, Siftables and many others. In multiplicitous media artworks, each individual device is a programmable node connected to other nodes via a network connection, and may combine any number of sensors and actuators. A number of development technologies support artists and designers to configure and create media multiplicities, but this domain offers new challenges for creative practitioners. This workshop aims to bring together experts in creative coding and interaction design to discuss and conceptualise frameworks for the practice of media multiplicities. Open challenges include: speed of setup; ease of hardware configuration; speed of code deployment; ability to model and simulate works in VR; network connectivity and stability; and understanding network, computation and power constraints.

The project investigates how interactions with complex (biologically inspired swarming) behaviors of multiple robots are understood by human participants within a performative and dramaturgical system. Nonanthropomorphic robots in the form of roller skates are used in innovative ways by creating social formations from their movements, for example a leader and followers in a conga line. Synchronized audio signals and speech-like sonic structures are used in innovative ways by influencing and engaging the participant's interactions with the robots. Localization data of the robots in space is mapped to control the surround sound and lighting within the space. This is used to enhance audience immersion and engagement within the interactive performance work.

This workshop will introduce creative coding audio for the Raspberry Pi, using the 'beads' platform for audio programming, and the 'HappyBrackets' platform for inter-device communication and sensor data acquisition. We will demonstrate methods to allow each self-contained battery-powered device to acquire sensor data about its surroundings and the way it is being interacted with, as well as methods for designing systems where groups of these devices wirelessly communicate their state, allowing new interaction possibilities and approaches.

This paper describes the development of the hardware and software for Bloom, a light installation installed at Kew Gardens, London in December of 2016. The system is made up of a set of nearly 1000 distributed pixel devices each with LEDs, GPS sensor, and sound hardware, networked together with WiFi to form a display system. Media design for this system required consideration of the distributed nature of the devices. We outline the software and hardware designed for this system, and describe two approaches to the software and media design, one whereby we employ the distributed devices themselves for computation purposes (the approach we ultimately selected), and another whereby the devices are controlled from a central server that is performing most of the computation necessary. We then review these approaches and outline possibilities for future research.

This paper describes the performance, mapping, transformation and representation phases of a model for gesture-triggered musical creativity. These phases are articulated in an example creative environment, Hyper-Shaku (Border-Crossing), an audio-visually augmented shakuhachi performance to demonstrate the adaptive, empathetic response of the generative systems. The shakuhachi is a Japanese traditional end-blown bamboo Zen flute. Its 5 holes and simple construction require subtle and complex gestural movements to produce its diverse range of pitches, vibrato and pitch inflections, making it an ideal candidate for gesture capture. The environment uses computer vision, gesture sensors and computer listening to process and generate electronic music and visualization in real time response to the live performer. The integration of looming auditory motion and Neural Oscillator Network (NOSC) generative modules are implemented in this example.

Musicians in training need to understand the sound they are producing in order to improve its deficient aspects. Verbal feedback from musical masters is the usual method used for attaining this understanding. However, using realtime sound visualisation as a complementary form of feedback allows the large amounts of data typical of real-time acoustic analysis to be employed within training. This improves the efficiency of the feedback loop normally present within musical training and pedagogy. The implementation and effect of such a system is discussed.