We present a modular, sensor-based violin bow interface designed to augment musical performance and audiovisual composition. The system integrates real-time audio processing, visual feedback, and machine learning models for gesture recognition. Through an expert user study with seven violinists, we identified themes of ergonomic adaptation, co-agency, and creative potential. While results highlight the toolkit’s promise as both instrument and collaborator, limitations include the absence of non-expert perspectives, training protocols, and audience evaluation. This work contributes to human–agent interaction by positioning augmented instruments as sites of improvisational dialogue and outlines future directions in unifying system components, quantitative evaluation, and broader user studies.

We present ViolinBrush, a real‑time interactive system that transfers the nuanced bowing gestures of a live violinist to the strokes of a robotic painting arm. Unlike previous creative‑robotics projects that treat the machine as an autonomous or pre‑programmed agent, ViolinBrush is conceived as an experiment into whether embodied musical expressiveness can imbue a robot with a perceived humanness. Continuous descriptors of bow pitch and roll are mapped, respectively, to stroke stipple density and jitter, allowing micro-fluctuations in performance energy to visibly texture each mark.

This paper describes a prototype audio-visual performance of a Ufactory Uarm Swift and a live musician. In this setting, the robotic arm was used as an AI agent to create a visual representation of a musical work in real-time. An A4 white canvas was gradually filled with a mixture of black, blue, red and yellow paints across the span of approximately eight minutes. The musician, performing on an acoustic violin, fitted with a custom built audio interface, performed multiple versions of an improvisatory work developed specifically for the prototype performance. The following sections discuss our technical approach to programming and implementing the Ufactory Uarm Swift as a painting arm, reflections of the musical process and propose future directions for this project.