Live music provides a uniquely rich setting for studying creativity and interaction due to its spontaneous nature. The pursuit of live music agents—intelligent systems supporting real-time music performance and interaction—has captivated researchers across HCI, AI, and computer music for decades, and recent advancements in AI suggest unprecedented opportunities to evolve their design. However, the interdisciplinary nature of music has led to fragmented development across research communities, hindering effective communication and collaborative progress. In this work, we bring together perspectives from these diverse fields to map the current landscape of live music agents. Based on our analysis of 184 systems across both academic literature and video, we develop a comprehensive design space that categorizes dimensions spanning usage contexts, interactions, technologies, and ecosystems. By highlighting trends and gaps in live music agents, our design space offers researchers, designers, and musicians a structured lens to understand existing systems and shape future directions in real-time human-AI music co-creation. We release our annotated systems as a living artifact at https://live-music-agents.github.io.

Recent years have seen a surge in musical performances accompanied by generative agents. Artificial voices, timbres synthesized by neural networks, and agents that mirror or respond to human performers are rapidly taking the stage. In parallel, practitioners in human-computer interaction (HCI) and music technology have called for practice-based research that identifies the most salient affordances of these developments by examining their use in the real-world contexts of music making. To advance practice-based research on human-AI music creation, we present a longitudinal account of two months of codesign with top local jazz musicians, spanning early explorations, the identification of emerging goals, and rehearsals. Our work culminates in a public concert for a live audience of 97, featuring three pieces co-improvised with AI agents. Drawing on systems including VampNet, Somax2, and the jam_bot, each piece was tailored to the stylistic strengths of the performers and the unique strengths and limitations of each system. Through this extensive iterative process, we uncovered a wide range of design interventions, from augmenting GenAI systems with a guitar pedal to situate it in a loop-based creative practice, to enabling musicians to anticipate AI response by visually forecasting its predictions. Where musicians tended to rein in the wilder qualities of the generative systems, some audience members expected a human-AI performance to allow as much agency and spontaneity as possible. In post-concert reflection, musicians also expressed the desire to practice more which in turn could enable them to let the agency of the systems shine. They also encouraged future musicians to lean more into the uncertainty. Together, we see a unique practice emerging through this musician-AI live improv medium.

CHI ’25, Yokohama, Japan

Synthesizers are powerful tools that allow musicians to create dynamic and original sounds. Existing commercial interfaces for synthesizers typically require musicians to interact with complex low-level parameters or to manage large libraries of premade sounds. To address these challenges, we implement SynthScribe — a fullstack system that uses multimodal deep learning to let users express their intentions at a much higher level. We implement features which address a number of difficulties, namely 1) searching through existing sounds, 2) creating completely new sounds, and 3) making meaningful modifications to a given sound. This is achieved with three main features: a multimodal search engine for a large library of synthesizer sounds; a user centered genetic algorithm by which completely new sounds can be created and selected given the users preferences; a sound editing support feature which highlights and gives examples for key control parameters with respect to a text or audio based query. The results of our user studies show SynthScribe is capable of reliably retrieving and modifying sounds while also affording the ability to create completely new sounds that expand a musicians creative horizon.

Text-to-image generative models have demonstrated remarkable capabilities in generating high-quality images based on textual prompts. However, crafting prompts that accurately capture the user’s creative intent remains challenging. It often involves laborious trial-and-error procedures to ensure that the model interprets the prompts in alignment with the user’s intention. To address these challenges, we present Promptify, an interactive system that supports prompt exploration and refinement for text-to-image generative models. Promptify utilizes a suggestion engine powered by large language models to help users quickly explore and craft diverse prompts. Our interface allows users to organize the generated images flexibly, and based on their preferences, Promptify suggests potential changes to the original prompt. This feedback loop enables users to iteratively refine their prompts and enhance desired features while avoiding unwanted ones. Our user study shows that Promptify effectively facilitates the text-to-image workflow, allowing users to create visually appealing images on their first attempt while requiring significantly less cognitive load than a widely-used baseline tool.