Learning to play music is an embodied process, but traditional tools like scores and recordings overlook the role of gesture. While video tutorials offer visual cues, they remain detached from the instrument. We present ReTouche, an interactive system that projects synchronized notes and hand gestures directly onto the actuated keys of a player piano. The system includes a pipeline for adapting publicly available overhead-view YouTube videos and supports interactions for video-based learning, such as sectional practice, layered guidance, and self-recording. We evaluate ReTouche through a comparative structured observation study with YouTube-based self-learning (n=18), a two-week autoethnography study (n=3), and a focus group with professional piano teachers (n=4). Our findings show that embodied representations can ground self-guided piano learning by anchoring gesture, sound, and action within the instrument. Learners appropriated these representations to develop strategies and sustain motivation, while teachers saw potential for integrating ReTouche as a complement to conventional pedagogy.

While legacy sketchnotes capture rich design rationales and inspirations, they are rarely reused in contemporary practice. We present Co-Ideation Across Time (CIAT), utilizing Large Language Models (LLMs) to transform decades-old design sketchnotes into interactive "AI-augmented Knowledge Objects". Our system digitizes over 2,000 pages of alumni sketchnotes and connects them with conversational agents trained on corresponding theses and publications, enabling current and future students to engage in multimodal dialogue with past ideas and researchers. An exploratory evaluation with 12 participants showed that interacting with the system stimulated deeper understanding of abstract concepts, idea diversity, and fostered a stronger sense of continuity with the community’s legacy. Our contributions are threefold: (1) a method for integrating design legacies with LLM-driven conversational agents; (2) an empirical study demonstrating how this approach supports learning and intergenerational knowledge sharing; and (3) a conceptual framing of AI-Augmented Knowledge Objects as active participants in design ideation.

MirrorMorphose is an interactive mirror that uses gaze as an emotional portal between space and time. When the viewer maintains eye contact with their reflection, the image gradually morphs into a childhood version of themselves, a family member, or another person, who stares back and listens carefully, inviting a dialogue across identity and time. Through this slow visual transformation, the project explores how sustained gaze fosters empathy, self-recognition, and the blurring of boundaries between self and other. Drawing from practices of meditation, hypnosis, and therapeutic mirroring, MirrorMorphose proposes the gaze as an active medium of connection, both technological and intimate. The project challenges the viewer to confront their own reflection as a lived encounter, where memory, emotion, and technology intertwine. In doing so, seeing becomes a form of becoming, engaging viewers in having reflections or conversations in a meditative act of meeting oneself through the eyes of another.

CHI ’25, Yokohama, Japan

This paper presents an autoethnography of 3 years of learning to play the theremin, an instrument lacking tangible feedback. While the theremin is typically invoked in HCI to emphasize the importance of the tactile modality, we interrogate how I, the player, attained musical proficiency without touch.

Typical foreign language (L2) pronunciation training focuses mainly on individual sounds. Intonation, the patterns of pitch change across words or phrases is often neglected, despite its key role in word-level intelligibility and in the expression of attitudes and affect. This paper examines hand-controlled real-time vocal synthesis, known as Performative Vocal Synthesis (PVS), as an interaction technique for practicing L2 intonation in computer aided pronunciation training (CAPT).

This paper explores how multi-modal body-syntonic interactive systems may be used to teach children to play the piano beyond the typical focus on reading musical scores and "surface correctness". Our work draws from Dalcroze Eurhythmics, a method of music pedagogy aimed at instilling an understanding of music rooted in the body. We present a Dalcrozian process of piano learning as a five-step iterative cycle of: listen, internalize, extend, analyze, and improvise. As a case study of how digital technologies may support this process, we present Andantino, a set of extensions of Andante, which projects musical lines as miniature light silhouettes that appear to walk on the keyboard of a player piano. We discuss features of Andantino based on each stage, or step, of the iterative framework and discuss directions for future research, based on two preliminary studies with children between the ages of 7 and 13.

This paper introduces a new framework to guide the design of interactive music learning systems, focusing on the piano. Taking a Reflective approach, we identify the implicit assumption behind most existing systems-that learning music is learning to play correctly according to the score-and offer an alternative approach. We argue that systems should help cultivate higher levels of musicianship beyond correctness alone for students of all levels. Drawing from both pedagogical literature and the personal experience of learning to play the piano, we identify three skills central to musicianship-listening, embodied understanding, and creative imagination-which we generalize to the Inspect, Embody, Invent framework. To demonstrate how this framework translates to design, we discuss two existing interfaces from our own research-MirrorFugue and Andante-both built on a digitally controlled player piano augmented by in-situ projection. Finally, we discuss the framework's relevance toward bigger themes of embodied interactions and learning beyond the domain of music.

Playing a musical instrument involves a complex set of continuous gestures, both to play the notes and to convey expression. To learn an instrument, a student must learn not only the music itself but also how to perform these bodily gestures. We present MirrorFugue, a set of three interfaces on a piano keyboard designed to visualize hand gesture of a remote collaborator. Based their spatial configurations, we call our interfaces Shadow, Reflection, and Organ. We describe the configurations and detail studies of our designs on synchronous, remote collaboration, focusing specifically on remote lessons for beginners. Based on our evaluations, we conclude that displaying the to-scale hand gestures of a teacher at the locus of interaction can improve remote piano learning for novices.

We introduce a technique for exploring multi-layered images by scraping arbitrary areas to determine meaningful relationships. Our system, called Wetpaint, uses perceptual depth cues to help users intuitively navigate between corresponding layers of an image, allowing a rapid assessment of changes and relationships between different views of the same area. Inspired by art diagnostic techniques, this tactile method could have distinct advantages in the general domain as shown by our user study. We propose that the physical metaphor of scraping facilitates the process of determining correlations between layers of an image because it compresses the process of planning, comparison and annotation into a single gesture. We discuss applications for geography, design, and medicine.