Ahstract- This study explores the potential of music driven sonification as an effective method for improving safety in humanrobot collaboration. Building on the rich expressive content of music, this study assesses the communicative potential of both low level musical features, such as pitch, tempo, and timbre; and high level music features of rhythmic stability and tension-release. Two music datasets have been created, labeled, and evaluated based on five criteria: safety/danger, approachability, risk of failure, and urgency. The first dataset consists of prerecorded song clips while the second one contains original compositions designed to isolate high-level musical features. 400 participants annotated our datasets base on the five criteria. Our findings reveal significant correlations between musical features such as timbre, harmonic tension, and note onset; and the perception of safety, urgency, and risk. Based on these results, we developed a framework and an audio plugin for music-driven sonification of robotic gestures to support safe human-robot interaction.

For the past few decades, robotics researchers have focused on Western music with only a handful of attempts to address music from other parts of the world such as South Indian Classical music (a.k.a. Carnatic music) - a music form popular in the southern part of India. In this work, we developed a Carnatic violin playing robot called Hathaani that uses pitch data to manipulate the left hand and amplitude data to make bow changes and modify dynamics. This work bears promise to revolutionize productions in Carnatic music by providing solutions to support gamakas - pitch based embellishments that are at the core of this music form. It can also have a significant contribution to education, by introducing a violin playing robot that can help students understand the nuances of gamaka playing in a repeated and systematic manner. A violin-playing robot can provide a much more accurate and expressive rendition of this music in comparison to software-based emulations.

This paper investigates how melodic prediction can enhance synchronization and fluency in human-robot collaborative tasks. We leverage humans’ natural ability to anticipate musical progressions to improve the fluency of handoff interactions. Through an experiment with 21 participants performing a dual-task scenario of sorting tiles while handing objects to a robot, we compared three sonification approaches: robot consequential sounds, musical scales, and musical melodies. Participants synchronized their handoffs based on the finale of the scale/melody, and aligned their actions with the leading tone rather than the final tonic note. Results demonstrate that both musical conditions significantly improved timing accuracy and enabled participants to better perform concurrent tasks compared to motor sounds alone. Melodies proved to be statistically significantly more consistent over repeated use of the stimuli, which shows that the diverse nature of tonal melodies can improve long term interactions without sacrificing performance. These findings suggest that the predictive qualities of tonal melodies provides an effective tool for anticipatory action.

Abstract This study explores the use of sonification in improving the In Vitro Fertilization (IVF) experience for expectant parents. The authors propose that the sonification of IVF videos, depicting the division of embryonic cells, can aid in enhancing understanding of embryonic development and foster emotional connections with medical imagery. To demonstrate this concept, the authors developed a system for creating emotional music synchronized with IVF videos, using a combination of rule-based methods and a neural network. The system was evaluated through two studies, including an online survey and a semi-structured interview, with participants being shown the sonified IVF videos. Results indicated a strong preference for videos with sonification among expectant mothers and a perceived relationship between the sonification and the video. The study supports previous findings on the trade-off between aesthetics and information conveyed through sonification, and suggests that sonification can improve the overall experience and stimulate curiosity without necessarily imparting new information about the embryos.

Lyric generation is a popular sub-field of natural language generation that has seen growth in recent years. Pop lyrics are of unique interest due to the genre’s unique style and content, in addition to the high level of collaboration that goes on behind the scenes in the professional pop songwriting process. In this paper, we present a collaborative line-level lyric generation system that utilizes transfer-learning via the T5 transformer model, which, till date, has not been used to generate pop lyrics. By working and communicating directly with professional songwriters, we develop a model that is able to learn lyrical and stylistic tasks like rhyming, matching line beat requirements, and ending lines with specific target words. Our approach compares favorably to existing methods for multiple datasets and yields positive results from our online studies and interviews with industry songwriters.

This paper describes a quasi-passive transradial prosthesis designed to restore drumstick grip control for amputee drummers. A compact motor with low rotor resistance driven by Electromyography (EMG) is implemented in the prosthesis to support real-time drum performances. A variety of grip profiles are achieved by adjusting the stiffness and damping of an impedance controller based on machine learning predictions from EMG data. In addition to the prosthetic design, the paper presents a dynamic model that simulates the drumstick trajectories and estimates the control gains for various natural bouncing patterns. We evaluate the effectiveness of the design through human experiments under real-time performing scenarios for two common drumming grip techniques. The results demonstrate that the prosthetic model can support similar bouncing patterns to the drummer’s healthy hand for single, double, and triple-stroke rolling techniques. The results also show that the system supports usable real-time periodic grip change from single to double-stroke, successfully simulating the common paradiddle drumming technique.

Research in human-robot interaction has focused on the relationship between a single robot and a single human participant. Only limited research has addressed the contrasting dynamic when humans interact with a group of robots. This dynamic adds additional human-robot interaction considerations, such as the level of entitativity, which is the identification of a group as a single entity as opposed to a collection of individuals. This paper proposes that emotional music prosody can play a key role in improving the interaction between humans and groups of robots by modifying the level of entitativity. Musical prosody refers to the use of pitch, rhythm and timbre features derived from language, but used without semantic meaning.We conducted a between-group experiment, presenting to subjects a group of industrial robotic arms performing a task either without sound, with the same emotional musical prosody voice for each robot, or with contrasting voices for different robots. We were able to show with significant results that the use of musical prosody improved likeability and trust over soundless gestures for groups of robots. We also demonstrate that, through subtle variations, prosody is able to alter the level of entitativity perceived by external observers. Finally, our results indicate a complex relationship between entitativity and common HRI metrics with higher levels of entitativity leading to improved performance, contradicting past literature.

Current research in human-agent interaction primarily focuses on short term interaction and rarely addresses day to day use. We propose a prototype system based on a genetic algorithm that places long term interaction as the core design goal. The goal of this system is to develop stand alone long-term development and provide a platform for future post-processing of deep learning generations. This paper addresses these issues through the domain of musical interaction and improvisation, a field that incorporates dialogue-like interaction built on stylistic constraints. We contend that the objectives of continual knowledge development and building relationships are key to long-term human interaction, and design the genetic algorithm specifically around these concepts. Our eventual goal of the prototype is a future application of post processing for deep learning generative systems.

As emotion plays a growing role in robotic research it is crucial to develop methods to analyze and compare among the wide range of approaches. To this end we present a survey of 1427 IEEE and ACM publications that include robotics and emotion. This includes broad categorizations of trends in emotion input analysis, robot emotional expression, studies of emotional interaction and models for internal processing. We then focus on 232 papers that present internal processing of emotion, such as using a human's emotion for better interaction or turning environmental stimuli into an emotional drive for robotic path planning. We conducted constant comparison analysis of the 232 papers and arrived at three broad categorization metrics - emotional intelligence, emotional model and implementation - each including two or three subcategories. The subcategories address the algorithm used, emotional mapping, history, the emotional model, emotional categories, the role of emotion, the purpose of emotion and the platform. Our results show a diverse field of study, largely divided by the role of emotion in the system, either for improved interaction, or improved robotic performance. We also present multiple future opportunities for research and describe intrinsic challenges common in all publications.

We present a divergent approach to robotic sonification with the goal of improving the quality and safety of human-robot interactions. Sonification (turning data into sound) has been underutilized in robotics, and has broad potential to convey robotic movement and intentions to users without requiring visual engagement. We design and evaluate six different sonifications of movements for a robot with four degrees of freedom. Our sonification techniques include a direct mapping from each degree of freedom to pitch and timbre changes, emotion-based sound mappings, and velocity-based mappings using different types of sounds such as motors and music. We evaluate these sonifications using metrics for ease of use, enjoyment/appeal, and conveyance of movement information. Based on our results, we make recommendations to inform decisions for future robot sonification design. We suggest that when using sonification to improve safety of human-robot collaboration, it is necessary not only to convey sufficient information about movements, but also to convey that information in a pleasing and even social way to to enhance the human-robot relationship.

Robotic Musicianship research at Georgia Tech Center for Music Technology (GTCMT) focuses on the construction of autonomous and wearable robots that can analyze, reason, and generate music. The goal of our research is to facilitate meaningful and inspiring musical interactions between humans and artificially creative machines. In this talk I present the work conducted by the Robotic Musicianship Group at GTCMT over the last 15 years, highlighting the motivation, research questions, platforms, methods, and underlining guidelines for our work.

As human-robot collaboration opportunities continue to expand, trust becomes ever more important for full engagement and utilization of robots. Affective trust, built on emotional relationship and interpersonal bonds is particularly critical as it is more resilient to mistakes and increases the willingness to collaborate. In this paper we present a novel model built on music-driven emotional prosody and gestures that encourages the perception of a robotic identity, designed to avoid uncanny valley. Symbolic musical phrases were generated and tagged with emotional information by human musicians. These phrases controlled a synthesis engine playing back pre-rendered audio samples generated through interpolation of phonemes and electronic instruments. Gestures were also driven by the symbolic phrases, encoding the emotion from the musical phrase to low degree-of-freedom movements. Through a user study we showed that our system was able to accurately portray a range of emotions to the user. We also showed with a significant result that our non-linguistic audio generation achieved an 8% higher mean of average trust than using a state-of-the-art text-to-speech system.

Embodied cognition is a theory stating that the processes and functions comprising the human mind are influenced by a person's physical body. Embodied musical cognition is a theory of the musical mind stating that the person's body largely influences his or her musical experiences and actions (such as performing, learning, or listening to music). In this work, a proof of concept demonstrating the utility of an embodied musical cognition for robotic musicianship is described. Though alternative theories attempting to explain human musical cognition exist (such as cognitivism and connectionism), this work contends that the integration of physical constraints and musical knowledge is vital for a robot in order to optimize note generating decisions based on limitations of sound generating motion and enable more engaging performance through increased coherence between the generated music and sound accompanying motion. Moreover, such a system allows for efficient and autonomous exploration of the relationship between music and physicality and the resulting music that is contingent on such a connection.

Visual cues-based anticipation is a fundamental aspect of human-human interaction, and it plays an especially important role in the time demanding medium of group performance. In this work we explore the importance of visual gesture anticipation in music performance involving human and robot. We study the case in which a human percussionist is playing a four-piece percussion set, and a robot musician is playing either the marimba, or a three-piece percussion set. Computer Vision is used to embed anticipation in the robotic response to the human gestures. We developed two algorithms for anticipation, predicting the strike location about 10 mili-seconds or about 100 mili-seconds before it occurs. Using the second algorithm, we show that the robot outperforms, on average, a group of human subjects, in synchronizing its gesture with a reference strike. We also show that, in the tested group of users, having some time in advance is important for a human to synchronize the strike with a reference player, but, from a certain time, that good influence stops increasing.

In this paper, we connect local properties in a mobile planar multi-robot team to the task of creating decentralized real time algorithmic music. Using a nonlinear formation control law inspired by the consensus equation, we map the local motion parameters of robots to Euclidean rhythms with the use of sequencers. The control parameters allow a human user to direct this decentralized musical process by guiding and interfering with the robots' motion, which subsequently affects their musical activity. We simulate such a robotic system in real time, demonstrating the expressiveness of the decentralized algorithmic musical output as well as a number of behaviors that arise out of the manipulation of the control parameters.

Mobile Music Touch (MMT) helps teach users to play piano melodies while they perform other tasks. MMT is a lightweight, wireless haptic music instruction system consisting of fingerless gloves and a mobile Bluetooth enabled computing device, such as a mobile phone. Passages to be learned are loaded into the mobile phone and are played repeatedly while the user performs other tasks. As each note of the music plays, vibrators on each finger in the gloves activate, indicating which finger is used to play each note. We present two studies on the efficacy of MMT. The first measures 16 subjects' ability to play a passage after using MMT for 30 minutes while performing a reading comprehension test. The MMT system was significantly more effective than a control condition where the passage was played repeatedly but the subjects' fingers were not vibrated. The second study compares the amount of time required for 10 subjects to replay short, randomly generated passages using passive training versus active training. Participants with no piano experience could repeat the passages after passive training while subjects with piano experience often could not.

We present Shimon, an interactive improvisational robotic marimba player, developed for research in Robotic Musicianship. The robot listens to a human musician and continuously adapts its improvisation and choreography, while playing simultaneously with the human. We discuss the robot's mechanism and motion-control, which uses physics simulation and animation principles to achieve both expressivity and safety. We then present a novel interactive improvisation system based on the notion of gestures for both musical and visual expression. The system also uses anticipatory beat-matched action to enable real-time synchronization with the human player. Our system was implemented on a full-length human-robot Jazz duet, displaying highly coordinated melodic and rhythmic human-robot joint improvisation. We have performed with the system in front of a live public audience.

Shimon is a interactive robotic marimba player, developed as part of our ongoing research in Robotic Musicianship (RM). One of the potential benefits of RM is that it provides human players with embodied information that relates spatial movement to tone generation. This can aid in anticipation and coordination of synchronous playing. As part of a human-robot Jazz improvisation system, we present an anticipatory system enabling beat-matched real-time synchronization. Our system enables flexible, yet coordinated call-and-response, a standard type of musical interaction. It was used in a live public human-robot joint Jazz performance. We also describe a preliminary study evaluating the effect of embodiment on this call-and-response musical synchronization task. We conducted a 3×2 within-subject study manipulating the level of embodiment (visual co-presence, physical presence but visual occlusion, and synthesized sound), and the accuracy of the robot's response. Our findings indicate that synchronization is aided by visual contact when uncertainty is high, but that pianists can resort to internal rhythmic coordination in more predictable settings. We find that visual coordination is more effective for synchronization for slow sequences compared to faster sequences; and that occluded physical presence may be less effective than audio-only note generation.

We present our approach for facilitating musical interaction between robotic musicians and humans (musicians as well as non-musicians) in an improvisatory and exploratory manner. By using the iPhone as a musical instrument, we offer a quick and effective way for anyone to start creating music and communicating directly with our marimba playing robot, Shimon. Through automated style analysis of jazz masters, our system creates a novel improvisatory and interactive musical language. In a call and response exchange, the robot expands on short tunes created by human performers, in a musically rich and meaningful manner. As part of the interaction, users can also controlling the musical influences that combine to make up Shimon's improvisatory language, creating novel and inspiring responses.

This paper describes the implementation of a leader-follower model in a musical HRI based on beat detection analysis and a novel turn taking scheme. The project enables Haile, a robotic percussionist, to fluidly interact with humans in the context of an improvisatory jam session. The long-term goal of this work is to facilitate dynamic interactions between humans and machines that will lead to novel and inspiring musical outcomes.

The paper introduces Shimon: a socially interactive and improvisational robotic marimba player. It presents the interaction schemes used by Shimon in the realization of an interactive musical jam session among human and robotic musicians.

We present new developments in the improvisational robotic percussionist project, aimed at improving human-robot interaction through design, mechanics, and perceptual modeling. Our robot, named Haile, listens to live human players, analyzes perceptual aspects in their playing in real-time, and uses the product of this analysis to play along in a collaborative and improvisatory manner. It is designed to combine the benefits of computational power in algorithmic music with the expression and visual interactivity of acoustic playing. Haile's new features include an anthropomorphic form, a linear-motor based robotic arm, a novel perceptual modeling implementation, and a number of new interaction schemes. The paper begins with an overview of related work and a presentation of goals and challenges based on Haile's original design. We then describe new developments in physical design, mechanics, perceptual implementation, and interaction design, aimed at improving human-robot interactions with Haile. The paper concludes with a description of a user study, conducted in an effort to evaluate the new functionalities and their effectiveness in facilitating expressive musical human-robot interaction. The results of the study show correlation between human's and Haile's rhythmic perception as well as user satisfaction regarding Haile's perceptual and mechanical abilties. The study also indicates areas for improvement such as the need for better timbre and loudness control and more advance and responsive interaction schemes.

The paper presents the theoretical background and the design scheme for a perceptual and improvisational robotic marimba player that interacts with human musicians in a visual and acoustic manner. Informed by an evaluation of a previously developed robotic percussionist, we present the extension of our work to melodic and harmonic realms with the design of a robotic player that listens to, analyzes and improvises pitch-based musical materials. After a discussion of the motivation for the project, theoretical background and related work, we present a set of research questions followed by our hardware and software approaches designed to address these questions. The paper concludes with a description of our plans to implement and embed these approaches in the robotic marimba player that will be used in workshops and concerts.

The paper presents our approach for human-machine interaction with an anthropomorphic mechanical percussionist that can listen to live players, analyze perceptual musical aspects in real-time, and use the product of this analysis to play along in a collaborative manner. Our robot, named Haile, is designed to combine the benefits of computational power, perceptual modeling, and algorithmic music with the richness, visual interactivity, and expression of acoustic playing. We believe that when interacting with live players, Haile can facilitate a musical experience that is not possible by any other means, inspiring users to collaborate with it in novel and expressive manners. Haile can, therefore, serve a test-bed for novel forms of musical human-machine interaction, bringing perceptual aspects of computer music into the physical world both visually and acoustically.