We introduce a data augmentation technique based on byte pair encoding and a BERTlike self-attention model to boost performance on spoken language understanding tasks.We compare and evaluate this method with a range of augmentation techniques encompassing generative models such as VAEs and performance-boosting techniques such as synonym replacement and back-translation.We show our method performs strongly on domain and intent classification tasks for a voice assistant and in a user-study focused on utterance naturalness and semantic similarity.

Social robots use gestures to express internal and affective states, but their interactive capabilities are hindered by relying on preprogrammed or hand-animated behaviors, which can be repetitive and predictable. We propose a method for automatically synthesizing affective robot movements given manually-generated examples. Our approach is based on techniques adapted from deep learning, specifically generative adversarial neural networks (GANs).

Using models to represent information about the world is a well known paradigm for successful robot control in the real world. Numerous methods exist today that can leverage these models to make robots perform tasks, either by directly exploiting the model structure or by accessing the model via a simulation. In this work, we explore how robots can use demonstrations to quickly build descriptive models of objects for manipulation tasks. Our framework uses demonstrations to incrementally build task-relevant geometric and physics-based object models that can be used to build simulations of the world that the robot is interacting with. We present experiments that involve estimating geometric features of an object when demonstration data from a user interacting with the object is available. We also demonstrate our method on the task of toppling a box with a 7-DoF manipulator equipped with a palm at its end. Using our approach, the robot is able to complete the task using only a few demonstrations.

Computer dialogue systems are designed with the intention of supporting meaningful interactions with humans. Common modes of communication include speech, text, and physical gestures. In this work we explore a communication paradigm in which the input and output channels consist of music. Specifically, we examine the musical interaction scenario of call and response. We present a system that utilizes a deep autoencoder to learn semantic embeddings of musical input. The system learns to transform these embeddings in a manner such that reconstructing from these transformation vectors produces appropriate musical responses. In order to generate a response the system employs a combination of generation and unit selection. Selection is based on a nearest neighbor search within the embedding space and for real-time application the search space is pruned using vector quantization. The live demo consists of a person playing a midi keyboard and the computer generating a response that is played through a loudspeaker.

Embodied cognition is a theory stating that the processes and functions comprising the human mind are influenced by a person's physical body. Embodied musical cognition is a theory of the musical mind stating that the person's body largely influences his or her musical experiences and actions (such as performing, learning, or listening to music). In this work, a proof of concept demonstrating the utility of an embodied musical cognition for robotic musicianship is described. Though alternative theories attempting to explain human musical cognition exist (such as cognitivism and connectionism), this work contends that the integration of physical constraints and musical knowledge is vital for a robot in order to optimize note generating decisions based on limitations of sound generating motion and enable more engaging performance through increased coherence between the generated music and sound accompanying motion. Moreover, such a system allows for efficient and autonomous exploration of the relationship between music and physicality and the resulting music that is contingent on such a connection.

Visual cues-based anticipation is a fundamental aspect of human-human interaction, and it plays an especially important role in the time demanding medium of group performance. In this work we explore the importance of visual gesture anticipation in music performance involving human and robot. We study the case in which a human percussionist is playing a four-piece percussion set, and a robot musician is playing either the marimba, or a three-piece percussion set. Computer Vision is used to embed anticipation in the robotic response to the human gestures. We developed two algorithms for anticipation, predicting the strike location about 10 mili-seconds or about 100 mili-seconds before it occurs. Using the second algorithm, we show that the robot outperforms, on average, a group of human subjects, in synchronizing its gesture with a reference strike. We also show that, in the tested group of users, having some time in advance is important for a human to synchronize the strike with a reference player, but, from a certain time, that good influence stops increasing.