This paper presents an experiment with three artificial agents adopting different strategies when being interrupted by human conversational partners. The agent either ignored the interruption (the most common behavior in conversational engines to date), yielded the turn to the human conversational partner right away, or acknowledged the interruption, finished its thought and then responded to the content of the interruption. Our results show that this change in the agent’s conversational behavior had a significant impact on which personality traits people assigned to the agent, as well as how much they enjoyed interacting with it. Moreover, the data also indicates that human interlocutors adapted their own conversational behavior. Our findings suggest that the interactive behavior of an artificial agent should be carefully designed to match its desired personality and the intended conversational dynamics.

This paper presents the slurk software, a lightweight interaction server for setting up dialog data collections and running experiments. slurk enables a multitude of settings including text-based, speech and video interaction between two or more humans or humans and bots, and a multimodal display area for presenting shared or private interactive context. The software is implemented in Python with an HTML and JavaScript frontend that can easily be adapted to individual needs. It also provides a setup for pairing participants on common crowdworking platforms such as Amazon Mechanical Turk and some example bot scripts for common interaction scenarios.

In this work, we study entrainment of users playing a creative reference resolution game with an autonomous dialogue system. The language understanding module in our dialogue system leverages annotated human-wizard conversational data, openly available knowledge graphs, and crowd-augmented data. Unlike previous entrainment work, our dialogue system does not attempt to make the human conversation partner adopt lexical items in their dialogue, but rather to adapt their descriptive strategy to one that is simpler to parse for our natural language understanding unit. By deploying this dialogue system through a crowd-sourced study, we show that users indeed entrain on a “strategy-level” without the change of strategy impinging on their creativity. Our work thus presents a promising future research direction for developing dialogue management systems that can strategically influence people’s descriptive strategy to ease the system’s language understanding in creative tasks.

Numerous studies in social psychology have shown that familiarization across repeated interactions improves people's perception of the other. If and how these findings relate to human-robot interaction (HRI) is not well understood, even though such knowledge is crucial when pursuing long-term interactions. In our work, we investigate the persistence of first impressions by asking 49 participants to play a geography game with a robot. We measure how their perception of the robot changes over three sessions with three to ten days of zero exposure in between. Our results show that different perceptual dimensions stabilize within different time frames, with the robot's competence being the fastest to stabilize and perceived threat the most fluctuating over time. We also found evidence that perceptual differences between robots with varying levels of humanlikeness persist across repeated interactions. This study has important implications for HRI design as it sheds new light on the influence of robots' embodiment and interaction abilities. Moreover, it also impacts HRI theory as it presents novel findings contributing to research on the uncanny valley and robot perception in general.

Detecting, modeling, and making sense of multimodal data from human users in the wild still poses numerous challenges. Starting from aspects of data quality and reliability of our measurement instruments, the multidisciplinary endeavor of developing intelligent adaptive systems in human-computer or human-robot interaction (HCI, HRI) requires a broad range of expertise and more integrative efforts to make such systems reliable, engaging, and user-friendly. At the same time, the spectrum of applications for machine learning and modeling of multimodal data in the wild keeps expanding. From the classroom to the robot-assisted operation theatre, our workshop aims to support a vibrant exchange about current trends and methods in the field of modeling multimodal data in the wild.

This paper presents a multimodal corpus of 209 spoken game dialogues between a human and a remote-controlled artificial agent. The interactions involve people collaborating with the agent to identify countries on the world map as quickly as possible, which allows studying rapid and spontaneous dialogue with complex anaphoras, disfluent utterances and incorrect descriptions. The corpus consists of two parts: 8 hours of game interactions have been collected with a virtual unembodied agent online and 26.8 hours have been recorded with a physically embodied robot in a research lab. In addition to spoken audio recordings available for both parts, camera recordings and skeleton-, facial expression- and eye-gaze tracking data have been collected for the lab-based part of the corpus. In this paper, we introduce the pedagogical reference resolution game (RDG-Map) and the characteristics of the corpus collected. We also present an annotation scheme we developed in order to study the dialogue strategies utilized by the players. Based on a subset of 330 minutes of interactions annotated so far, we discuss initial insights into these strategies as well as the potential of the corpus for future research.

The present study aims to examine the prevalent notion that people entrain to the vocabulary of a dialogue system.Although previous research shows that people will replace their choice of words with simple substitutes, studies using more challenging substitutions are sparse.In this paper, we investigate whether people adapt their speech to the vocabulary of a dialogue system when the system's suggested words are not direct synonyms.32 participants played a geographythemed game with a remote-controlled agent and were primed by referencing strategies (rather than individual terms) introduced in follow-up questions.Our results suggest that context-appropriate substitutes support convergence and that the convergence has a lasting effect within a dialogue session if the system's wording is more consistent with the norms of the domain than the original wording of the speaker.

With an ever increasing demand for personal service robots and artificial assistants, companies, start-ups and researchers aim to better understand what makes robot platforms more likable. Some argue that increasing a robot's humanlikeness leads to a higher acceptability. Others, however, find that extremely humanlike robots are perceived as uncanny and are consequently often rejected by users. When investigating people's perception of robots, the focus of the related work lies almost solely on the first impression of these robots, often measured based on images or video clips of the robots alone. Little is known about whether these initial positive or negative feelings persist when giving people the chance to interact with the robot. In this paper, 48 participants were gradually exposed to the capabilities of a robot and their perception of it was tracked from their first impression to after playing a short interactive game with it. We found that initial uncanny feelings towards the robot were significantly decreased after getting to know it better, which further highlights the importance of using real interactive scenarios when studying people's perception of robots. In order to elicit uncanny feelings, we used the 3D blended embodiment Furhat and designed four different facial textures for it. Our work shows that a blended platform can cause different levels of discomfort towards it depending on the facial texture and may thus be an interesting tool for further research on the uncanny valley.

To feel novel and engaging over time it is critical for an autonomous agent to have a large corpus of potential responses. As the size and multi-domain nature of the corpus grows, however, traditional hand-authoring of dialogue content is no longer practical. While crowdsourcing can help to overcome the problem of scale, a diverse set of authors contributing independently to an agent's language can also introduce inconsistencies in expressed behavior. In terms of affect or mood, for example, incremental authoring can result in an agent who reacts calmly at one moment but impatiently moments later with no clear reason for the transition. In contrast, affect in natural conversation develops over time based on both the agent's personality and contextual triggers. To better achieve this dynamic, an autonomous agent needs to (a) have content and behavior available for different desired affective states and (b) be able to predict what affective state will be perceived by a person for a given behavior. In this proof-of-concept paper, we explore a way to elicit and evaluate affective behavior using crowdsourcing. We show that untrained crowd workers are able to author content for a broad variety of target affect states when given semi-situated narratives as prompts. We also demonstrate that it is possible to strategically combine multimodal affective behavior and voice content from the authored pieces using a predictive model of how the expressed behavior will be perceived.

As social robots increasingly enter people's lives, coherence of personality is an important challenge for longterm human-robot interactions. We extend an architecture that acquires dialog through crowdsourcing to author both verbal and non-verbal indicators of personality. We demonstrate the efficacy of the approach through a four-day study in which teams of participants interacted with a social robot expressing one of two personalities as the host of a competitive game. Results indicate that the system is able to elicit personality-driven language behaviors from the crowd in an incremental and ongoing way and produce a coherent expression of that personality during face-to-face interactions over time.

Understanding emotional states is a challenging task which frequently leads to misinterpretation even in human observers. While the perception of emotions has been studied extensively in human psychology, little is known about what factors influence the human perception of emotions in robots and virtual characters. In this paper, we build on the Brunswik lens model to investigate the influence of (a) the agent's embodiment using a 2D virtual character, a 3D blended embodiment, a recording of the 3D platform and a recording of a human, as well as (b) the level of human-likeness on people's ability to interpret emotional facial expressions in an agent. In addition, we measure social traits of the human observers and analyze how they correlate to the success in recognizing emotional expressions. We find that interpersonal differences play a minor role in the perception of emotional states. However, both embodiment and human-likeness as well as related perceptual dimensions such as perceived social presence and uncanniness have an effect on the attribution of emotional states.

Mimicry plays an important role in social interaction. In human communication, it is used to establish rapport and bonding both with other humans, as well as robots and virtual characters. However, little is known about the underlying factors that elicit mimicry in humans when interacting with a robot. In this work, we study the influence of embodiment on participants' ability to mimic a social character. Participants were asked to intentionally mimic the laughing behavior of the Furhat mixed embodied robotic head and a 2D virtual version of the same character. To explore the effect of embodiment, we present two novel approaches to automatically assess people's ability to mimic based solely on videos of their facial expressions. In contrast to participants' self-assessment, the analysis of video recordings suggests a better ability to mimic when people interact with the 2D embodiment.

Most research on the uncanny valley effect is concerned with the influence of unimodal visual cues in the dimensions human-likeness and realism as a trigger of an uncanny feeling in humans. This leads to a lack of investigation how multimodality affects the feeling of uncanniness. In our research project, we use the back-projected robot head Furhat to study the influence of multimodal cues in facial texture, expressions, voice and behaviour to broaden the understanding of the underlying cause of the uncanny valley effect. Up to date, we mainly investigated the general perception of uncanniness in a back-projected head, with a special focus on multimodal gender cues. In the upcoming years, the focus shall shift towards interaction strategies of social robots and their interplay with the robot's appearance.

This paper investigates the influence of multimodal incongruent gender cues on the perception of a robot's uncanniness and gender in children. The back-projected robot head Furhat was equipped with a female and male face texture and voice synthesizer and the voice and facial cues were tested in congruent and incongruent combinations. 106 children between the age of 8 and 13 participated in the study. Results show that multimodal incongruent cues do not trigger the feeling of uncanniness in children. These results are significant as they support other recent research showing that the perception of uncanniness cannot be triggered by a categorical ambiguity in the robot. In addition, we found that children rely on auditory cues much stronger than on the facial cues when assigning a gender to the robot if presented with incongruent cues. These findings have implications for the robot design, as it seems possible to change the gender of a robot by only changing its voice without creating a feeling of uncanniness in a child.

In this paper, we present and evaluate an approach to incremental dialogue act (DA) segmentation and classification.Our approach utilizes prosodic, lexico-syntactic and contextual features, and achieves an encouraging level of performance in offline corpus-based evaluation as well as in simulated human-agent dialogues.Our approach uses a pipeline of sequential processing steps, and we investigate the contribution of different processing steps to DA segmentation errors.We present our results using both existing and new metrics for DA segmentation.The incremental DA segmentation capability described here may help future systems to allow more natural speech from users and enable more natural patterns of interaction.

This paper introduces Eve, a highperformance agent that plays a fast-paced image matching game in a spoken dialogue with a human partner.The agent can be optimized and operated in three different modes of incremental speech processing that optionally include incremental speech recognition, language understanding, and dialogue policies.We present our framework for training and evaluating the agent's dialogue policies.In a user study involving 125 human participants, we evaluate three incremental architectures against each other and also compare their performance to human-human gameplay.Our study reveals that the most fully incremental agent achieves game scores that are comparable to those achieved in human-human gameplay, are higher than those achieved by partially and nonincremental versions, and are accompanied by improved user perceptions of efficiency, understanding of speech, and naturalness of interaction.