Socially interactive agents are gaining prominence in domains like healthcare, education, and service contexts, particularly virtual agents due to their inherent scalability. To facilitate authentic interactions, these systems require verbal and nonverbal communication through e.g., facial expressions and gestures. While natural language processing technologies have rapidly advanced, incorporating human-like nonverbal behavior into real-world interaction contexts is crucial for enhancing the success of communication, yet this area remains underexplored. One barrier is creating autonomous systems with sophisticated conversational abilities that integrate human-like nonverbal behavior. This paper presents a distributed architecture using Epic Games' MetaHuman, combined with advanced conversational AI and camera-based user management, that supports methods like motion capture, handcrafted animation, and generative approaches for nonverbal behavior. We share insights into a system architecture designed to investigate nonverbal behavior in socially interactive agents, deployed in a three-week field study in the Deutsches Museum Bonn, showcasing its potential in realistic nonverbal behavior research.

Socially interactive agents (SIA) can be used to assist with information tasks in public spaces like museums or train stations. For the SIAs' learning and adaptability and the SIA operator, it is crucial to assess how users rate their satisfaction with the SIA to establish a foundation for potential modifications. A solution is needed to measure user satisfaction without interrupting robot interactions. One possibility is to gauge and analyze users' affective states. Based on a field test involving N=16 participants, a Furhat robot functioning as a fully autonomous service and information point, dyadic user interactions and satisfaction ratings were utilized to investigate how user affective states can be used to predict user satisfaction. The results indicate that predicting user satisfaction through affective state rating is possible. However, it is important that a significantly larger amount of interaction data, with increased class diversity, is necessary to make a more reliable statement.

The social capabilities of socially interactive agents (SIA) are a key to successful and smooth interactions between the user and the SIA. A successful start of the interaction is one of the essential factors for satisfying SIA interactions. For a service and information task in which the SIA helps with information, e.g. about the location, it is an important skill to master the opening of the conversation and to recognize which interlocutor opens the conversation and when. We are therefore investigating the extent to which the opening of the conversation can be trained using the user’s body language as an input for machine learning to ensure smooth conversation starts for the interaction. In this paper we propose the Interaction Initiation System (IIS) which we developed, trained and validated using an in-the-wild data set. In a field test at the Deutsches Museum Bonn, a Furhat robot from Furhat Robotics was used as a service and information point. Over the period of use we collected the data of N = 201 single user interactions for the training of the algorithms. We can show that the IIS, achieves a performance that allows the conclusion that this system is able to determine the greeting period and the opener of the interaction.

Evaluating the users’ subjective impression of social robots with questionnaires in field trials is time-consuming, and it is impossible to analyze every interaction as users cannot always participate for various reasons. In social robotics, social signals (e.g., gestures, facial expressions, body language) can be used to control and adapt the robot’s behavior to improve interaction. Social signals should allow a statement to be made about how the user feels in a situation or what attitude or opinion they have about something. This paper focuses on finding out to what extent social signals can contribute to an automatic evaluation of the subjective user satisfaction without the need for user questionnaires and solely through social signals. For this purpose, it is first relevant whether social signals occur and if they can be used in the present use case. For this purpose, a field test was carried out in the entrance area of a city administration, in which the social robot Furhat was used as a point for service and information. Four interactions were recorded on video with the consent of the users, and a post-questionnaire on satisfaction was collected for each case. A qualitative video analysis was used to examine the interactions concerning potential social signals displayed by the users. The results suggest that the direction of view and the head orientation are the body parts that are mostly moved in the interaction. All users interacted with the system with relatively little movement of the rest of the body. The results suggest that it is not recommended to focus only on social signals alone.