Human-like nonverbal behaviors, such as gestures or facial expressions, play a crucial role in face-to-face communication, yet their impact on user interactions with virtual agents (VAs) in real-world environments remains underexplored. This study examines how co-speech gestures influence engagement and perception in interactions with a VA deployed in a museum. Using a mixed-methods approach, we conducted an A/B test comparing a gesturing VA to a nongesturing VA. Quantitative analysis of system logs revealed no significant difference in the number of conversational turns between conditions. Qualitative findings revealed that gestures may positively influence usability by making the interaction more intuitive and reducing user confusion regarding the VA’s internal states. However, perceptions of nonverbal behavior varied, with some users finding gestures engaging while others considered them uncanny. Regardless of condition, system response times, verbal communication errors, and answer quality were key drivers of user dissatisfaction. These findings highlight both the benefits and challenges of incorporating and studying nonverbal behavior in VAs and underscore the need for robust verbal communication to support meaningful interactions.

Socially interactive agents are gaining prominence in domains like healthcare, education, and service contexts, particularly virtual agents due to their inherent scalability. To facilitate authentic interactions, these systems require verbal and nonverbal communication through e.g., facial expressions and gestures. While natural language processing technologies have rapidly advanced, incorporating human-like nonverbal behavior into real-world interaction contexts is crucial for enhancing the success of communication, yet this area remains underexplored. One barrier is creating autonomous systems with sophisticated conversational abilities that integrate human-like nonverbal behavior. This paper presents a distributed architecture using Epic Games' MetaHuman, combined with advanced conversational AI and camera-based user management, that supports methods like motion capture, handcrafted animation, and generative approaches for nonverbal behavior. We share insights into a system architecture designed to investigate nonverbal behavior in socially interactive agents, deployed in a three-week field study in the Deutsches Museum Bonn, showcasing its potential in realistic nonverbal behavior research.

Socially interactive agents (SIA) can be used to assist with information tasks in public spaces like museums or train stations. For the SIAs' learning and adaptability and the SIA operator, it is crucial to assess how users rate their satisfaction with the SIA to establish a foundation for potential modifications. A solution is needed to measure user satisfaction without interrupting robot interactions. One possibility is to gauge and analyze users' affective states. Based on a field test involving N=16 participants, a Furhat robot functioning as a fully autonomous service and information point, dyadic user interactions and satisfaction ratings were utilized to investigate how user affective states can be used to predict user satisfaction. The results indicate that predicting user satisfaction through affective state rating is possible. However, it is important that a significantly larger amount of interaction data, with increased class diversity, is necessary to make a more reliable statement.