Human-like nonverbal behaviors, such as gestures or facial expressions, play a crucial role in face-to-face communication, yet their impact on user interactions with virtual agents (VAs) in real-world environments remains underexplored. This study examines how co-speech gestures influence engagement and perception in interactions with a VA deployed in a museum. Using a mixed-methods approach, we conducted an A/B test comparing a gesturing VA to a nongesturing VA. Quantitative analysis of system logs revealed no significant difference in the number of conversational turns between conditions. Qualitative findings revealed that gestures may positively influence usability by making the interaction more intuitive and reducing user confusion regarding the VA’s internal states. However, perceptions of nonverbal behavior varied, with some users finding gestures engaging while others considered them uncanny. Regardless of condition, system response times, verbal communication errors, and answer quality were key drivers of user dissatisfaction. These findings highlight both the benefits and challenges of incorporating and studying nonverbal behavior in VAs and underscore the need for robust verbal communication to support meaningful interactions.

Socially interactive agents are gaining prominence in domains like healthcare, education, and service contexts, particularly virtual agents due to their inherent scalability. To facilitate authentic interactions, these systems require verbal and nonverbal communication through e.g., facial expressions and gestures. While natural language processing technologies have rapidly advanced, incorporating human-like nonverbal behavior into real-world interaction contexts is crucial for enhancing the success of communication, yet this area remains underexplored. One barrier is creating autonomous systems with sophisticated conversational abilities that integrate human-like nonverbal behavior. This paper presents a distributed architecture using Epic Games' MetaHuman, combined with advanced conversational AI and camera-based user management, that supports methods like motion capture, handcrafted animation, and generative approaches for nonverbal behavior. We share insights into a system architecture designed to investigate nonverbal behavior in socially interactive agents, deployed in a three-week field study in the Deutsches Museum Bonn, showcasing its potential in realistic nonverbal behavior research.

This paper investigates the impact of a group-adaptive conversation design in two socially interactive agents (SIAs) through two real-world studies. Both SIAs – Furhat, a social robot, and MetaHuman, a virtual agent – were equipped with a conversational artificial intelligence (CAI) backend combining hybrid retrieval and generative models. The studies were carried out in an in-the-wild setting with a total of N = 188 participants who interacted with the SIAs - in dyads, triads or larger groups – at a German museum. Although the results did not reveal a significant effect of the group-sensitive conversation design on perceived satisfaction, the findings provide valuable insights into the challenges of adapting CAI for multi-party interactions and across different embodiments (robot vs. virtual agent) highlighting the need for multimodal strategies beyond linguistic pluralization. These insights contribute to the fields of Human-Agent Interaction (HAI), Human-Robot Interaction (HRI), and broader Human-Machine Interaction (HMI), providing insights for future research on effective dialogue adaptation in group settings.

In this work, we introduce the adaptation of ego network analysis (ENA) as a methodology to investigate group-robot interactions (GRI) in public spaces. Based on a field study with 16 GRI, our research explores the complexities of group dynamics and offers a reflection on the use of ENA to assess these interactions. Our results contribute to a nuanced understanding of how group structures influence the perception of social robots in GRI. The insights on the use of ENA in human-robot interaction (HRI) lay the foundation for strengthening methodologies and advancing our understanding of social dynamics in HRI.

Social robots are increasingly used in public spaces, but they still struggle to handle complex social dynamics and interactions with multiple users. This study assessed the interaction between visitors and a Furhat robot connected to an artificial intelligence (AI)-based dialogue system in an oceanographic museum. Our findings from a video analysis of 176 interactions highlight the importance of understanding social dynamics in social robotics research. At the same time, it suggests that social robots are still limited in their ability to handle multi-party interactions (MPI) and understand complex social dynamics. Those limitations are due, in part, to technical constraints, as well as the need for further research on the social and cultural factors that shape human-robot interactions.

Current research on the impact of gender appearance in virtual agents and social robots highlights the danger of transmitting and solidifying existing gender stereotypes. To investigate gender-stereotyped interaction at public spaces in dependency of virtual agents gender, we varied the gender of a virtual agent at a metro station. We used an ethnographic study approach, combining a two-day behavior observation with semi-structured interviews with descriptive and qualitative system log analysis of four weeks. Our results show that topics of conversation differ in dependency of the virtual agents gender: the male virtual agent was asked about topics such as brothels, drugs and alcohol and insulted frequently, while the female one was asked for relationship status or about flirting.