Human-like nonverbal behaviors, such as gestures or facial expressions, play a crucial role in face-to-face communication, yet their impact on user interactions with virtual agents (VAs) in real-world environments remains underexplored. This study examines how co-speech gestures influence engagement and perception in interactions with a VA deployed in a museum. Using a mixed-methods approach, we conducted an A/B test comparing a gesturing VA to a nongesturing VA. Quantitative analysis of system logs revealed no significant difference in the number of conversational turns between conditions. Qualitative findings revealed that gestures may positively influence usability by making the interaction more intuitive and reducing user confusion regarding the VA’s internal states. However, perceptions of nonverbal behavior varied, with some users finding gestures engaging while others considered them uncanny. Regardless of condition, system response times, verbal communication errors, and answer quality were key drivers of user dissatisfaction. These findings highlight both the benefits and challenges of incorporating and studying nonverbal behavior in VAs and underscore the need for robust verbal communication to support meaningful interactions.

The rise of generative AI, particularly large language models (LLMs), prompts a re-examination of human-agent interaction (HAI), inviting us to imagine novel roles for agents in society. This paper explores this theme of “Interaction and Imagination” through a multi-stage exploratory study on enhancing an embodied conversational agent (ECA) for older adults’ health monitoring. First, to ground our work in user needs, we conducted a co-creation workshop with older users (N=3) who had extensive, real-world experience with a pre-existing, intent-based health ECA. The goal was to identify its core limitations (limited contextual interpretation; rigid dialog) and to let users envision the key features of an ideal successor. This revealed a strong desire for an ECA with greater interactional flexibility, contextual understanding of health data, and social engagement. Second, guided by these co-created requirements based on voiced limitations and desires, we developed an LLM-enhanced prototype featuring a hybrid rule-based and generative dialog model. Finally, to evaluate whether this successor addressed the initial limitations, we conducted an exploratory within-subjects mixed-methods comparative study (N=7) directly contrasting the new LLM-ECA against the original intent-based system. Results indicate a ‘double-edged sword’: While the LLM-ECA was perceived as more ‘organic’, it was not rated higher on overall anthropomorphism and presented significant challenges in intuitiveness and cognitive load, exacerbated by user interface (UI) limitations such as push-to-talk voice activation. This study highlights the critical tension between the imagined potential of fluid, AI-driven conversation and the practical realities of interaction for older adults. We provide early empirical evidence underscoring the necessity of hybrid systems that balance generative adaptability with guided interaction, alongside robust UI design and structured onboarding. Our work offers actionable considerations for designing LLM-ECAs that effectively bridge the gap between imaginative possibilities and impactful HAI in sensitive domains.

Socially interactive agents are gaining prominence in domains like healthcare, education, and service contexts, particularly virtual agents due to their inherent scalability. To facilitate authentic interactions, these systems require verbal and nonverbal communication through e.g., facial expressions and gestures. While natural language processing technologies have rapidly advanced, incorporating human-like nonverbal behavior into real-world interaction contexts is crucial for enhancing the success of communication, yet this area remains underexplored. One barrier is creating autonomous systems with sophisticated conversational abilities that integrate human-like nonverbal behavior. This paper presents a distributed architecture using Epic Games' MetaHuman, combined with advanced conversational AI and camera-based user management, that supports methods like motion capture, handcrafted animation, and generative approaches for nonverbal behavior. We share insights into a system architecture designed to investigate nonverbal behavior in socially interactive agents, deployed in a three-week field study in the Deutsches Museum Bonn, showcasing its potential in realistic nonverbal behavior research.

Using social robots and virtual agents (VAs) as interfaces for health monitoring systems for older adults offers the possibility of more engaging interactions that can support long-term health and well-being. While robots are characterized by their physical presence, software-based VAs are more scalable and flexible. Few comparisons of these interfaces exist in the human-robot and humanagent interaction domains, especially in long-term and real-world studies. In this work, we examined impressions of social robots and VAs at the beginning and end of an eight-week study in which older adults interacted with these systems independently in their homes. Using a between-subjects design, participants could choose which interface to evaluate during the study. While participants perceived the social robot as somewhat more likable, the VA was perceived as more intelligent. Our work provides a basis for further studies investigating factors most relevant for engaging interactions with social interfaces for long-term health monitoring.

Socially interactive agents (SIA) can be used to assist with information tasks in public spaces like museums or train stations. For the SIAs' learning and adaptability and the SIA operator, it is crucial to assess how users rate their satisfaction with the SIA to establish a foundation for potential modifications. A solution is needed to measure user satisfaction without interrupting robot interactions. One possibility is to gauge and analyze users' affective states. Based on a field test involving N=16 participants, a Furhat robot functioning as a fully autonomous service and information point, dyadic user interactions and satisfaction ratings were utilized to investigate how user affective states can be used to predict user satisfaction. The results indicate that predicting user satisfaction through affective state rating is possible. However, it is important that a significantly larger amount of interaction data, with increased class diversity, is necessary to make a more reliable statement.

This paper investigates the impact of a group-adaptive conversation design in two socially interactive agents (SIAs) through two real-world studies. Both SIAs – Furhat, a social robot, and MetaHuman, a virtual agent – were equipped with a conversational artificial intelligence (CAI) backend combining hybrid retrieval and generative models. The studies were carried out in an in-the-wild setting with a total of N = 188 participants who interacted with the SIAs - in dyads, triads or larger groups – at a German museum. Although the results did not reveal a significant effect of the group-sensitive conversation design on perceived satisfaction, the findings provide valuable insights into the challenges of adapting CAI for multi-party interactions and across different embodiments (robot vs. virtual agent) highlighting the need for multimodal strategies beyond linguistic pluralization. These insights contribute to the fields of Human-Agent Interaction (HAI), Human-Robot Interaction (HRI), and broader Human-Machine Interaction (HMI), providing insights for future research on effective dialogue adaptation in group settings.

The social capabilities of socially interactive agents (SIA) are a key to successful and smooth interactions between the user and the SIA. A successful start of the interaction is one of the essential factors for satisfying SIA interactions. For a service and information task in which the SIA helps with information, e.g. about the location, it is an important skill to master the opening of the conversation and to recognize which interlocutor opens the conversation and when. We are therefore investigating the extent to which the opening of the conversation can be trained using the user’s body language as an input for machine learning to ensure smooth conversation starts for the interaction. In this paper we propose the Interaction Initiation System (IIS) which we developed, trained and validated using an in-the-wild data set. In a field test at the Deutsches Museum Bonn, a Furhat robot from Furhat Robotics was used as a service and information point. Over the period of use we collected the data of N = 201 single user interactions for the training of the algorithms. We can show that the IIS, achieves a performance that allows the conclusion that this system is able to determine the greeting period and the opener of the interaction.

Stroke-induced aphasia, an acquired speech impairment, poses significant challenges to individuals’ communication abilities. Our robot-assisted speech training app aims to facilitate home-based, self-administered training for individuals with aphasia to complement their speech therapy. We evaluated our app in a single-session study with four individuals with aphasia in a rehabilitation setting. Each participant had a different severity grade of the condition, including one case of global aphasia. Our findings suggest an overall positive user experience, with indications that the training facilitated by the robot is suitable for individuals across all severity grades of aphasia, though exercise customization is crucial. Participants, even those with limited technical experience, adapted to using the system by themselves quickly. However, usability issues specific to this diverse target group were noted, such as the length of explanations and the abundant use of robot gestures, which will be addressed in future iterations.

In this work, we introduce the adaptation of ego network analysis (ENA) as a methodology to investigate group-robot interactions (GRI) in public spaces. Based on a field study with 16 GRI, our research explores the complexities of group dynamics and offers a reflection on the use of ENA to assess these interactions. Our results contribute to a nuanced understanding of how group structures influence the perception of social robots in GRI. The insights on the use of ENA in human-robot interaction (HRI) lay the foundation for strengthening methodologies and advancing our understanding of social dynamics in HRI.

Social robots and virtual agents can provide a low-threshold social interface for the independent self-health monitoring of older adults, thereby supporting their self-care and empowering them to take charge of their own health. In this work, we present a study on the interface preference of 35 older adults for a health monitoring system which they will be evaluating for eight weeks in their own homes. We found that participants who prefer the social robot have a higher affinity for technology interaction (ATI) score and are more likely to use assistance systems in their daily lives, while participants with a slightly lower ATI score prefer the virtual agent. Participants cited more personality and an interest in robots as reasons for the robot preference, and space and flexibility as reasons to prefer the virtual agent. These results underscore the importance of a personalized introduction of social technologies for health monitoring into the daily lives of older adults.

Evaluating the users’ subjective impression of social robots with questionnaires in field trials is time-consuming, and it is impossible to analyze every interaction as users cannot always participate for various reasons. In social robotics, social signals (e.g., gestures, facial expressions, body language) can be used to control and adapt the robot’s behavior to improve interaction. Social signals should allow a statement to be made about how the user feels in a situation or what attitude or opinion they have about something. This paper focuses on finding out to what extent social signals can contribute to an automatic evaluation of the subjective user satisfaction without the need for user questionnaires and solely through social signals. For this purpose, it is first relevant whether social signals occur and if they can be used in the present use case. For this purpose, a field test was carried out in the entrance area of a city administration, in which the social robot Furhat was used as a point for service and information. Four interactions were recorded on video with the consent of the users, and a post-questionnaire on satisfaction was collected for each case. A qualitative video analysis was used to examine the interactions concerning potential social signals displayed by the users. The results suggest that the direction of view and the head orientation are the body parts that are mostly moved in the interaction. All users interacted with the system with relatively little movement of the rest of the body. The results suggest that it is not recommended to focus only on social signals alone.

Aphasia is a common symptom of stroke. Patients affected may experience difficulties in all aspects of language. To complement traditional logopedic therapy, we developed a speech training application for a social robot, allowing affected individuals to perform additional training independently. In an early evaluation, a representative of each main stakeholder group we identified - namely persons with aphasia, care staff, and speech therapists - evaluated our application in terms of perceived usefulness, ease of use, and overall user experience. The robot guided the participants through the training session autonomously and most exercises were completed without help, thus proving the feasibility of our concept. The participants rated the application overall as positive and we achieved promising results in terms of attitude towards and intention to use the system. Furthermore, the social component of the training with the robot was very well received among the participants. In the future, the training content will be revised with the help of a linguist, to adequately support the training needs of persons with aphasia.

Social robots are increasingly used in public spaces, but they still struggle to handle complex social dynamics and interactions with multiple users. This study assessed the interaction between visitors and a Furhat robot connected to an artificial intelligence (AI)-based dialogue system in an oceanographic museum. Our findings from a video analysis of 176 interactions highlight the importance of understanding social dynamics in social robotics research. At the same time, it suggests that social robots are still limited in their ability to handle multi-party interactions (MPI) and understand complex social dynamics. Those limitations are due, in part, to technical constraints, as well as the need for further research on the social and cultural factors that shape human-robot interactions.

Current research on the impact of gender appearance in virtual agents and social robots highlights the danger of transmitting and solidifying existing gender stereotypes. To investigate gender-stereotyped interaction at public spaces in dependency of virtual agents gender, we varied the gender of a virtual agent at a metro station. We used an ethnographic study approach, combining a two-day behavior observation with semi-structured interviews with descriptive and qualitative system log analysis of four weeks. Our results show that topics of conversation differ in dependency of the virtual agents gender: the male virtual agent was asked about topics such as brothels, drugs and alcohol and insulted frequently, while the female one was asked for relationship status or about flirting.

The evolution of Virtual and Mixed Reality worlds is closely related to the development of digitalization and increasing technologies - aspects that capture the changes marking the twenty-first century. With the emergence of studies showing that the use of Virtual and Mixed Reality elements in learning processes positively influence the students' motivation and their control of learning-outcomes, these technologies are becoming more and more popular practices in education and learning contexts. Nevertheless, studies about the students' perspective in Germany are rarely conducted. Therefore, a status and requirement analysis of the students' demands on Mixed Reality in higher education has been developed and conducted. Questions of research are composed of the students' general attitude towards Mixed Reality in higher education and the influence of user factors on their attitude. Furthermore, the study covers the participants' perception of the current usage of Mixed Reality in higher education. First results show that there is a general positive attitude towards Mixed Reality. Especially the innovative character of Mixed Reality is a strong benefit for students. The younger the students are the stronger their requirements for Mixed Reality in higher education are.

The ongoing modernization of today's workplaces within the framework of industry 4.0 is leading to radical new developments and forcing fundamental changes to the way companies and factories are established. The purpose of this paper is to develop and improve a concept that organizes the workplace on a basic level in order to meet the upcoming challenge of accounting for robots as key players in it. There are several organizational structures that include the traditional hierarchical structures as well as the modern agile management models that are rising in the world of industry 4.0. Within it, the usage of robots is increasing due to the vast and multiple functions each robot is able to perform. This implies automation, but it also implies that different forms of collaboration with humans are becoming necessary as factories are tending to produce customized products. Individualized products require different methods and different skills which span different sets of abilities of humans and robots. How does this increase of robots' usage affect the applied organization structures and flows of the decision-making processes? What are necessary changes that need to be considered to maximize the efficiency and effectiveness of a workplace with robots that already existing structures do not compensate for? This paper highlights the existing gap and addresses the required changes in the organization models. Following that is an evaluation and outlook on that validates the legitimacy of the approach.

The term "Industry 4.0" symbolizes new forms of technology and artificial intelligence within production technologies. Smart robots are going to be the game changers within the factories of the future and will work with humans in indispensable teams within many processes. With this fourth industrial revolution, classical production lines are going through comprehensive modernization, e.g. in terms of in-the-box manufacturing, where humans and machines work side by side in so-called "hybrid teams". Questions about how to prepare for newly needed engineering competencies for the age of Industry 4.0, how to assess them and how to teach and train e.g. human-robot-teams have to be tackled in future engineering education. The paper presents theoretical aspects and empirical results of a series of studies, carried out to investigate the competencies of virtual collaboration and joint problem solving in virtual worlds.

Today there is a flood of e-learning and e-learning related solutions for engineering education. It is at least a time consuming task for a teacher to find an e-learning system, which matches their requirements. To assist teachers with this information overload, a web-based recommendation system for related e-learning solutions is under development to support teachers in the field of engineering education to find a matching e-learning system within minutes. Because the e-learning market is subject of very fast changes, an agile engineering process is used to ensure the capability to react on these changes. To solve the challenges of this project, an own user-flow visual programming language and an algorithm are under development. A special software stack is chosen to accelerate the development. Instead of classical back-office software to administer and maintain the project, a web-based approach is used - even for a complex editor. The determining of the necessary catalog of related solutions within real-time is based on big data technologies, data mining methods and statistically text analysis.

Engineering students are, compared to their counterparts in other disciplines, less mobile resulting in limited intercultural skills. Globalization requires professionals being excellent in their fields and being able to work on a global scale at the same time. So far, engineering education has put too little stress on integrating intercultural competences into curricula. This paper shows new approaches to incorporate international experiences into higher engineering education. First, it analyzes the current situation of international student mobility in Germany, before emphasizing the general motivation for international student exchange especially in engineering science. A consortium of three excellent German engineering universities was put up to introduce new measures for increasing student mobility as is described subsequently. This paper represents work in progress. Thus, further results will be published continuously.