Virtual characters require the real-time streaming of verbal and nonverbal behaviors for the expression of dynamically generated humor. In this paper, we present the Live Link Animator, a real-time solution for multimodal animation of Unreal Engine characters using individual blendshapes. We demonstrate the tool through an example interaction with a MetaHuman character and outline potential areas of application in the domain of virtual agent humor research.

Humor often serves as a catalyst for smoother interpersonal communication, enhancing interaction experience between individuals.While virtual characters can also gain from these benefits, implementing humor naturally in human-character interactions remains an open challenge.In this paper, we propose the Joking and Multimodally Amusing Real-Time Character (J-MARC) system, combining a photorealistic character with advanced large language model (LLM) techniques to contextualize jokes within small talk.In the real-time interaction, the character is able to present the jokes multimodally and to apply nonverbal behavior while listening.

We conducted a within-subjects study to examine how realistic faces and cartoon faces on avatars affect communication, task satisfaction, sense of presence, and mood perception in mixed reality meetings. Over the course of two weeks, six groups of co-workers (14 people) held recurring meetings using Microsoft HoloLens2 devices, each person embodying a personal full-body avatar with either a realistic face or cartoon face. Half of the groups started with the realistic face avatar and switched to the cartoon face version halfway through (RC condition), and the other half with the cartoon-face avatar first (CR condition). Results showed that participants in the RC condition may have had higher expectations and more errors in perceiving their colleagues’ moods. Participants in the CR condition reported that the avatars’ appearance mattered less over time and experienced increased comfort and improved identification of their colleagues. Participants rated words, tone of voice, and movement as the most useful cues for perceiving colleagues’ moods, regardless of avatar rendering style. In the RC condition, participants rated gaze as more useful than facial expressions, while in the CR condition, both gaze and facial expressions were rated as the least useful. Results also suggested that participants had more errors when perceiving negative moods in their colleagues, with this trend appearing for most moods, but depending on conditions. Implications of these findings for mixed and virtual reality meetings are discussed. This work contributes to the field of remote collaboration by providing insights from longitudinal data on the impact of avatar appearance on various aspects of work meetings in virtual environments. • Realistic avatars may lead to higher expectations and errors in perceiving moods. • Cartoon avatars may increase the comfort and identification of colleagues over time. • For both avatars, words, voice, and movement were the most useful to perceive moods. • Avatar’s evaluation should be longitudinal as attitudes towards them change over time. • Avatar attitudes depend on context; evaluation should prioritise ecological validity. • Avatars’ identification, communication, and emotional trust may outweigh appearance.

People can understand how human interaction unfolds and can pinpoint social attitudes such as showing interest or social engagement with a conversational partner. However, summarising this with a set of rules is difficult, as our judgement is sometimes subtle and subconscious. Hence, it is challenging to program agents or non-player characters (NPCs) to react towards social signals appropriately, which is important for immersive narrative games in Virtual Reality (VR). We present a collaborative work between two game studios (Maze Theory and Dream Reality Interactive) and academia to develop an immersive machine learning (ML) pipeline for detecting social engagement. Here we introduce the motivation and the methodology of the immersive ML pipeline, then we cover the motivation for the industry-academia collaboration, how it progressed, the implications of joined work on the industry and reflective insights on the collaboration. Overall, we highlight the industry-academia collaborative work on an immersive ML pipeline for detecting social engagement. We demonstrate how creatives could use ML and VR to expand their ability to design more engaging commercial games.

Nonverbal cues have multiple roles in social encounters, with gaze behaviour facilitating interactions and conversational flow. In this work, we explore the conversation dynamics in dyadic settings in a free-flow discussion. Using automatic analysis (rather than manual labelling), we investigate how the gaze behaviour of one person is related to how much the other person changes their gaze (frequency in gaze change) and what their gaze target is (direct or avert gaze). Our results show that when one person is looked at they change their gaze direction with a higher frequency compared to when they are not looked at. They also tend to maintain a direct gaze to the other person when they are not looked at.