Virtual characters require the real-time streaming of verbal and nonverbal behaviors for the expression of dynamically generated humor. In this paper, we present the Live Link Animator, a real-time solution for multimodal animation of Unreal Engine characters using individual blendshapes. We demonstrate the tool through an example interaction with a MetaHuman character and outline potential areas of application in the domain of virtual agent humor research.

Humor often serves as a catalyst for smoother interpersonal communication, enhancing interaction experience between individuals.While virtual characters can also gain from these benefits, implementing humor naturally in human-character interactions remains an open challenge.In this paper, we propose the Joking and Multimodally Amusing Real-Time Character (J-MARC) system, combining a photorealistic character with advanced large language model (LLM) techniques to contextualize jokes within small talk.In the real-time interaction, the character is able to present the jokes multimodally and to apply nonverbal behavior while listening.

Humor is an important communicative skill in human interactions. Intelligent virtual agents can leverage it to increase their believability and overall interaction experience. In this paper, we focus on transferring and implementing existing multimodal irony markers from the literature to a photorealistic virtual character. The verbal content is generated dynamically by an irony generator. We demonstrate how the ironic turn can be augmented with prosodic and facial markers. An expressivity parameter allows us to manipulate the encoding of the irony style.

Music is a great way of supporting a story. It adds a new layer of affective information and as such substantially increases the listening experience in storytelling scenarios. However, in real-time settings, creating emotionally fitting music requires permanent adaptation to the story's mood. While methods to compose and modify music according to emotional states are widely explored, current research rarely uses those techniques in a real-time setting, where such accompanying background music still requires improvisation by human musicians. In this work, we introduce the Affective Bar Piano, a virtual agent that assesses the mood of a story in real time. At the same time, the agent adapts its play to mirror the sensed affect of a human storyteller. In the presented demonstration scenario, the virtual agent is embodied by a 3D piano character playing music in a Wild West saloon setting.

Non-verbal sounds are an essential communication channel for social robots. However, it requires expert knowledge to create and compose synthesizers, develop melodic structures or record samples which express a robot's internal intentions and emotions. This paper presents an approach for adapting a robot's timbre based on non-expert human comparative feedback in order to personalize the sonic interaction design to an individual user's preferences. An evolution strategy learns parameters of real-time sound synthesis for different intentions and emotions. Ultimately, the strategy aims to improve the perceived goodness of how well a specific melody's sound maps to a specific emotion or intention. In order to demonstrate the feasibility of the approach, we report on a user study with a robot, 6 exemplary melodies and 27 participants. Our study results show that the strategy indeed results in improved and preferred sound designs and that many participants are willing to apply such a process to improve their robots' expressivity.

A shared sense of humor can result in positive feelings associated with amusement, laughter, and moments of bonding. If robotic companions could acquire their human counterparts' sense of humor in an unobtrusive manner, they could improve their skills of engagement. In order to explore this assumption, we have developed a dynamic user modeling approach based on Reinforcement Learning, which allows a robot to analyze a person's reaction while it tells jokes and continuously adapts its sense of humor. We evaluated our approach in a test scenario with a Reeti robot acting as an entertainer and telling different types of jokes. The exemplary adaptation process is accomplished only by using the audience's vocal laughs and visual smiles, but no other form of explicit feedback. We report on results of a user study with 24 participants, comparing our approach to a baseline condition (with a non-learning version of the robot) and conclude by providing limitations and implications of our approach in detail.

When looking at Socially Interactive Robots, adaptation to the user's preferences plays an important role in today's Human-Robot Interaction to keep interaction interesting and engaging over a long period of time. Findings indicate an increase in user engagement for robots with adaptive behavior and personality, but also that it depends on the task context whether a similar or opposing robot personality is preferred. We present an approach based on Reinforcement Learning, which gets its reward directly from social signals in real-time during the interaction, to quickly learn about and dynamically address individual human preferences. Our scenario involves a Reeti robot in the role of a story teller talking about the main characters in the novel “Alice's Adventures in Wonderland” by generating descriptions with varying degree of introversion/extraversion. After initial simulation results, an interactive prototype is presented which allows to explore the learning process adapting to the human interaction partner's engagement.