Virtual characters require the real-time streaming of verbal and nonverbal behaviors for the expression of dynamically generated humor. In this paper, we present the Live Link Animator, a real-time solution for multimodal animation of Unreal Engine characters using individual blendshapes. We demonstrate the tool through an example interaction with a MetaHuman character and outline potential areas of application in the domain of virtual agent humor research.

Humor often serves as a catalyst for smoother interpersonal communication, enhancing interaction experience between individuals.While virtual characters can also gain from these benefits, implementing humor naturally in human-character interactions remains an open challenge.In this paper, we propose the Joking and Multimodally Amusing Real-Time Character (J-MARC) system, combining a photorealistic character with advanced large language model (LLM) techniques to contextualize jokes within small talk.In the real-time interaction, the character is able to present the jokes multimodally and to apply nonverbal behavior while listening.

As immersive technologies are increasingly being adopted by artists, dancers and developers in their creative work, there is a demand for tools and methods to design compelling ways of embodied interaction within virtual environments. Interactive Machine Learning allows creators to quickly and easily implement movement interaction in their applications by performing examples of movement to train a machine learning model. A key aspect of this training is providing appropriate movement data features for a machine learning model to accurately characterise the movement then recognise it from incoming data. We explore methodologies that aim to support creators’ understanding of movement feature data in relation to machine learning models and ask how these models hold the potential to inform creators’ understanding of their own movement. We propose a 5-day hackathon, bringing together artists, dancers and designers, to explore designing movement interaction and create prototypes using new interactive machine learning tool InteractML.

Interactive Machine Learning offers a method for designing movement interaction that supports creators in implementing even complex movement designs in their immersive applications by simply performing them with their bodies. We introduce a new tool, InteractML, and an accompanying ideation method, which makes movement interaction design faster, adaptable and accessible to creators of varying experience and backgrounds, such as artists, dancers and independent game developers. The tool is specifically tailored to non-experts as creators configure and train machine learning models via a node-based graph and VR interface, requiring minimal programming. We aim to democratise machine learning for movement interaction to be used in the development of a range of creative and immersive applications.

Videogame systems incorporate varied sensors to increase the range of player interactions and improve player experience. However, implementing robust recognisers for player actions with sensors presents significant challenges to developers. Further, sensor-based controls offer little player customisation compared to traditional input interfaces (gamepads, keyboards and joysticks). Past research on motion-driven music systems has successfully used interactive machine learning (IML) techniques to facilitate the development and customisation of sensor-based interfaces, both by developers and end users. However, existing standalone software tools for IML are not ideal for use in game development and distribution. In order to support more effective and flexible use of sensors by game developers and players, we developed an integrated IML solution for Unity3D in the form of a visual node system supporting classification, regression and time series analysis of sensor data.