Expressive facial animation depends on models that can convey subtle, context‑dependent emotions. Procedural methods often rely on manually-tuned heuristics, while data‑driven techniques are constrained by the diversity of their training data. This paper explores the zero-shot potential of large language models (LLMs) to generate facial animations. Using the cognitively-grounded Ortony-Clore-Collins (OCC) model as a framework, we designed 110 text-based scenarios and evaluated the ability of different LLMs – Gemini-2.5 Pro, GPT-4o, and Llama 3.1-8b – to generate corresponding facial animations using the Facial Action Coding System (FACS). A perceptual user study confirmed that animations from Gemini-2.5 Pro were highly recognizable, with participants successfully matching facial expressions to the correct context at rates significantly above chance. A quantitative analysis of the generated Action Units (AUs) indicated both consistency within each OCC emotion category and diversity across different scenarios. Further analysis revealed that the emotional expressions group into six clusters: happiness, sadness, anger/disgust, fear/surprise, shame, and neutral. This work demonstrates a viable, lightweight pipeline connecting textual narrative directly to motion generation without requiring custom training or large-scale motion capture datasets.

Designing games with branching story lines, object annotations, scene details, and dialog can be challenging due to the intensive authoring required. We investigate the potential for authoring open-ended behaviors for point-and-click narrative games using GPT-3.5, a large language model. In our approach, we extend a behavior tree scripting system with nodes that query GPT-3.5 to generate object descriptions, conversations with characters, and responses to player actions. GPT-3.5 is used to generate content when it hasn’t been scripted manually and to update game state by asking questions about whether a player’s input achieves a particular game goal. We demonstrate our approach with puzzles based on scenes from an episode of Star Trek Voyager. Our approach aims to blend a specific plot with open-ended story elements while keeping the authoring work minimal. Based on a pilot study of 16 participants and our own testing, we find that the generated responses have high coherency and show signs of humor and novelty, but that utterances could be improved to be more interesting and better support the designer’s intent.

In this work, we describe the Curation Tree framework that has matured as part of our game-based research and teaching. We use this framework to manage the top-level experience of our players via a text-based, authorable behavior tree system that is easy to implement, test, and re-use. We present three case studies based on Unity games built within our framework: the first involves virtual reality games involving grasping objects with the hands; the second, Treatment X, uses the framework to implement a causal learning game; and the last involves narrative games based on a large language model. Based on our experiences with this system, we reflect on its limitations and make several recommendations based on its strengths, namely to centralize game logic with behaviors, to centralize user event handling, to prefer simple behaviors over complex ones, to use text-based scripts for authoring with version control, and to separate behaviors from persistent state.

In this work, we investigate three control strategies – joystick, laser, and tilt – for playing a platform game in mobile augmented reality. We analyze these strategies using both objective game metrics as well as self-reported measures of autonomy, competence, and intuitiveness. We found no significant differences in self-reported autonomy and competence between our three controllers, despite clear differences in duration needed to play, clear differences in the amount of movement of the character, and significant differences in intuitiveness ratings. All controllers were effective for playing the game. However, despite the joystick’s faster completion times and higher intuitiveness ratings, half of our players still chose our laser or tilt controller as their favorite. These results were consistent even with different difficulties of game level.

Detailed hand motions play an important role in face-to-face communication to emphasize points, describe objects, clarify concepts, or replace words altogether. While shared virtual reality (VR) spaces are becoming more popular, these spaces do not, in most cases, capture and display accurate hand motions. In this article, we investigate the consequences of such errors in hand and finger motions on comprehension, character perception, social presence, and user comfort. We conduct three perceptual experiments where participants guess words and movie titles based on motion captured movements. We introduce errors and alterations to the hand movements and apply techniques to synthesize or correct hand motions. We collect data from more than 1000 Amazon Mechanical Turk participants in two large experiments, and conduct a third experiment in VR. As results might differ depending on the virtual character used, we investigate all effects on two virtual characters of different levels of realism. We furthermore investigate the effects of clip length in our experiments. Amongst other results, we show that the absence of finger motion significantly reduces comprehension and negatively affects people’s perception of a virtual character and their social presence. Adding some hand motions, even random ones, does attenuate some of these effects when it comes to the perception of the virtual character or social presence, but it does not necessarily improve comprehension. Slightly inaccurate or erroneous hand motions are sufficient to achieve the same level of comprehension as with accurate hand motions. They might however still affect the viewers’ impression of a character. Finally, jittering hand motions should be avoided as they significantly decrease user comfort.

A primary goal of the Virtual Reality ( VR ) community is to build fully immersive and presence-inducing environments with seamless and natural interactions. To reach this goal, researchers are investigating how to best directly use our hands to interact with a virtual environment using hand tracking. Most studies in this field require participants to perform repetitive tasks. In this article, we investigate if results of such studies translate into a real application and game-like experience. We designed a virtual escape room in which participants interact with various objects to gather clues and complete puzzles. In a between-subjects study, we examine the effects of two input modalities (controllers vs. hand tracking) and two grasping visualizations (continuously tracked hands vs. virtual hands that disappear when grasping) on ownership, realism, efficiency, enjoyment, and presence. Our results show that ownership, realism, enjoyment, and presence increased when using hand tracking compared to controllers. Visualizing the tracked hands during grasps leads to higher ratings in one of our ownership questions and one of our enjoyment questions compared to having the virtual hands disappear during grasps as is common in many applications. We also confirm some of the main results of two studies that have a repetitive design in a more realistic gaming scenario that might be closer to a typical user experience.

In this work, we investigate the influence of different visualizations on a manipulation task in virtual reality (VR). Without the haptic feedback of the real world, grasping in VR might result in intersections with virtual objects. As people are highly sensitive when it comes to perceiving collisions, it might look more appealing to avoid intersections and visualize non-colliding hand motions. However, correcting the position of the hand or fingers results in a visual-proprioceptive discrepancy and must be used with caution. Furthermore, the lack of haptic feedback in the virtual world might result in slower actions as a user might not know exactly when a grasp has occurred. This reduced performance could be remediated with adequate visual feedback.

Augmented reality (AR) gaming is becoming widely available thanks to improvements in hand-held devices such as phones and tablets. In this work, we describe our system for generating levels for the AR game, Q*bird. In Q*bird, the player must visit every cell in the level while avoiding bees and cannon balls, similarly to the 1982 arcade game, Q*bert. To create a new level, designers place game elements using virtual cards. The system then generates the remainder of the level, ensuring that it’s navigable. Designers can edit these levels by dragging and dropping the created geometry. To test, the designer can drop a character into the level and play it. This system aids playtesting and level design by allowing levels to be quickly specified and tested in the same environment in which the game is played. Furthermore, this system offers an example of how the design of AR levels can also be performed in AR.

Most commercial virtual reality applications with self avatars provide users with a “one-size fits all” avatar. While the height of this body may be scaled to the user's height, other body proportions, such as limb length and hand size, are rarely customized to fit an individual user. Prior research has shown that mismatches between users' avatars and their actual bodies can affect size perception and feelings of body ownership. In this paper, we consider how concepts related to the virtual hand illusion, user experience, and task efficiency are influenced by variations between the size of a user's actual hand and their avatar's hand. We also consider how using a tracked controller or tracked gestures affect these concepts. We conducted a 2×3 within-subjects study (n=20), with two levels of input modality: using tracked finger motion vs. a hand-held controller (Glove vs. Controller), and three levels of hand scaling (Small, Fit, and Large). Participants completed 2 block-assembly trials for each condition (for a total of 12 trials). Time, mistakes, and a user experience survey were recorded for each trial. Participants experienced stronger feelings of ownership and realism in the Glove condition. Efficiency was higher in the Controller condition and supported by play data of more time spent, blocks grabbed, and blocks dropped in the Glove condition. We did not find enough evidence for a change in agency and the intensity of the virtual hand illusion depending on hand size. Over half of the participants indicated preferring the Glove condition over the Controller condition, mentioning fun and efficiency as factors in their choices. Preferences on hand scaling were mixed but often attributed to efficiency. Participants liked the appearance of their virtual hand more while using the Fit instead of Large hands. Several interaction effects were observed between input modality and hand scaling, for example, for smaller hands, tracked hands evoked stronger feelings of ownership compared to using a controller. Our results show that the virtual hand illusion is stronger when participants are able to control a hand directly rather than with a hand-held device, and that the virtual reality task must first be considered to determine which modality and hand size are the most applicable.

Abstract Real‐time animation controllers are fundamental for animating characters in response to player input in digital games. However, the design of such controllers requires making trade‐offs between the naturalness of the character's motions and the promptness of the character's response. In this paper, we investigate the effects of such trade‐offs on the players' enjoyment, control, satisfaction, and opinion of the character in a simple platform game. In our first experiment, we compare three controllers having the same responsiveness, but varying levels of naturalness. In the second experiment, we compare three controllers having increasing realism at the expense of responsiveness. In our third experiment, we keep motion naturalness and responsiveness constant but vary the avatar. Not surprisingly, our least responsive controller negatively affects players' performance and perceived ability to control the character. However, we also find that players are most satisfied with their own performance using our least natural controller, that differences in animation can significantly alter players' enjoyment with responsiveness being equal, that players do not report increased motion quality with our most natural controller, although viewers outside of the game context do, and that the avatar model affected the perception of the character but not players' enjoyment or perceived realism.

Response lag in digital games is known to negatively affect a player's game experience. Particularly with networked multiplayer games, where lag is typically unavoidable, the impact of delays needs to be well understood so that its effects can be mitigated. In this paper, we investigate two aspects of lag independently: latency (constant delay) and jitter (varying delay). We evaluate how latency and jitter each affect a player's enjoyment, frustration, performance, and experience as well as the extent to which players can adjust to such delays after a few minutes of gameplay. We focus on a platform game where the player controls a virtual character through a world. We find that delays up to 300ms do not impact the players' experience as long as they are constant. When jitter was added to a delay of 200ms, however, the lag was noticed by participants more often, hindered players' ability to improve with practice, increased how often they failed to reach the goal of the game, and reduced the perceived motion quality of the character.

Viewers effortlessly decouple action from style for human motion. Regardless of whether style refers to the subtle differences between individuals (John Wayne's walk versus Charlie Chaplin's walk) or to the manner in which the same action is expressed (such as a sad walk versus a nervous walk), the core intent of an action is readily recognizable.

Real-time animation controllers are fundamental for animating characters in response to player input. However, the design of such controllers requires making trade-offs between the naturalness of the character's motions and the promptness of the character's response. In this paper, we investigate the effects of such tradeoffs on the players' enjoyment, control, satisfaction, and opinion of the character in a simple platform game. In our first experiment, we compare three controllers having the same responsiveness, but varying levels of naturalness. In the second experiment, we compare three controllers having increasing realism but at the expense of decreased responsiveness. Not surprisingly, our least responsive controller negatively affects players' performance and perceived ability to control the character. However, we also find that players are most satisfied with their own performance using our least natural controller, in which the character moves around the environment in a static pose; that differences in animation can significantly alter players' enjoyment with responsiveness being equal; and that players do not report increased motion quality with our most natural controller, despite viewers outside of a game context rating the same controller as significantly more natural than our other conditions.

Crowd activities are often randomized to create the appearance of heterogeneity. However, the parameters that control randomization are frequently hard to tune because it is unclear how changes at the character level affect the high-level appearance of the crowd. We propose a method for computing randomization parameters that supports direct animator control. Given details about the environment, available activities, timing information and the desired high-level appearance of the crowd, we model the problem as a graph, formulate a convex optimization problem, and solve for a set of stochastic transition rates which satisfy the constraints. Unlike the use of heuristics for adding randomness to crowd activities, our approach provides guarantees on convergence to the desired result, allows for decentralized simulation, and supports a variety of constraints. In addition, because the rates can be pre-computed, no additional runtime processing is needed during simulation.

Motion capture remains a popular and widely-used method for animating virtual characters. However, all practical applications of motion capture rely on motion editing techniques to increase the reusability and flexibility of captured motions. Because humans are proficient in detecting and interpreting subtle details in human motion, understanding the perceptual consequences of motion editing is essential. Thus in this work, we perform three experiments to gain a better understanding of how motion editing might affect the emotional content of a captured performance, particularly changes in posture and dynamics, two factors shown to be important perceptual indicators of bodily emotions. In these studies, we analyse the properties (angles and velocities) and perception (recognition rates and perceived intensities) of a varied set of full-body motion clips representing the six emotions anger, disgust, fear, happiness, sadness, and surprise. We have found that emotions are mostly conveyed through the upper body, that the perceived intensity of an emotion can be reduced by blending with a neutral motion, and that posture changes can alter the perceived emotion but subtle changes in dynamics only alter the intensity.

Adventure role playing games (RPGs) provide players with increasingly expansive worlds, compelling storylines, and meaningful fictional character interactions. Despite the fast-growing richness of these worlds, the majority of interactions between the player and non-player characters (NPCs) still remain scripted. In this paper we propose using an NPC's animations to reflect how they feel towards the player and as a proof of concept, investigate the potential for a straightforward gaze model to convey trust. Through two perceptual experiments, we find that viewers can distinguish between high and low trust animations, that viewers associate the gaze differences specifically with trust and not with an unrelated attitude (aggression), and that the effect can hold for different facial expressions and scene contexts, even when viewed by participants for a short (five second) clip length. With an additional experiment, we explore the extent that trust is uniquely conveyed over other attitudes associated with gaze, such as interest, unfriendliness, and admiration.

Digital games with realistic virtual characters have become very popular. The ability for players to promptly control their character is a crucial feature of these types of games, be it platform games, first-person shooters, or role-playing games. Delays in the responsiveness of a player's character, for example due to extensive AI calculations or to network latencies, can considerably reduce the player's enjoyment of a game. In this paper, we present a thorough analysis of the consequences of such delays on the players' experience across two games with different levels of difficulty. We investigate the effects of responsiveness on the player's experience, performance, and perception of the virtual character, as well as the player's adaptability to delays. We find that delay affects the enjoyment of the games as well as the performance, but only becomes really important when a game becomes more challenging. Furthermore, players can get used to delay within a few minutes of play, so that their performance does not significantly differ from players without a delay handicap.