In Mixed Reality (MR), in-the-moment guidance can support physical tasks under time and attention constraints. In contexts where such guidance is embodied, users must rapidly evaluate whether to trust and follow an advisor’s recommendations. Prior work has examined how avatar representation shapes trust and social perception, but less is known about user responses when guidance conflicts and multiple embodied agents compete for attention in real time. We study trust-relevant perception and behavior in an MR paradigm where two co-present agents represent the same advisor and deliver step-by-step instructions with predefined disagreements. In a within-subjects study, we compared a volumetric avatar (VA) and a high-fidelity 3D character avatar (CA), holding identity and expressive behavior constant, and measured trust, social presence, and behavioral adherence during contradictory steps. Rendering style produced no detectable differences in trust, social presence, or co-presence. Instead, participants managed disagreement through mixed adherence and switching between agents, suggesting that instruction competition and time pressure strongly shape decision-making. These findings position conflicting guidance as a distinct context for studying trust in embodied agents, where compliance may reflect rapid prioritization rather than stable commitment to a representation. We discuss implications for designing and evaluating MR agents when timing and attention constraints can overshadow representation cues.

Concerts and performances in virtual reality are becoming increasingly popular. Accurately visualizing the detailed hand motions of musicians playing instruments is challenging. In this work, we present a real-time motion synthesis method that generates the detailed fretting hand motion for playing guitar. Our approach first involves capturing and post-processing hand motion data from guitar performances to create a data set for training. The post-processing ensures that the fingertip positions are placed as accurately as possible regarding distance from the fretboard and location between the frets. We then train a neural network that learns to predict hand and finger poses based on guitar tabs and previous poses. We found that our method produces reasonably stable motion and evaluate our results using accuracy measures and visual evaluation.

When communicating with embodied conversational agents (ECAs) in virtual reality, there might be delays in the responses of the agents lasting several seconds, for example, due to more extensive computations of the answers when large language models are used. Such delays might lead to unnatural or frustrating interactions. In this paper, we investigate filler types to mitigate these effects and lead to a more positive experience and perception of the agent. In a within-subject study, we asked 24 participants to communicate with ECAs in virtual reality, comparing four strategies displayed during the delays: a multimodal behavioral filler consisting of conversational and gestural fillers, a base condition with only idle motions, and two symbolic indicators with progress bars, one embedded as a badge on the agent, the other one external and visualized as a thinking bubble. Our results indicate that the behavioral filler improved perceived response time, three subscales of presence, humanlikeness, and naturalness. Participants looked away from the face more often when symbolic indicators were displayed, but the visualizations did not lead to a more positive impression of the agent or to increased presence. The majority of participants preferred the behavioral fillers, only 12.5% and 4.2% favored the symbolic embedded and external conditions, respectively.

In this work, we describe the Curation Tree framework that has matured as part of our game-based research and teaching. We use this framework to manage the top-level experience of our players via a text-based, authorable behavior tree system that is easy to implement, test, and re-use. We present three case studies based on Unity games built within our framework: the first involves virtual reality games involving grasping objects with the hands; the second, Treatment X, uses the framework to implement a causal learning game; and the last involves narrative games based on a large language model. Based on our experiences with this system, we reflect on its limitations and make several recommendations based on its strengths, namely to centralize game logic with behaviors, to centralize user event handling, to prefer simple behaviors over complex ones, to use text-based scripts for authoring with version control, and to separate behaviors from persistent state.

Advancing virtual reality technologies are enabling real-time virtual-face to virtual-face communication. Hand tracking systems that are integrated into Head-Mounted Displays (HMD) enable users to directly interact with their environments and with each other using their hands as opposed to using controllers. Due to the novelties of these technologies our understanding of how they impact our interactions is limited. In this paper, we investigate the consequences of using different interaction control systems, hand tracking or controllers, when interacting with others in a virtual environment. We design and implement NASA’s Survival on the Moon teamwork evaluation exercise in virtual reality (VR) and test for effects with and without allowing verbal communication. We evaluate social presence, perceived comprehension, team cohesion, group synergy, task workload, as well as task performance and duration. Our findings reveal that audio communication significantly enhances social presence, perceived comprehension, and team cohesion, but it also increases effort workload and negatively impacts group synergy. The choice of interaction control systems has limited impact on various aspects of virtual collaboration in this scenario, although participants using hand tracking reported lower effort workload, while participants using controllers reported lower mental workload in the absence of audio.

Eye movement plays an important role in face-to-face communication. In this work, we present a deep learning approach for synthesizing the eye movements of avatars for two-party conversations and evaluate viewer perception of different types of eye motions. We aim to synthesize believable gaze behavior based on head motions and audio features as they would typically be available in virtual reality applications. To this end, we captured the head motion, eye motion, and audio of several two-party conversations and trained an RNN-based model to predict where an avatar looks in a two-person conversational scenario. We evaluated our approach with a user study on the perceived quality of the eye animation and compared our method with other eye animation methods. While our model was not rated highest, our model and our user study lead to a series of insights on model features, viewer perception, and study design that we present.

Detailed hand motions play an important role in face-to-face communication to emphasize points, describe objects, clarify concepts, or replace words altogether. While shared virtual reality (VR) spaces are becoming more popular, these spaces do not, in most cases, capture and display accurate hand motions. In this article, we investigate the consequences of such errors in hand and finger motions on comprehension, character perception, social presence, and user comfort. We conduct three perceptual experiments where participants guess words and movie titles based on motion captured movements. We introduce errors and alterations to the hand movements and apply techniques to synthesize or correct hand motions. We collect data from more than 1000 Amazon Mechanical Turk participants in two large experiments, and conduct a third experiment in VR. As results might differ depending on the virtual character used, we investigate all effects on two virtual characters of different levels of realism. We furthermore investigate the effects of clip length in our experiments. Amongst other results, we show that the absence of finger motion significantly reduces comprehension and negatively affects people’s perception of a virtual character and their social presence. Adding some hand motions, even random ones, does attenuate some of these effects when it comes to the perception of the virtual character or social presence, but it does not necessarily improve comprehension. Slightly inaccurate or erroneous hand motions are sufficient to achieve the same level of comprehension as with accurate hand motions. They might however still affect the viewers’ impression of a character. Finally, jittering hand motions should be avoided as they significantly decrease user comfort.

We present an empirical evaluation of immersion and self-avatars as compared to desktop viewing in Virtual Reality (VR) for learning computer programming and computational thinking in middle school education using an educational VR simulation. Students were asked to programmatically choreograph dance performances for virtual characters within an educational desktop application we built earlier called Virtual Environment Interactions (VEnvI). As part of a middle school science class, 90 students from the 6th and 7th grades participated in our study. All students first visually programmed dance choreography for a virtual character they created in VEnvI on a laptop. Then, they viewed and interacted with the resulting dance performance in a between-subjects design in one of the three conditions. We compared and contrasted the benefits of embodied immersive virtual reality (EVR) viewing utilizing a head-mounted display with a body-scaled and gender-matched self-avatar, immersive virtual reality only (IVR) viewing, and desktop VR (NVR) viewing with VEnvI on pedagogical outcomes, programming performance, presence, and attitudes towards STEM and computational thinking. Results from a cognition questionnaire showed that, in the learning dimensions of Knowledge and Understanding (Bloom's taxonomy) as well as Multistructural (SOLO taxonomy), participants in EVR and IVR scored significantly higher than NVR. Also, participants in EVR scored significantly higher than IVR. We also discovered similar results in objective programming performance and presence scores in VEnvI. Furthermore, students' attitudes towards computer science, programming confidence, and impressions significantly improved to be the highest in EVR and then IVR as compared to NVR condition. Our work suggests that educators and developers of educational VR simulations, who want to enhance knowledge and understanding as well as simultaneous acquisition of multiple abstract concepts, can do so by employing immersion and self-avatars in VR learning experiences.

Abstract The importance of eyes for virtual characters stems from the intrinsic social cues in a person's eyes. While previous work on computer generated eyes has considered realism and naturalness, there has been little investigation into how details in the eye animation impact the perception of an avatar's internal emotional state. We present three large scale experiments (N≈500) that investigate the extent to which viewers can identify if an avatar is scared. We find that participants can identify a scared avatar with accuracy using cues in the eyes including pupil size variation, gaze, and blinks. Because eye trackers return pupil diameter in addition to gaze, our experiments inform practitioners that animating the pupil correctly will add expressiveness to a virtual avatar with negligible additional cost. These findings also have implications for creating expressive eyes in intelligent conversational agents and social robots.

A primary goal of the Virtual Reality ( VR ) community is to build fully immersive and presence-inducing environments with seamless and natural interactions. To reach this goal, researchers are investigating how to best directly use our hands to interact with a virtual environment using hand tracking. Most studies in this field require participants to perform repetitive tasks. In this article, we investigate if results of such studies translate into a real application and game-like experience. We designed a virtual escape room in which participants interact with various objects to gather clues and complete puzzles. In a between-subjects study, we examine the effects of two input modalities (controllers vs. hand tracking) and two grasping visualizations (continuously tracked hands vs. virtual hands that disappear when grasping) on ownership, realism, efficiency, enjoyment, and presence. Our results show that ownership, realism, enjoyment, and presence increased when using hand tracking compared to controllers. Visualizing the tracked hands during grasps leads to higher ratings in one of our ownership questions and one of our enjoyment questions compared to having the virtual hands disappear during grasps as is common in many applications. We also confirm some of the main results of two studies that have a repetitive design in a more realistic gaming scenario that might be closer to a typical user experience.

In this work, we investigate the influence that audio and visual feedback have on a manipulation task in virtual reality (VR). Without the tactile feedback of a controller, grasping virtual objects using one’s hands can result in slower interactions because it may be unclear to the user that a grasp has occurred. Providing alternative feedback, such as visual or audio cues, may lead to faster and more precise interactions, but might also affect user preference and perceived ownership of the virtual hands. In this study, we test four feedback conditions for virtual grasping. Three of the conditions provide feedback for when a grasp or release occurs, either visual, audio, or both, and one provides no feedback for these occurrences. We analyze the effect each feedback condition has on interaction performance, measure their effect on the perceived ownership of the virtual hands, and gauge user preference. In an experiment, users perform a pick-and-place task with each feedback condition. We found that audio feedback for grasping is preferred over visual feedback even though it seems to decrease grasping performance, and found that there were little to no differences in ownership between our conditions.

The gaze behavior of virtual avatars is critical to social presence and perceived eye contact during social interactions in Virtual Reality. Virtual Reality headsets are being designed with integrated eye tracking to enable compelling virtual social interactions. This paper shows that the near infra-red cameras used in eye tracking capture eye images that contain iris patterns of the user. Because iris patterns are a gold standard biometric, the current technology places the user's biometric identity at risk. Our first contribution is an optical defocus based hardware solution to remove the iris biometric from the stream of eye tracking images. We characterize the performance of this solution with different internal parameters. Our second contribution is a psychophysical experiment with a same-different task that investigates the sensitivity of users to a virtual avatar's eye movements when this solution is applied. By deriving detection threshold values, our findings provide a range of defocus parameters where the change in eye movements would go unnoticed in a conversational setting. Our third contribution is a perceptual study to determine the impact of defocus parameters on the perceived eye contact, attentiveness, naturalness, and truthfulness of the avatar. Thus, if a user wishes to protect their iris biometric, our approach provides a solution that balances biometric protection while preventing their conversation partner from perceiving a difference in the user's virtual avatar. This work is the first to develop secure eye tracking configurations for VR/AR/XR applications and motivates future work in the area.

In this work, we investigate the influence of different visualizations on a manipulation task in virtual reality (VR). Without the haptic feedback of the real world, grasping in VR might result in intersections with virtual objects. As people are highly sensitive when it comes to perceiving collisions, it might look more appealing to avoid intersections and visualize non-colliding hand motions. However, correcting the position of the hand or fingers results in a visual-proprioceptive discrepancy and must be used with caution. Furthermore, the lack of haptic feedback in the virtual world might result in slower actions as a user might not know exactly when a grasp has occurred. This reduced performance could be remediated with adequate visual feedback.

Research has shown that keyframed eye motions are perceived as more realistic when some noise is added to eyeball motions and to pupil size changes. We investigate whether this noise, in contrast to being motion captured, can be synthesized with standard techniques, e.g., procedural or data-driven approaches. In a two-alternative forced choice task, we compare eye animations created with four different techniques: motion captured, procedural, data-driven, and keyframed (lacking noise). Our perceptual experiment uses three character models with different levels of realism and two motions. Our results suggest that procedural and data-driven noise can be used to create animations at similar perceived naturalness to our motion captured approach. Participants’ eye movements when viewing the animations show that animations without jitter yielded fewer fixations, suggesting ease of dismissal as unnatural.

We evaluate subtle, emotionally-driven models of eye movement animation. Two models are tested, reading and face scanning, each based on recorded gaze transition probabilities. For reading, simulated emotional mood is governed by the probability density function that varies word advancement, i.e., re-fixations, forward, or backward skips. For face scanning, gaze behavior depends on task (gender or emotion discrimination) or the facial emotion portrayed. Probability density functions in both cases are derived from empirically observed transitions that significantly alter viewing behavior, captured either during mood-induced reading or during scanning faces expressing different emotions. A perceptual study shows that viewers can distinguish between reading and face scanning eye movements. However, viewers could not gauge the emotional valence of animated eye motion. For animation, our contribution shows that simulated emotionally-driven viewing behavior is too subtle to be discerned, or it needs to be exaggerated to be effective.

Most commercial virtual reality applications with self avatars provide users with a “one-size fits all” avatar. While the height of this body may be scaled to the user's height, other body proportions, such as limb length and hand size, are rarely customized to fit an individual user. Prior research has shown that mismatches between users' avatars and their actual bodies can affect size perception and feelings of body ownership. In this paper, we consider how concepts related to the virtual hand illusion, user experience, and task efficiency are influenced by variations between the size of a user's actual hand and their avatar's hand. We also consider how using a tracked controller or tracked gestures affect these concepts. We conducted a 2×3 within-subjects study (n=20), with two levels of input modality: using tracked finger motion vs. a hand-held controller (Glove vs. Controller), and three levels of hand scaling (Small, Fit, and Large). Participants completed 2 block-assembly trials for each condition (for a total of 12 trials). Time, mistakes, and a user experience survey were recorded for each trial. Participants experienced stronger feelings of ownership and realism in the Glove condition. Efficiency was higher in the Controller condition and supported by play data of more time spent, blocks grabbed, and blocks dropped in the Glove condition. We did not find enough evidence for a change in agency and the intensity of the virtual hand illusion depending on hand size. Over half of the participants indicated preferring the Glove condition over the Controller condition, mentioning fun and efficiency as factors in their choices. Preferences on hand scaling were mixed but often attributed to efficiency. Participants liked the appearance of their virtual hand more while using the Fit instead of Large hands. Several interaction effects were observed between input modality and hand scaling, for example, for smaller hands, tracked hands evoked stronger feelings of ownership compared to using a controller. Our results show that the virtual hand illusion is stronger when participants are able to control a hand directly rather than with a hand-held device, and that the virtual reality task must first be considered to determine which modality and hand size are the most applicable.

No abstract available.

Hand tracking and haptics are gaining more importance as key technologies of virtual reality (VR) systems. For designing such systems, it is fundamental to understand how the appearance of the virtual hands influences user experience and how the human brain integrates vision and haptics. However, it is currently unknown whether multi-sensory integration of visual and haptic feedback can be influenced by the appearance of virtual hands in VR. We performed a user study in VR to gain insight into the effect of hand appearance on how the brain combines visual and haptic signals using a cue-conflict paradigm. In this paper, we show that the detection of surface irregularities (bumps and holes) sensed by eyes and hands is affected by the rendering of avatar hands. However, sensitivity changes do not correlate with the degree of perceived limb ownership. Qualitative feedback provides insights into potentially distracting cues in visual-haptic integration.

Abstract Eye movements are an essential part of non‐verbal behavior. Non‐player characters, as they occur in many games, communicate with the player through dialogue and non‐verbal behavior and can have a strong influence on player experience or even on gameplay. In this paper, we evaluate a procedural model designed to synthesize the subtleties of eye motion. More specifically, our model adds microsaccadic jitter and pupil unrest both modeled by 1/fαor pink noise to the saccadic main sequence. In a series of perceptual two‐alternative forced‐choice experiments, we explore the perceived naturalness of different parameters of pink noise by comparing synthesized motions to rendered motion of recorded eye movements at extreme close shot and close shot distances. Our results show that, on average, animations based on a procedural model with pink noise were perceived and evaluated as highly natural, whereas data‐driven motion without any jitter or with unfiltered jitter were consistently selected as the least natural in appearance.

Abstract Real‐time animation controllers are fundamental for animating characters in response to player input in digital games. However, the design of such controllers requires making trade‐offs between the naturalness of the character's motions and the promptness of the character's response. In this paper, we investigate the effects of such trade‐offs on the players' enjoyment, control, satisfaction, and opinion of the character in a simple platform game. In our first experiment, we compare three controllers having the same responsiveness, but varying levels of naturalness. In the second experiment, we compare three controllers having increasing realism at the expense of responsiveness. In our third experiment, we keep motion naturalness and responsiveness constant but vary the avatar. Not surprisingly, our least responsive controller negatively affects players' performance and perceived ability to control the character. However, we also find that players are most satisfied with their own performance using our least natural controller, that differences in animation can significantly alter players' enjoyment with responsiveness being equal, that players do not report increased motion quality with our most natural controller, although viewers outside of the game context do, and that the avatar model affected the perception of the character but not players' enjoyment or perceived realism.

Eye motions constitute an important part of our daily face-to-face interactions. Even subtle details in the eyes’ motions give us clues about a person’s thoughts and emotions. Believable and natural animation of the eyes is therefore crucial when creating appealing virtual characters. In this article, we investigate the perceived naturalness of detailed eye motions, more specifically of jitter of the eyeball rotation and pupil diameter on three virtual characters with differing levels of realism. Participants watched stimuli with six scaling factors from 0 to 1 in increments of 0.2, varying eye rotation and pupil size jitter individually, and they had to indicate if they would like to increase or decrease the level of jitter to make the animation look more natural. Based on participants’ responses, we determine the scaling factors for noise attenuation perceived as most natural for each character when using motion-captured eye motions. We compute the corresponding average jitter amplitudes for the eyeball rotation and pupil size to serve as guidelines for other characters. We find that the amplitudes perceived as most natural depend on the character, with our character with a medium level of realism requiring the largest scaling factors.

The ability to select or customize characters in educational applications and games has been shown to influence factors related to learning effects such as transfer, self-efficacy, and motivation. Most previous conclusions on the perception of virtual characters and the effect of character assignment in interactive applications have been reached through short, one-task experiments. To investigate more long-term effects of assigning versus customizing characters as well as explore perceptions of personal character appearance, we conduct a study in which sixth and seventh grade students are introduced to programming concepts with the software VEnvI (Virtual Environment Interactions) in seven one-hour sessions over two weeks. In VEnvI, students create performances for virtual characters by assembling blocks. With a between-subjects design, in which some of the students can alter their character and others are not given that possibility, we examine the influence of the presence or absence of character choice options on learning.

No abstract available.

How does the appearance of a virtual hand affect own-body perception? Previous studies have compared either two or three hand models at a time, with their appearances limited to realistic hands and abstract or simple objects. To investigate the effects of different realisms, render styles, and sensitivities to pain on the virtual hand illusion (VHI), we conduct two studies in which participants take on controllable hand models with six distinct appearances. We collect questionnaire data and comments regarding responses to impacts and threats to assess differences in the strength of the VHI.

A convolution-filtering technique is introduced for the synthesis of eye gaze data. Its purpose is to produce, in a controlled manner, a synthetic stream of raw gaze position coordinates, suitable for: (1) testing event detection filters, and (2) rendering synthetic eye movement animations for testing eye tracking gaze estimation algorithms. Synthetic gaze data is parameterized by sampling rate, microsaccadic jitter, and simulated measurement error. Sampled synthetic gaze data is compared against real data captured by an eye tracker showing similar signal characteristics.

Fifty-four middle school girls were a part of weeklong program geared towards encouraging young women in STEM careers. During a two-day computer science intervention, we introduced students to our VEnvI software where students program virtual characters to perform choreography. During this time, students were introduced programming concepts: sequencing, loops, and variables. They also spent time performing to learn some of the movements in VEnvI and discussed different aspects of the VEnvI software design including motion capture & Unity 3D Development. The latter conversations were couched in relation to being a computer scientist. Data were collected from the students on the first day of the camp as well as at the end of the second day. Each student was given a piece of paper that only contained a code corresponding to her. The instructions were to write, "What do computer scientists do?" on one side of the page, and to draw a picture of a computer scientist on the other side. Our findings suggest that, at least in the short term, this type of intervention can impact students' perceptions of computer science as well as broaden their perspectives of computer scientists. More students drew female computer scientists in the post-test and showed computer scientists working in teams rather than alone. Further, students' ideas of what computer scientists "do" shifted from fixing computers and making websites to creating, designing, and inventing with computers.

The virtual hand illusion is a body ownership illusion that occurs in a virtual environment. Previous studies reached different conclusions on the effect of realism of the controllable virtual hand model on the intensity of the perceived illusion. In our experiments, we compare participants' responses to virtual impacts and threats when using hand models with different levels of realism. Our findings indicate that an illusion can be created for any model but that the effect is perceived weakest for a non-anthropomorphic block model and strongest for a realistic human hand model in direct comparison. We furthermore find that the reactions to our experiments highly vary between participants.

We detail the design, implementation, and an initial evaluation of a virtual reality education and entertainment (edutainment) application called Virtual Environment Interactions (VEnvI). VEnvI is an application in which students learn computer science concepts through the process of choreographing movement for a virtual character using a fun and intuitive interface. In this exploratory study, 54 participants as part of a summer camp geared towards promoting participation of women in science and engineering programmatically crafted a dance performance for a virtual human. A subset of those students participated in an immersive embodied interaction metaphor in VEnvI. In creating this metaphor that provides first-person, embodied experiences using self-avatars, we seek to facilitate engagement, excitement and interest in computational thinking. We qualitatively and quantitatively evaluated the extent to which the activities of the summer camp, programming the dance moves, and the embodied interaction within VEnvI facilitated students' edutainment, presence, interest, excitement, and engagement in computing, and the potential to alter their perceptions of computing and computer scientists. Results indicate that students enjoyed the experience and successfully engaged the virtual character in the immersive embodied interaction, thus exhibiting high telepresence and social presence. Students also showed increased interest and excitement regarding the computing field at the end of their summer camp experience using VEnvI.

American Sign Language (ASL) fingerspelling is the act of spelling a word letter-by-letter when a specific sign does not exist to represent it. Synthesizing intelligible ASL, which includes fingerspelling as an integral part, is important to create signing virtual characters for training and communicating in virtual environments or further applications. The rhythm and speed of fingerspelling play a large role in how well fingerspelling is understood. Using motion capture technologies, we record fingerspelling and analyze timing information about letters in the words. Our goal is to identify fingerspelling timing information and use it to create fingerspelling animations that are natural and understandable.

In an initial study, we characterized the properties of human reach motion in the presence or absence of visuo-haptic feedback in real and Immersive Virtual Environments (IVEs) or virtual reality within a participant’s maximum arm reach. Our goal is to understand how physical reaching actions to the perceived location of targets in the presence or absence of visuo-haptic feedback are different between real and virtual viewing conditions. Typically, participants reach to the perceived location of objects in the three-dimensional (3D) environment to perform selection and manipulation actions during 3D interaction in applications such as virtual assembly or rehabilitation. In these tasks, participants typically have distorted perceptual information in the IVE as compared to the real world, in part due to technological limitations such as minimal visual field of view, resolution, latency, and jitter. In an empirical evaluation, we asked the following questions: (i) how do the perceptual differences between virtual and real world affect our ability to accurately reach to the locations of 3D objects, and (ii) how do the motor responses of participants differ between the presence or absence of visual and haptic feedback? We examined factors such as velocity and distance of physical reaching behavior between the real world and IVE, both in the presence or absence of visuo-haptic information. The results suggest that physical reach responses vary systematically between real and virtual environments, especially in situations involving the presence or absence of visuo-haptic feedback. The implications of our study provide a methodological framework for the analysis of reaching motions for selection and manipulation with novel 3D interaction metaphors and to successfully characterize visuo-haptic versus non-visuo-haptic physical reaches in virtual and real-world situations.

Virtual reality has reached the consumer market and exhibited the potential to become a large social and commercial platform for mainstream markets. New, low-cost devices for virtual reality or mixed reality such as the Oculus Rift, Sony's Project Morpheus, or Microsoft's HoloLens are already available or have been announced and might even outperform previous high-cost systems [Young et al. 2014]. Techniques to measure hand motions are in the making with the goal of being able to control virtual hands using one's own hand movements as well as interact with objects in virtual reality applications. Prototypes such as the combination of the Leap Motion Controller and the Oculus Rift are already available. Applications for these devices will certainly follow.

Eye movements are an essential part of non-verbal behavior. Non-player characters (NPCs), as they occur in many games, communicate with the player through dialogues and non-verbal behavior and can have a strong influence on the player experience or even on gameplay. In this paper we propose a procedural model to synthesize the subtleties of eye motions. More specifically, our model adds microsaccadic jitter and pupil unrest both modeled by 1/f or pink noise to the standard main sequence. In a perceptual two-alternative forced-choice (2AFC) experiment we explore the perceived naturalness of different parameters of pink noise by comparing synthesized motions to rendered motion of recorded eye movements at extreme close shot and close shot distances. Our results show that, on average, data-driven motion is perceived as most natural, followed by parameterized pink noise, with motion lacking microsaccadic jitter being consistently selected as the least natural in appearance.

Abstract The human hand is a complex biological system able to perform numerous tasks with impressive accuracy and dexterity. Gestures furthermore play an important role in our daily interactions, and humans are particularly skilled at perceiving and interpreting detailed signals in communications. Creating believable hand motions for virtual characters is an important and challenging task. Many new methods have been proposed in the Computer Graphics community within the last years, and significant progress has been made towards creating convincing, detailed hand and finger motions. This state of the art report presents a review of the research in the area of hand and finger modeling and animation. Starting with the biological structure of the hand and its implications for how the hand moves, we discuss current methods in motion capturing hands, data‐driven and physics‐based algorithms to synthesize their motions, and techniques to make the appearance of the hand model surface more realistic. We then focus on areas in which detailed hand motions are crucial such as manipulation and communication. Our report concludes by describing emerging trends and applications for virtual hand animation.

Response lag in digital games is known to negatively affect a player's game experience. Particularly with networked multiplayer games, where lag is typically unavoidable, the impact of delays needs to be well understood so that its effects can be mitigated. In this paper, we investigate two aspects of lag independently: latency (constant delay) and jitter (varying delay). We evaluate how latency and jitter each affect a player's enjoyment, frustration, performance, and experience as well as the extent to which players can adjust to such delays after a few minutes of gameplay. We focus on a platform game where the player controls a virtual character through a world. We find that delays up to 300ms do not impact the players' experience as long as they are constant. When jitter was added to a delay of 200ms, however, the lag was noticed by participants more often, hindered players' ability to improve with practice, increased how often they failed to reach the goal of the game, and reduced the perceived motion quality of the character.

Real-time animation controllers are fundamental for animating characters in response to player input. However, the design of such controllers requires making trade-offs between the naturalness of the character's motions and the promptness of the character's response. In this paper, we investigate the effects of such tradeoffs on the players' enjoyment, control, satisfaction, and opinion of the character in a simple platform game. In our first experiment, we compare three controllers having the same responsiveness, but varying levels of naturalness. In the second experiment, we compare three controllers having increasing realism but at the expense of decreased responsiveness. Not surprisingly, our least responsive controller negatively affects players' performance and perceived ability to control the character. However, we also find that players are most satisfied with their own performance using our least natural controller, in which the character moves around the environment in a static pose; that differences in animation can significantly alter players' enjoyment with responsiveness being equal; and that players do not report increased motion quality with our most natural controller, despite viewers outside of a game context rating the same controller as significantly more natural than our other conditions.

n this paper, we introduce an embodied pedagogical approach for learning computational concepts, utilizing computational practices, and developing computational perspectives. During a five-week pilot, a group of students spent after-school time learning the basic elements of dance and then using them to program three-dimensional characters that could perform. Throughout the pilot, we found students consistently standing up in front of their computers and using their bodies to think through the actuation of their characters. Preliminary results suggest that designing a virtual-physical dance performance is a motivating and engaging social context in which to introduce students, especially girls, to alternative applications in computing.

Motion capture remains a popular and widely-used method for animating virtual characters. However, all practical applications of motion capture rely on motion editing techniques to increase the reusability and flexibility of captured motions. Because humans are proficient in detecting and interpreting subtle details in human motion, understanding the perceptual consequences of motion editing is essential. Thus in this work, we perform three experiments to gain a better understanding of how motion editing might affect the emotional content of a captured performance, particularly changes in posture and dynamics, two factors shown to be important perceptual indicators of bodily emotions. In these studies, we analyse the properties (angles and velocities) and perception (recognition rates and perceived intensities) of a varied set of full-body motion clips representing the six emotions anger, disgust, fear, happiness, sadness, and surprise. We have found that emotions are mostly conveyed through the upper body, that the perceived intensity of an emotion can be reduced by blending with a neutral motion, and that posture changes can alter the perceived emotion but subtle changes in dynamics only alter the intensity.

This paper introduces a method for producing high quality hand motion using a small number of markers. The proposed "handover" animation technique constructs joint angle trajectories with the help of a reference database. Utilizing principle component analysis (PCA) applied to the database, the system automatically determines the sparse marker set to record. Further, to produce hand animation, PCA is used along with a locally weighted regression (LWR) model to reconstruct joint angles. The resulting animation is a full-resolution hand which reflects the original motion without the need for capturing a full marker set. Comparing the technique to other methods reveals improvement over the state of the art in terms of the marker set selection. In addition, the results highlight the ability to generalize the motion synthesized, both by extending the use of a single reference database to new motions, and from distinct reference datasets, over a variety of freehand motions.

Digital games with realistic virtual characters have become very popular. The ability for players to promptly control their character is a crucial feature of these types of games, be it platform games, first-person shooters, or role-playing games. Delays in the responsiveness of a player's character, for example due to extensive AI calculations or to network latencies, can considerably reduce the player's enjoyment of a game. In this paper, we present a thorough analysis of the consequences of such delays on the players' experience across two games with different levels of difficulty. We investigate the effects of responsiveness on the player's experience, performance, and perception of the virtual character, as well as the player's adaptability to delays. We find that delay affects the enjoyment of the games as well as the performance, but only becomes really important when a game becomes more challenging. Furthermore, players can get used to delay within a few minutes of play, so that their performance does not significantly differ from players without a delay handicap.

Capturing the body movements of actors to create animations for movies, games, and VR applications has become standard practice, but finger motions are usually added manually as a tedious post-processing step. In this paper, we present a surprisingly simple method to automate this step for gesturing and conversing characters. In a controlled environment, we carefully captured and post-processed finger and body motions from multiple actors. To augment the body motions of virtual characters with plausible and detailed finger movements, our method selects finger motion segments from the resulting database taking into account the similarity of the arm motions and the smoothness of consecutive finger motions. We investigate which parts of the arm motion best discriminate gestures with leave-one-out cross-validation and use the result as a metric to select appropriate finger motions. Our approach provides good results for a number of examples with different gesture types and is validated in a perceptual experiment.

Virtual characters are much in demand for animated movies, games, and other applications. Rapid advances in performance capture and advanced rendering techniques have allowed the movie industry in particular to create characters that appear very human-like. However, with these new capabilities has come the realization that such characters are yet not quite “right.” One possible hypothesis is that these virtual humans fall into an “Uncanny Valley”, where the viewer's emotional response is repulsion or rejection, rather than the empathy or emotional engagement that their creators had hoped for. To explore these issues, we created three animated vignettes of an arguing couple with detailed motion for the face, eyes, hair, and body. In a set of perceptual experiments, we explore the relative importance of different anomalies using two different methods: a questionnaire to determine the emotional response to the full-length vignettes, with and without facial motion and audio; and a 2AFC (two alternative forced choice) task to compare the performance of a virtual “actor” in short clips (extracts from the vignettes) depicting a range of different facial and body anomalies. We found that the facial anomalies are particularly salient, even when very significant body animation anomalies are present.

In this paper, our aim is to determine factors that influence the perceived sex of virtual characters. In Experiment 1, four different model types were used: highly realistic male and female models, an androgynous character, and a point light walker. Three different types of motion were applied to all models: motion captured male and female walks, and neutral synthetic walks. We found that both form and motion influence sex perception for these characters: for neutral synthetic motions, form determines perceived sex, whereas natural motion affects the perceived sex of both androgynous and realistic forms. These results indicate that the use of neutral walks is better than creating ambiguity by assigning an incongruent motion. In Experiment 2 we investigated further the influence of body shape and motion on realistic male and female models and found that adding stereotypical indicators of sex to the body shapes influenced sex perception. Also, that exaggerated female body shapes influences sex judgements more than exaggerated male shapes. These results have implications for variety and realism when simulating large crowds of virtual characters.

In order to analyze the emotional content of motions portrayed by different characters, we created real and virtual replicas of an actor exhibiting six basic emotions: sadness, happiness, surprise, fear, anger, and disgust. In addition to the video of the real actor, his actions were applied to five virtual body shapes: a low- and high-resolution virtual counterpart, a cartoon-like character, a wooden mannequin, and a zombie-like character (Figures 1 and 2). In a point light condition, we also tested whether the absence of a body affected the perceived emotion of the movements. Participants were asked to rate the actions based on a list of 41 more complex emotions. We found that the perception of emotional actions is highly robust and to the most part independent of the character's body, so long as form is present. When motion alone is present, emotions were generally perceived as less intense than in the cases where form was present.