How have player experiences changed over the years? For instance, have there been general shifts in what kinds of emotions players experience and express? We probe these questions with help of recent methodological advances in psychology and Large Language Models (LLMs), in particular the possibility to predict Likert-scale responses based on free-form text. Applying this at scale to three player experience questionnaires (PXI, CORGIS, AESTHEMOS) and 152143 Metacritic user reviews from years 2010-2024, we reveal trends such as an increasing portion of reviews expressing emotional challenge, meaning, and nostalgia. We then analyze the contributions of different genres and games to the trends, in addition to reasons explicitly indicated by the reviews, and establish correlations between review scores and different player experience constructs. Taken together, our results provide novel insights into how player experiences have evolved. Methodologically, we propose and demonstrate a novel and scalable method for analyzing game reviews.

Abstract We present the first motion generation system for playtesting virtual reality (VR) games. Our player model generates VR headset and handheld controller movements from in‐game object arrangements, guided by style reference gameplay examples. We train on the large BOXRR‐23 dataset and apply our framework on the popular VR game Beat Saber . The resulting model Robo‐Saber reproduces skilled performance and captures diverse player behaviors present in the training data. Robo‐Saber demonstrates promise in synthesizing rich gameplay data for predictive applications and enabling whole‐body physics‐based VR playtesting.

Abstract Movement-based design methods have gained increased attention across various research fields and practices, particularly in interaction design. By engaging the body in movement, these methods have the potential to explore a richer, more intuitive, and immersive user experience. A recent project MeCaMinD1 brought together researchers from interaction and sports design to explore, collect, and document movement-based methods and make them practically useful across domains. The methods were explored through a series of workshops, furthering the team’s understanding of their use and practical applicability. This understanding was compiled into a set of design cards that we present here. We discuss the experience of using the resulting cards in movement-based design sessions. We found that while the cards are mostly used in designing, planning, and preparing sessions, they also function as support during the design sessions, used by both facilitators and participants. Based on a final ideation session with both novice and experienced facilitators, we sketch ways to support managing the cards during sessions, integrating them with movement and physical action.

Entertainment videogames have been recognized for their potential therapeutic benefits, but there is a need for more in-depth, game-specific explorations of the game features that could contribute to such benefits. This study examines how players of Dark Souls describe the game as helping them cope with depression. We conducted a thematic analysis of Reddit discussions where players narrate their mental health experiences with the game, using AI tools to assist in identifying relevant data for a purposive sample. Our findings suggest that Dark Souls could support players’ mental health, for example, by (1) cultivating resilience and perseverance through its challenging gameplay, (2) triggering existential reflections through symbolic representations of depression, and (3) enabling supportive online communities and interactions. Our findings offer rich, player-centered insights into the perceived mental health benefits of commercial videogames, highlighting their potential to transcend entertainment and inform the design of engaging digital mental health tools.

This paper seeks to understand the connections between two previously disjoint subfields of game research and design: 1) the study of emotionally impactful games and 2) the study of game feel. Regarding games and emotion, we now understand aspects such as how negative emotions are appreciated in games and can be a desirable quality for designers and players alike. We also understand aspects of game feel such as the importance of responsive player character control and juicy (i.e. exaggerated) feedback for player actions. However, the literature on game feel rarely links to emotion research and focuses on a narrow subset of emotions/feelings such as satisfaction and control. Research is lacking on how game feel design can impact a wider palette of emotions, including negative ones, and how this may require one to "break the rules" of good game feel design, e.g., making it purposefully hard to control the player character. We bridge this gap by employing Constructivist Grounded Theory Analysis to understand a dataset comprising of interview data from 15 participants and 116 game mechanics from a diverse selection of games such as Journey, Celeste, and Freedom Bridge. Through this, we propose Expectation Modulation as the core theory to capture how game feel can elicit emotional experiences. Additionally, we identify 9 design techniques as central to crafting emotional experiences through game feel design.

Figure 1: Our VR dodge roll technique is a hybrid of teleporting and Out-of-Body Locomotion.The images show both user action and the 1st person view captured from a Meta Quest headset.Left: Initial situation with a giant enemy and the yellow dodge roll target and red sword attack target used in our user study.Middle images: The player initiates the dodge roll by thrusting their arms forward.A 3rd person avatar emerges inside the 1st person view and performs the roll animation.Right: The 1st person camera transitions to the new avatar location and the player hits the target with the sword.To see the system in action, please check our supplemental video: https://youtu.be/XGfiUTbJLyY.

Waiting is an everyday activity that is often present in video games. Waiting situations in games can take place during, for instance, loading screens, turn-based action, and cutscenes. Experiences of waiting can encompass a variety of emotions in players, such as anticipation, frustration, and boredom. Thus, understanding how waiting relates to players’ overall experience can be beneficial when designing or analysing games. However, academic discussion on waiting in games is quite scattered, and there is no comprehensive overview available on the subject. This paper contributes a semi-systematic literature review on the topic, augmented with a follow-up survey study. Based on the survey findings, we outline five perspectives from which waiting experiences can be analysed. These are 1) causes of waiting, 2) player goals for waiting, 3) player behaviour when waiting, 4) felt experience of waiting, and 5) player reasoning and decision-making. Our findings give an overview of sources of waiting in games and highlight that, in addition to affecting players' emotions, waiting is an aspect of gameplay that has an effect on players' decisions and behaviour.

The field of Sports Human-Computer Interaction (SportsHCI) investigates interaction design to support a physically active human being. Despite growing interest and dissemination of SportsHCI literature over the past years, many publications still focus on solving specific problems in a given sport. We believe in the benefit of generating fundamental knowledge for SportsHCI more broadly to advance the field as a whole. To achieve this, we aim to identify the grand challenges in SportsHCI, which can help researchers and practitioners in developing a future research agenda. Hence, this paper presents a set of grand challenges identified in a five-day workshop with 22 experts who have previously researched, designed, and deployed SportsHCI systems. Addressing these challenges will drive transformative advancements in SportsHCI, fostering better athlete performance, athlete-coach relationships, spectator engagement, but also immersive experiences for recreational sports or exercise motivation, and ultimately, improve human well-being.

Dance games are one of the most popular game genres in Virtual Reality (VR), and active dance communities have emerged on social VR platforms such as VR Chat. However, effective instruction of dancing in VR or through other computerized means remains an unsolved human-computer interaction problem. Existing approaches either only instruct movements partially, abstracting away nuances, or require learning and memorizing symbolic notation. In contrast, we investigate how realistic, full-body movements designed by a professional choreographer can be instructed on the fly, without prior learning or memorization. Towards this end, we describe the design and evaluation of WAVE, a novel anticipatory movement visualization technique where the user joins a group of dancers performing the choreography with different time offsets, similar to spectators making waves in sports events. In our user study (N=36), the participants more accurately followed a choreography using WAVE, compared to following a single model dancer.

This paper reports the research method of the “Game Expats Story (GES)” project that used qualitative longitudinal research (“QLR”) incorporated with art-based research (“ABR”) in the context of game research. To facilitate greater participant engagement and a higher retention rate of longitudinal participants, we created comic artworks simultaneously while researching the case of migrant/expatriate game developers (“game expats”) in Finland 2020-2023 in two phases: (i) art creation as part of the qualitative data analysis to supplement the researcher’s inductive abstraction of the patterns, and (ii) artwork as a communication and recall tool when periodically engaging with the informants over the multi-year project span. Our findings suggest that the method of QLR-ABR helps game research as it positively influences the researcher’s abstractions of longitudinal data and participants’ continuous engagement with a high retention rate of 89%. We conclude that incorporating artistic methods provides new opportunities for ethnographic research on game development.

Automated biomechanical testing has great potential for the development of VR applications, as initial insights into user behaviour can be gained in silico early in the design process. In particular, it allows prediction of user movements and ergonomic variables, such as fatigue, prior to conducting user studies. However, there is a fundamental disconnect between simulators hosting state-of-the-art biomechanical user models and simulators used to develop and run VR applications. Existing user simulators often struggle to capture the intricacies of real-world VR applications, reducing ecological validity of user predictions. In this paper, we introduce sim2vr, a system that aligns user simulation with a given VR application by establishing a continuous closed loop between the two processes. This, for the first time, enables training simulated users directly in the same VR application that real users interact with. We demonstrate that sim2vr can predict differences in user performance, ergonomics and strategies in a fast-paced, dynamic arcade game. In order to expand the scope of automated biomechanical testing beyond simple visuomotor tasks, advances in cognitive models and reward function design will be needed.

Quests represent an integral part of role-playing games (RPGs). While evocative, narrative-rich quests are still mostly hand-authored, player demands towards more and richer game content, as well as business requirements for continuous player engagement necessitate alternative, procedural quest generation methods. While existing methods produce mostly uninteresting, mechanical quest descriptions, recent advances in AI have brought forth generative language models with promising computational storytelling capabilities. We leverage two of the most successful Transformer models, GPT-2 and GPT-3, to procedurally generate RPG video game quest descriptions. We gathered, processed and openly published a data set of 978 quests and their descriptions from six RPGs. We fine-tuned GPT-2 on this data set with a range of optimizations informed by several mini studies. We validated the resulting Quest-GPT-2 model via an online user study involving 349 RPG players. Our results indicate that one in five quest descriptions would be deemed acceptable by a human critic, yet the variation in quality across individual quests is large. We provide recommendations on current applications of Quest-GPT-2. This is complemented by case-studies on GPT-3 to highlight the future potential of state-of-the-art natural language models for quest generation.

Collecting data is one of the bottlenecks of Human-Computer Interaction (HCI) research. Motivated by this, we explore the potential of large language models (LLMs) in generating synthetic user research data. We use OpenAI’s GPT-3 model to generate open-ended questionnaire responses about experiencing video games as art, a topic not tractable with traditional computational user models. We test whether synthetic responses can be distinguished from real responses, analyze errors of synthetic data, and investigate content similarities between synthetic and real data. We conclude that GPT-3 can, in this context, yield believable accounts of HCI experiences. Given the low cost and high speed of LLM data generation, synthetic data should be useful in ideating and piloting new experiments, although any findings must obviously always be validated with real data. The results also raise concerns: if employed by malicious users of crowdsourcing services, LLMs may make crowdsourcing of self-report data fundamentally unreliable.

Virtual character animation and movement synthesis have advanced rapidly during recent years, especially through a combination of extensive motion capture datasets and machine learning. A remaining challenge is interactively simulating characters that fatigue when performing extended motions, which is indispensable for the realism of generated animations. However, capturing such movements is problematic, as performing movements like backflips with fatigued variations up to exhaustion raises capture cost and risk of injury. Surprisingly, little research has been done on faithful fatigue modeling. To address this, we propose a deep reinforcement learning-based approach, which—for the first time in literature—generates control policies for full-body physically simulated agents aware of cumulative fatigue. For this, we first leverage Generative Adversarial Imitation Learning (GAIL) to learn an expert policy for the skill; Second, we learn a fatigue policy by limiting the generated constant torque bounds based on endurance time to non-linear, state- and time-dependent limits in the joint-actuation space using a Three-Compartment Controller (3CC) model. Our results demonstrate that agents can adapt to different fatigue and rest rates interactively, and discover realistic recovery strategies without the need for any captured data of fatigued movement.

Virtual Reality (VR) games and experiences predominantly have the users interact while standing or seated. However, this only represents a fraction of the full diversity of human movement. In this paper, we explore a novel non-standing approach to VR locomotion where the user performs locomotion movements in the air or only slightly touching the ground with their feet. For instance, the user may lie supine on the ground, reminiscent of the Bicycle Crunch, a core training movement common in Pilates and other forms of bodyweight exercise. Although this cannot generally replace traditional VR locomotion, it provides two benefits that we believe can be of use for specific application domains such as VR exergames: First, the user's lower body movement is not impeded by a small real-life space, allowing versatile navigation of large virtual worlds using walking, running, strafing, and jumping. Second, we allow new ways to activate parts of the body that remain passive in most existing VR interactions. We describe and discuss four different variants of the approach, and investigate two prototypes further in a qualitative user study, to better understand their strengths, weaknesses, and application potential.

Text-to-image generation (TTIG) models, a recent addition to creative AI, can generate images based on a text description. These models have begun to rival the work of professional creatives, and sparked discussions on the future of creative work, loss of jobs, and copyright issues, amongst other important implications. To support the sustainable adoption of TTIG, we must provide rich, reliable and transparent insights into how professionals perceive, adopt and use TTIG. Crucially though, the public debate is shallow, narrow and lacking transparency, while academic work has focused on studying the use of TTIG in a general artist population, but not on the perceptions and attitudes of professionals in a specific industry. In this paper, we contribute a qualitative, exploratory interview study on TTIG in the Finnish videogame industry. Through a Template Analysis on semi-structured interviews with 14 game professionals, we reveal 12 overarching themes, structured into 39 sub-themes on professionals' perception, adoption and use of TTIG in games industry practice. Experiencing (yet another) change of roles and creative processes, our participants' reflections can inform discussions within the industry, be used by policymakers to inform urgently needed legislation, and support researchers in games, HCI and AI to support the sustainable, professional use of TTIG, and foster games as cultural artefacts.

Dancing is a universal human activity, and also a domain of enduring significance in Human-Computer Interaction (HCI) research. However, there has been limited investigation into how computing supports the experiences of recreational dancers. Concurrently, a diverse and sizeable dance community has been emerging in VRChat. Little is known about these dancers’ experiences, motivations, and practices. Yet shedding light into these could inform both VR technology development and the design of systems that better support embodied and complex social interactions. To bridge this gap, we interviewed participants active in the VRChat dance scene. Through thematic analysis, we identified six central facets of their experiences related to freedom, community, dance as an individual experience, dance as a shared experience, dance as a performance, and self-expression and -exploration. Based on these findings, we discuss emerging tensions and highlight beneficial impacts of dancing in VR as well as problems that still await resolving.

Design practices targeting playful movement-based interaction are changing rapidly with both technological and societal developments. In this paper, we provide a snapshot of movement-based interaction designers’ experiences during the Covid-19 pandemic, by interviewing designers working in diverse roles that integrate physical human body movement and digital technologies (i.e., exergame and wearable game designers, playground and landscape architects, sports and dance trainers, and Sports-HCI professionals). Using grounded theory, we have identified two tactics from the designers’ experiences: (i) the significance of face-to-face embodied interactions throughout the entire movement-based interaction design process, and (ii) the importance of positive, yet critical, attitudes to technology for designing bodily experiences, including mixed use of non-technical materials and tools for rapid prototyping and iteration. However, it was evident that such tactics are often not feasible without physical interaction between the designer and users. The restrictions imposed by Covid-19, therefore, further revealed the importance of body movement to designing a playful movement-based interaction — from ideation to execution and testing. This paper offers a worrying view of movement-based interaction design during the Covid-19 era, while calling for further investigation to enrich the discourse that could contribute to this field, possibly also beyond the circumstances of the Covid-19 pandemic.

Deep reinforcement learning (DRL) algorithms for movement control are typically evaluated and benchmarked on sequential decision tasks where imprecise actions may be corrected with later actions, thus allowing high returns with noisy actions. In contrast, we focus on an under-researched class of high-risk, high-precision motion control problems where actions carry irreversible outcomes, driving sharp peaks and ridges to plague the state-action reward landscape. Using computational pool as a representative example of such problems, we propose and evaluate State-Conditioned Shooting (SCOOT), a novel DRL algorithm that builds on advantage-weighted regression (AWR) with three key modifications: 1) Performing policy optimization only using elite samples, allowing the policy to better latch on to the rare high-reward action samples; 2) Utilizing a mixture-of-experts (MoE) policy, to allow switching between reward landscape modes depending on the state; 3) Adding a distance regularization term and a learning curriculum to encourage exploring diverse strategies before adapting to the most advantageous samples. We showcase our features’ performance in learning physically-based billiard shots demonstrating high action precision and discovering multiple shot strategies for a given ball configuration.

Forward biomechanical simulation in HCI holds great promise as a tool for evaluation, design, and engineering of user interfaces. Although reinforcement learning (RL) has been used to simulate biomechanics in interaction, prior work has relied on unrealistic assumptions about the control problem involved, which limits the plausibility of emerging policies. These assumptions include direct torque actuation as opposed to muscle-based control; direct, privileged access to the external environment, instead of imperfect sensory observations; and lack of interaction with physical input devices. In this paper, we present a new approach for learning muscle-actuated control policies based on perceptual feedback in interaction tasks with physical input devices. This allows modelling of more realistic interaction tasks with cognitively plausible visuomotor control. We show that our simulated user model successfully learns a variety of tasks representing different interaction methods, and that the model exhibits characteristic movement regularities observed in studies of pointing. We provide an open-source implementation which can be extended with further biomechanical models, perception models, and interactive environments.

The accurate and personalized estimation of task difficulty provides many opportunities for optimizing user experience. However, user diversity makes such difficulty estimation hard, in that empirical measurements from some user sample do not necessarily generalize to others.

Cutscenes form an integral part of many video games, but their creation is costly, time-consuming, and requires skills that many game developers lack. While AI has been leveraged to semi-automate cutscene production, the results typically lack the internal consistency and uniformity in style that is characteristic of professional human directors. We overcome this shortcoming with Cine-AI, an open-source procedural cinematography toolset capable of generating in-game cutscenes in the style of eminent human directors. Implemented in the popular game engine Unity, Cine-AI features a novel timeline and storyboard interface for design-time manipulation, combined with runtime cinematography automation. Via two user studies, each employing quantitative and qualitative measures, we demonstrate that Cine-AI generates cutscenes that people correctly associate with a target director, while providing above-average usability. Our director imitation dataset is publicly available, and can be extended by users and film enthusiasts.

We propose a novel method for exploring the dynamics of physically based animated characters, and learning a task-agnostic action space that makes movement optimization easier. Like several previous article, we parameterize actions as target states, and learn a short-horizon goal-conditioned low-level control policy that drives the agent's state towards the targets. Our novel contribution is that with our exploration data, we are able to learn the low-level policy in a generic manner and without any reference movement data. Trained once for each agent or simulation environment, the policy improves the efficiency of optimizing both trajectories and high-level policies across multiple tasks and optimization algorithms. We also contribute novel visualizations that show how using target states as actions makes optimized trajectories more robust to disturbances; this manifests as wider optima that are easy to find. Due to its simplicity and generality, our proposed approach should provide a building block that can improve a large variety of movement optimization methods and applications.

A large body of animation research focuses on optimization of movement control, either as action sequences or policy parameters. However, as closed-form expressions of the objective functions are often not available, our understanding of the optimization problems is limited. Building on recent work on analyzing neural network training, we contribute novel visualizations of high-dimensional control optimization landscapes; this yields insights into why control optimization is hard and why common practices like early termination and spline-based action parameterizations make optimization easier. For example, our experiments show how trajectory optimization can become increasingly ill-conditioned with longer trajectories, but parameterizing control as partial target states—e.g., target angles converted to torques using a PD-controller—can act as an efficient preconditioner. Both our visualizations and quantitative empirical data also indicate that neural network policy optimization scales better than trajectory optimization for long planning horizons. Our work advances the understanding of movement optimization and our visualizations should also provide value in educational use.

This paper presents a novel approach to automated playtesting for the prediction of human player behavior and experience. We have previously demonstrated that Deep Reinforcement Learning (DRL) game-playing agents can predict both game difficulty and player engagement, operationalized as average pass and churn rates. We improve this approach by enhancing DRL with Monte Carlo Tree Search (MCTS). We also motivate an enhanced selection strategy for predictor features, based on the observation that an AI agent's best-case performance can yield stronger correlations with human data than the agent's average performance. Both additions consistently improve the prediction accuracy, and the DRL-enhanced MCTS outperforms both DRL and vanilla MCTS in the hardest levels. We conclude that player modelling via automated playtesting can benefit from combining DRL and MCTS. Moreover, it can be worthwhile to investigate a subset of repeated best AI agent runs, if AI gameplay does not yield good predictions on average.

A common problem of mid-air interaction is excessive arm fatigue, known as the "Gorilla arm" effect. To predict and prevent such problems at a low cost, we investigate user testing of mid-air interaction without real users, utilizing biomechanically simulated AI agents trained using deep Reinforcement Learning (RL). We implement this in a pointing task and four experimental conditions, demonstrating that the simulated fatigue data matches human fatigue data. We also compare two effort models: 1) instantaneous joint torques commonly used in computer animation and robotics, and 2) the recent Three Compartment Controller (3CC-) model from biomechanical literature. 3CC- yields movements that are both more efficient and relaxed, whereas with instantaneous joint torques, the RL agent can easily generate movements that are quickly tiring or only reach the targets slowly and inaccurately. Our work demonstrates that deep RL combined with the 3CC- provides a viable tool for predicting both interaction movements and user experiencein silico, without users.

The general trend in exercise interventions, including those based on exergames, is to see high initial enthusiasm but significantly declining adherence. Social play is considered a core tenet of the design of exercise interventions help foster motivation to play. To determine whether social play aids in adherence to exergames, we analyzed data from a study involving five waves of six-week exergame trials between a single-player and multiplayer group. In this paper, we examine the multiplayer group to determine who might benefit from social play and why. We found that people who primarily engage in group play have superior adherence to people who primarily play alone. People who play alone in a multiplayer exergame have worse adherence than playing a single-player version, which can undo any potential benefit of social play. The primary construct distinguishing group versus alone players is their sense of program belonging. Program belonging is, thus, crucial to multiplayer exergame design.

This paper addresses the problem of synthesizing simulated humanoid climbing movements given the target holds, e.g., by the player of a climbing game. We contribute the first deep reinforcement learning solution that can handle interactive physically simulated humanoid climbing with more than one limb switching holds at the same time. A key component of our approach is Self-Supervised Episode State Initialization (SS- ESI), which ensures diverse exploration and speeds up learning, compared to a baseline approach where the climber is reset to an initial pose after failure. Our results also show that training with a multi-step action parameterization can produce both smoother movements and enable learning from slightly fewer explored actions at the cost of increased simulation time per action.

Animation and machine learning research have shown great advancements in the past decade, leading to robust and powerful methods for learning complex physically-based animations. However, learning can take hours or days, especially if no reference movement data is available. In this paper, we propose and evaluate a novel combination of techniques for accelerating the learning of stable locomotion movements through self-imitation learning of synthetic animations. First, we produce synthetic and cyclic reference movement using a recent online tree search approach that can discover stable walking gaits in a few minutes. This allows us to use reinforcement learning with Reference State Initialization (RSI) to find a neural network controller for imitating the synthesized reference motion. We further accelerate the learning using a novel curriculum learning approach called Termination Curriculum (TC), that adapts the episode termination threshold over time. The combination of the RSI and TC ensures that simulation budget is not wasted in regions of the state space not visited by the final policy. As a result, our agents can learn locomotion skills in just a few hours on a modest 4-core computer. We demonstrate this by producing locomotion movements for a variety of characters.

Efficient algorithms for 3D character control in continuous control setting remain an open problem in spite of the remarkable recent advances in the field. We present a sampling-based model-predictive controller that comes in the form of a Monte Carlo tree search (MCTS). The tree search utilizes information from multiple sources including two machine learning models. This allows rapid development of complex skills such as 3D humanoid locomotion with less than a million simulation steps, in less than a minute of computing on a modest personal computer. We demonstrate locomotion of 3D characters with varying topologies under disturbances such as heavy projectile hits and abruptly changing target direction. In this paper we also present a new way to combine information from the various sources such that minimal amount of information is lost. We furthermore extend the neural network, involved in the algorithm, to represent stochastic policies. Our approach yields a robust control algorithm that is easy to use. While learning, the algorithm runs in near real-time, and after learning the sampling budget can be reduced for real-time operation.

This paper presents a review of intrinsic motivation in player modeling, with a focus on simulation-based game testing. Modern AI agents can learn to win many games; from a game testing perspective, a remaining research problem is how to model the aspects of human player behavior not explained by purely rational and goal-driven decision making. A major piece of this puzzle is constituted by intrinsic motivations, i.e., psychological needs that drive behavior without extrinsic reinforcement such as game score. We first review the common intrinsic motivations discussed in player psychology research and artificial intelligence, and then proceed to systematically review how the various motivations have been implemented in simulated player agents. Our work reveals that although motivations such as competence and curiosity have been studied in AI, work on utilizing them in simulation-based game testing is sparse, and other motivations such as social relatedness, immersion, and domination appear particularly underexplored.

Abstract We propose a novel learning‐based solution for motion planning of physically‐based humanoid climbing that allows for fast and robust planning of complex climbing strategies and movements, including extreme movements such as jumping. Similar to recent previous work, we combine a high‐level graph‐based path planner with low‐level sampling‐based optimization of climbing moves. We contribute through showing that neural network models of move success probability, effortfulness, and control policy can make both the high‐level and low‐level components more efficient and robust. The models can be trained through random simulation practice without any data. The models also eliminate the need for laboriously hand‐tuned heuristics for graph search. As a result, we are able to efficiently synthesize climbing sequences involving dynamic leaps and one‐hand swings, i.e. there are no limits to the movement complexity or the number of limbs allowed to move simultaneously. Our supplemental video also provides some comparisons between our AI climber and a real human climber.

This paper addresses the problem of offline path and movement planning for wall climbing humanoid agents. We focus on simulating bouldering, i.e. climbing short routes with diverse moves, although we also demonstrate our system on a longer wall. Our approach combines a graph-based high-level path planner with low-level sampling-based optimization of climbing moves. Although the planning problem is complex, our system produces plausible solutions to bouldering problems (short climbing routes) in less than a minute. We further utilize a k-shortest paths approach, which enables the system to discover alternative paths - in climbing, alternative strategies often exist, and what might be optimal for one climber could be impossible for others due to individual differences in strength, flexibility, and reach. We envision our system could be used, e.g. in learning a climbing strategy, or as a test and evaluation tool for climbing route designers. To the best of our knowledge, this is the first paper to solve and simulate rich humanoid wall climbing, where more than one limb can move at the same time, and limbs can also hang free for balance or use wall friction in addition to predefined holds.

We present the design and evaluation of the Augmented Climbing Wall (ACW). The system combines computer vision and interactive projected graphics for motivating and instructing indoor wall climbing. We have installed the system in a commercial climbing center, where it has been successfully used by hundreds of climbers, including both children and adults. Our primary contribution is a novel movement-based game system that can inform the design of future games and augmented sports. We evaluate ACW based on three user studies (N=50, N=10, N=10) and further observations and interviews. We highlight three central themes of how digital augmentation can contribute to a sport: increasing diversity of movement and challenges, enabling user-created content in an otherwise risky environment, and enabling procedurally generated content. We further discuss how ACW represents an underexplored class of interactive systems, i.e., proximity interaction on wall-sized interactive surfaces, which presents novel human-computer interaction challenges.

We present SaDDP, a sampled version of the widely used differential dynamic programming (DDP) control algorithm. We contribute through establishing a novel connection between two major branches of robotics control research, that is, gradient-based methods such as DDP, and Monte Carlo methods such as path integral control (PI) that utilize random simulated trajectory rollouts. One of our key observations is that the Taylor-expansion, central to DDP, can be reformulated in terms of second-order statistics computed of the sampled trajectories. SaDDP makes few assumptions about the controlled system and works with black-box dynamics simulations with non-smooth contacts. Our simulation results show that the method outperforms PI and CMA-ES in both a simple linear-quadratic problem, and a multilink arm reaching task with obstacles.

This paper presents a novel algorithm for real-time path-planning in a dynamic environment such as a computer game. We utilize a real-time sampling approach based on the Rapidly Exploring Random Tree (RRT) algorithm that has enjoyed wide success in robotics. More specifically, our algorithm is based on the RRT* and informed RRT* variants. We contribute by introducing an online tree rewiring strategy that allows the tree root to move with the agent without discarding previously sampled paths. Our method also does not have to wait for the tree to be fully built, as tree expansion and taking actions are interleaved. To our knowledge, this is the first real-time variant of RRT*.

We present a novel, general-purpose Model-Predictive Control (MPC) algorithm that we call Control Particle Belief Propagation (C-PBP). C-PBP combines multimodal, gradient-free sampling and a Markov Random Field factorization to effectively perform simultaneous path finding and smoothing in high-dimensional spaces. We demonstrate the method in online synthesis of interactive and physically valid humanoid movements, including balancing, recovery from both small and extreme disturbances, reaching, balancing on a ball, juggling a ball, and fully steerable locomotion in an environment with obstacles. Such a large repertoire of movements has not been demonstrated before at interactive frame rates, especially considering that all our movement emerges from simple cost functions. Furthermore, we abstain from using any precomputation to train a control policy offline, reference data such as motion capture clips, or state machines that break the movements down into more manageable subtasks. Operating under these conditions enables rapid and convenient iteration when designing the cost functions.

We present a Model-Predictive Control (MPC) system for online synthesis of interactive and physically valid character motion. Our system enables a complex (36-DOF) 3D human character model to balance in a given pose, dodge projectiles, and improvise a get up strategy if forced to lose balance, all in a dynamic and unpredictable environment. Such contact-rich, predictive and reactive motions have previously only been generated offline or using a handcrafted state machine or a dataset of reference motions, which our system does not require. For each animation frame, our system generates trajectories of character control parameters for the near future --- a few seconds --- using Sequential Monte Carlo sampling. Our main technical contribution is a multimodal, tree-based sampler that simultaneously explores multiple different near-term control strategies represented as parameter splines. The strategies represented by each sample are evaluated in parallel using a causal physics engine. The best strategy, as determined by an objective function measuring goal achievement, fluidity of motion, etc., is used as the control signal for the current frame, but maintaining multiple hypotheses is crucial for adapting to dynamically changing environments.

We investigate consciously controlled pupil size as an input modality. Pupil size is affected by various processes, e.g., physical activation, strong emotional experiences and cognitive effort. Our hypothesis is that given continuous feedback, users can learn to control pupil size via physical and psychological self-regulation. We test it by measuring the magnitude of self evoked pupil size changes following seven different instructions, while providing real time graphical feedback on pupil size. Results show that some types of voluntary effort affect pupil size on a statistically significant level. A second controlled experiment confirms that subjects can produce pupil dilation and construction on demand during paced tasks. Applications and limitations to using voluntary pupil size manipulation as an input modality are discussed.

This paper presents a novel way of using real-time heart rate information to control a physically interactive biathlon (skiing and shooting) computer game. Instead of interfacing the game to an exercise bike or other equipment with speed output, the skiing speed is directly proportional to heart rate. You can freely choose the form of physical exercise, which makes it easier for people with different skill levels and backgrounds to play together. The system can be used with any exercise machine or form. To make playing meaningful instead of simply exercising as hard as you can, a high heart rate impedes the shooting part of the game by making the sight less steady. This balancing mechanism lets the player try out different tactics, varying from very slow skiing and sharp shooting to fast skiing and random shooting. The game has been evaluated in a user study with eight participants. The results show that heart rate interaction is fun and usable interaction method.

This paper presents Kick Ass Kung-Fu, a martial arts game installation where the player fights virtual enemies with kicks and punches as well as acrobatic moves such as cartwheels. Using real-time image processing and computer vision, the video image of the user is embedded inside 3D graphics. Compared to previous work, our system uses a profile view and two displays, which allows an improved view of many martial arts techniques. We also explore exaggerated motion and dynamic slow-motion effects to transform the aesthetic of kung-fu movies into an interactive, embodied experience. The system is described and analyzed based on results from testing the game in a theater, in a television show, and in a user study with 46 martial arts practitioners.

This paper describes the use of the Wizard of Oz (WOz) method in the design of computer vision based action games controlled with body movements. A WOz study was carried out with 34 children of ages 7 to 9 in order to find out the most intuitive movements for game controls and to evaluate the relationship between avatar and player actions. Our study extends the previous Wizard of Oz studies by showing that WOz prototyping of perceptive action games is feasible despite the delay caused by the wizard. The results also show that distinctive movement categories and gesture patterns can be found by observing the children playing games controlled by a human wizard. The approach minimizes the need for fully functional prototypes in the early stages of the design and provides video material for testing and developing computer vision algorithms, as well as guidelines for animating the game character.

This paper describes Animaatiokone, an installation for experimenting and learning about stop-motion animation. Located in a movie theater, it allows people to create clay animation while waiting for a movie. Collaboration between users is supported, for example, by sharing of clay actors. The installation's user interface allows even beginners to create and edit animation with help of automatic onion-skinning and simple controls developed through iterative testing and prototyping. In test use, the installation has been popular and hundreds of animations have been created and made available via the installation's homepage http://www.animaatiokone.net

In this paper, we present two game tasks (flying game and spider game) in QuiQui's Giant Bounce, a computer game based on computer vision and hearing technology. The game consists of several game tasks with different themes of movement. The tasks presented here have been designed and tested with children [2] and experts in the fields of children's physical, social and cognitive development.

This paper presents a novel approach to usability evaluation with children called peer tutoring . Peer tutoring means that children teach other children to use the software that is evaluated. The basic philosophy behind this is to view software as a part of child's play, so that the teaching process is analogous to explaining the rules of a game such as hide and seek. If the software is easy to teach and learn, it is more likely that the amount of users increases in a social setting such as a school. The peer tutoring approach provides information about teachability and learnability of software and it also promotes communication in the test situation, compared to a test person communicating with an adult instructor. The approach has been applied to the development of a perceptually interactive user interface in QuiQui's Giant Bounce, a physically and vocally interactive computer game for 4–9 year old children. The results and experiences of using peer tutoring are promising and it has proved to be effective in detecting usability flaws and in improving the design of the game.