Abstract Human motion is fundamentally driven by continuous physical interaction with the environment. Whether walking, running, or simply standing, the forces exchanged between our feet and the ground provide crucial insights for understanding and reconstructing human movement. Recent advances in wearable insole devices offer a compelling solution for capturing these forces in diverse, real‐world scenarios. Sensor insoles pose no constraint on the users' motion (unlike mocap suits) and are unaffected by line‐of‐sight limitations (in contrast to optical systems). These qualities make sensor insoles an ideal choice for robust, unconstrained motion capture, particularly in outdoor environments. Surprisingly, leveraging these devices with recent motion reconstruction methods remains largely unexplored. Aiming to fill this gap, we present Step2Motion , the first approach to reconstruct human locomotion from multi‐modal insole sensors. Our method utilizes pressure and inertial data—accelerations and angular rates—captured by the insoles to reconstruct human motion. We evaluate the effectiveness of our approach across a range of experiments to show its versatility for diverse locomotion styles, from simple ones like walking or jogging up to moving sideways, on tiptoes, slightly crouching, or dancing. The complete source code, trained model, data, and supplementary material used in this paper can be found at: https://vcai.mpi-inf.mpg.de/projects/Step2Motion/

We present a reinforcement learning framework for crowd simulation that balances collision avoidance with navigation preferences guided by soft environmental constraints. Our approach integrates two complementary perception components: entity perception, which handles hard constraints imposed by physical obstacles, and implicit environmental perception, encoded as suitability maps that guide movement preferences due to soft constraints. To ensure robust generalization across complex scenes, we employ a modular, two-phase training strategy utilizing curriculum-based environmental templates. The proposed framework functions as an intuitive crowd authoring tool: artists can influence crowd behavior in real time by interactively editing suitability maps or dynamically adjusting perception weights, either globally or per-agent. This enables adaptive navigation behaviors, such as dynamic trajectory prioritization based on the environment, to emerge from local interactions. We validate our approach using both quantitative metrics and qualitative analysis, including a user study confirming that the resulting behaviors align with pedestrian traffic patterns observed in real-world settings. We further present an example of emergent social behavior through the formation and gradual evolution of desire paths. This work contributes to the field of crowd simulation by offering a robust, learning-based framework that supports heterogeneous navigation modeling and intuitive authoring for interactive applications.

Rule-breaking behaviors, such as jaywalking or skipping queues, are common in crowds but difficult to study in real-world settings due to limited control and observability. Virtual reality (VR) provides a controlled alternative, but its validity depends on whether VR elicits realistic rule-breaking behavior. We conducted a VR study with 65 participants navigating a virtual city with four scenarios differing in social norms: walking on grass, crossing outside a crosswalk, jaywalking at a red light, and skipping a line. In each scenario, the proportion of rule-breaking virtual characters was manipulated (0%, 25%, 50%). Participants' movements and gaze were recorded to assess behavior and attention. Results showed higher rule-breaking as the number of violators increased, except in the low-stakes crossing scenario. Rule-breakers attended more to violating characters, whereas rule-followers focused on compliant ones. Participants cited efficiency, safety, and social norms as key factors guiding their decisions. Overall, VR reproduced natural patterns of social compliance and noncompliance, supporting its use for studying rule-breaking and applications in crowd simulation, urban design, safety training, and immersive media.

Current crowd simulation models focus mostly on steering towards a goal while avoiding collisions based on the agent’s direction of movement. This leads to robot-like simulations since agents’ appear to always have their attention perfectly aligned with the direction of movement. In the real world, we observe that humans move in a crowd performing collision avoidance driven by attention, gaze, and non-verbal coordination with incoming traffic. In addition, humans exhibit different steering strategies based on whether they walk alone or in a group, whether they can look ahead and plan their best local movement, or react more abruptly because their gaze diverts from their direction of movement. Human gaze can be driven by movement, but also by distractions such as being engaged in conversation with other people or using mobile phones. These human features are overlooked in crowd simulation, often leading to perfectly smooth local movements of individuals. Unfortunately, this lack of social behaviour and variety in animations may be perceived as unrealistic when observing the results on a 2D display, and it may become even more apparent in immersive scenarios where the participant is at eye level with the virtual humans. This paper proposes a novel approach to enhance the realism of a rule-based crowd simulation model by incorporating social rules and gaze-driven attention with consistent animations. The ultimate goal is to make immersive virtual crowds more realistic. Our proposed method enhances existing crowd simulation frameworks by integrating social behavior models that affect both individual and collective dynamics, and drives gaze behavior to better simulate attention. We conducted validation user studies on both a 2D display and in immersive VR, and observed that applying these models to both the steering and animation levels significantly improves the realism of the crowd simulation. The 2D display based user study based on video comparisons showed that our model was perceived as more realistic and consistent with social behaviors compared to traditional collision avoidance approaches, that used only locomotion or random animations. The immersive user study showed that participants effectively detected the social behaviours included in our model as intended. The results revealed significant differences in the participants’ perceptions of the various behaviours exhibited by our social crowd model.

Abstract Simulating believable crowds for applications like movies or games is challenging due to the many components that comprise a realistic outcome. Users typically need to manually tune a large number of simulation parameters until they reach the desired results. We introduce MPACT, a framework that leverages image‐based encoding to convert unlabelled crowd data into meaningful and controllable parameters for crowd generation. In essence, we train a parameter prediction network on a diverse set of synthetic data, which includes pairs of images and corresponding crowd profiles. The learned parameter space enables: (a) implicit crowd authoring and control, allowing users to define desired crowd scenarios using real‐world trajectory data, and (b) crowd analysis, facilitating the identification of crowd behaviours in the input and the classification of unseen scenarios through operations within the latent space. We quantitatively and qualitatively evaluate our framework, comparing it against real‐world data and selected baselines, while also conducting user studies with expert and novice users. Our experiments show that the generated crowds score high in terms of simulation believability, plausibility and crowd behaviour faithfulness.

Abstract High‐quality motion reconstruction that follows the user's movements can be achieved by high‐end mocap systems with many sensors. However, obtaining such animation quality with fewer input devices is gaining popularity as it brings mocap closer to the general public. The main challenges include the loss of end‐effector accuracy in learning‐based approaches, or the lack of naturalness and smoothness in IK‐based solutions. In addition, such systems are often finely tuned to a specific number of trackers and are highly sensitive to missing data, e.g., in scenarios where a sensor is occluded or malfunctions. In response to these challenges, we introduce DragPoser, a novel deep‐learning‐based motion reconstruction system that accurately represents hard and dynamic constraints, attaining real‐time high end‐effectors position accuracy. This is achieved through a pose optimization process within a structured latent space. Our system requires only one‐time training on a large human motion dataset, and then constraints can be dynamically defined as losses, while the pose is iteratively refined by computing the gradients of these losses within the latent space. To further enhance our approach, we incorporate a Temporal Predictor network, which employs a Transformer architecture to directly encode temporality within the latent space. This network ensures the pose optimization is confined to the manifold of valid poses and also leverages past pose data to predict temporally coherent poses. Results demonstrate that DragPoser surpasses both IK‐based and the latest data‐driven methods in achieving precise end‐effector positioning, while it produces natural poses and temporally coherent motion. In addition, our system showcases robustness against on‐the‐fly constraint modifications, and exhibits adaptability to various input configurations and changes. The complete source code, trained model, animation databases, and supplementary material used in this paper can be found at https://upc-virvig.github.io/DragPoser

Interactive applications demand believable characters that respond naturally to dynamic environments. Traditional character animation techniques often struggle to handle arbitrary situations, leading to a growing trend of dynamically selecting motion-captured animations based on predefined features. While Motion Matching has proven effective for locomotion by aligning to target trajectories, animating environment interactions and crowd behaviors remains challenging due to the need to consider surrounding elements. Existing approaches often involve manual setup or lack the naturalism of motion capture. Furthermore, in crowd animation, body animation is frequently treated as a separate process from trajectory planning, leading to inconsistencies between body pose and root motion. To address these limitations, we present Environment-aware Motion Matching , a novel real-time system for full-body character animation that dynamically adapts to obstacles and other agents, emphasizing the bidirectional relationship between pose and trajectory. In a preprocessing step, we extract shape, pose, and trajectory features from a motion capture database. At runtime, we perform an efficient search that matches user input and current pose while penalizing collisions with a dynamic environment. Our method allows characters to naturally adjust their pose and trajectory to navigate crowded scenes.

Immersive Virtual Reality typically requires a head-mounted display (HMD) to visualize the environment and hand-held controllers to interact with the virtual objects. Recently, many applications display full-body avatars to represent the user and animate the arms to follow the controllers. Embodiment is higher when the self-avatar movements align correctly with the user. However, having a full-body self-avatar following the user’s movements can be challenging due to the disparities between the virtual body and the user’s body. This can lead to misalignments in the hand position that can be noticeable when interacting with virtual objects. In this work, we propose five different interaction modes to allow the user to interact with virtual objects despite the self-avatar and controller misalignment and study their influence on embodiment, proprioception, preference, and task performance. We modify aspects such as whether the virtual controllers are rendered, whether controllers are rendered in their real physical location or attached to the user’s hand, and whether stretching the avatar arms to always reach the real controllers. We evaluate the interaction modes both quantitatively (performance metrics) and qualitatively (embodiment, proprioception, and user preference questionnaires). Our results show that the stretching arms solution, which provides body continuity and guarantees that the virtual hands or controllers are in the correct location, offers the best results in embodiment, user preference, proprioception, and performance. Also, rendering the controller does not have an effect on either embodiment or user preference.

This paper proposes a novel approach to enhance the realism of a rule-based crowd simulation model by incorporating social rules and gaze-driven attention with consistent animations. Current crowd simulation models focus mostly on steering towards a goal while avoiding collision based on the agent’s direction of movement. This leads to robot-like simulations since agents’ appear to always have their attention perfectly aligned with the direction of movement. In the real world we observe that humans move in a crowd performing collision avoidance driven by attention, gaze, and non-verbal coordination with incoming traffic. In addition, humans exhibit different steering strategies based on whether they walk alone or in a group, whether they can look ahead and plan their best local movement, or react more abruptly because their gaze diverts from their direction of movement. Human gaze can be driven by movement but also by distractions such as being engaged in conversation with other people or using mobile phones. These human features are overlooked in crowd simulation leading often to perfectly smooth local movements of individuals. Our proposed method enhances existing crowd simulation frameworks by integrating social behavior models that affect both individual and collective dynamics, and drives gaze behavior to better simulate attention. By applying these models at both the steering and animation levels, we significantly improve the realism of the crowd simulation. A user study shows that our model was perceived as being more realistic and consistent with social behaviors when compared to a more traditional collision avoidance with just locomotion or with random animations.

An increasing number of virtual reality applications require environments that emulate real-world conditions. These environments often involve dynamic virtual humans showing realistic behaviors. Understanding user perception and navigation among these virtual agents is key for designing realistic and effective environments featuring groups of virtual humans. While collision risk significantly influences human locomotion in the real world, this risk is largely absent in virtual settings. This paper studies the impact of the expected collision feedback on user perception and interaction with virtual crowds. We examine the effectiveness of commonly used collision feedback techniques (auditory cues and tactile vibrations) as well as inducing participants to expect that a physical bump with a real person might occur, as if some virtual humans actually correspond to real persons embodied into them and sharing the same physical space. Our results indicate that the expected collision feedback significantly influences both participant behavior—encompassing global navigation and local movements—and subjective perceptions of presence and copresence. Specifically, the introduction of a perceived risk of actual collision was found to significantly impact global navigation strategies and increase the sense of presence. Auditory cues had a similar effect on global navigation and additionally enhanced the sense of copresence. In contrast, vibrotactile feedback was primarily effective in influencing local movements.

Accurate and reliable human motion reconstruction is crucial for creating natural interactions of full-body avatars in Virtual Reality (VR) and entertainment applications. As the Metaverse and social applications gain popularity, users are seeking cost-effective solutions to create full-body animations that are comparable in quality to those produced by commercial motion capture systems. In order to provide affordable solutions though, it is important to minimize the number of sensors attached to the subject’s body. Unfortunately, reconstructing the full-body pose from sparse data is a heavily under-determined problem. Some studies that use IMU sensors face challenges in reconstructing the pose due to positional drift and ambiguity of the poses. In recent years, some mainstream VR systems have released 6-degree-of-freedom (6-DoF) tracking devices providing positional and rotational information. Nevertheless, most solutions for reconstructing full-body poses rely on traditional inverse kinematics (IK) solutions, which often produce non-continuous and unnatural poses. In this article, we introduce SparsePoser, a novel deep learning-based solution for reconstructing a full-body pose from a reduced set of six tracking devices. Our system incorporates a convolutional-based autoencoder that synthesizes high-quality continuous human poses by learning the human motion manifold from motion capture data. Then, we employ a learned IK component, made of multiple lightweight feed-forward neural networks, to adjust the hands and feet toward the corresponding trackers. We extensively evaluate our method on publicly available motion capture datasets and with real-time live demos. We show that our method outperforms state-of-the-art techniques using IMU sensors or 6-DoF tracking devices, and can be used for users with different body dimensions and proportions.

Abstract Human and animal presence in natural landscapes is initially revealed by the immediate impact of their locomotion, from footprints to crushed grass. In this work, we present an approach to model the effects of virtual characters on natural terrains, focusing on the impact of human locomotion. We introduce a lightweight solution to compute accurate foot placement on uneven ground and infer dynamic foot pressure from kinematic animation data and the mass of the character. A ground and vegetation model enables us to effectively simulate the local impact of locomotion on soft soils and plants over time, resulting in the formation of visible paths. As our results show, we can parameterize various soil materials and vegetation types validated with real-world data. Our method can be used to significantly increase the realism of populated natural landscapes and the sense of presence in virtual applications and games.

The use of self-avatars is gaining popularity thanks to affordable VR headsets. Unfortunately, mainstream VR devices often use a small number of trackers and provide low-accuracy animations. Previous studies have shown that the Sense of Embodiment, and in particular the Sense of Agency, depends on the extent to which the avatar's movements mimic the user's movements. However, few works study such effect for tasks requiring a precise interaction with the environment, i.e., tasks that require accurate manipulation, precise foot stepping, or correct body poses. In these cases, users are likely to notice inconsistencies between their self-avatars and their actual pose. In this paper, we study the impact of the animation fidelity of the user avatar on a variety of tasks that focus on arm movement, leg movement and body posture. We compare three different animation techniques: two of them using Inverse Kinematics to reconstruct the pose from sparse input (6 trackers), and a third one using a professional motion capture system with 17 inertial sensors. We evaluate these animation techniques both quantitatively (completion time, unintentional collisions, pose accuracy) and qualitatively (Sense of Embodiment). Our results show that the animation quality affects the Sense of Embodiment. Inertial-based MoCap performs significantly better in mimicking body poses. Surprisingly, IK-based solutions using fewer sensors outperformed MoCap in tasks requiring accurate positioning, which we attribute to the higher latency and the positional drift that causes errors at the end-effectors, which are more noticeable in contact areas such as the feet.

Simulating crowds with realistic behaviors is a difficult but very important task for a variety of applications. Quantifying how a person balances between different conflicting criteria such as goal seeking, collision avoidance and moving within a group is not intuitive, especially if we consider that behaviors differ largely between people. Inspired by recent advances in Deep Reinforcement Learning, we propose Guided REinforcement Learning (GREIL) Crowds, a method that learns a model for pedestrian behaviors which is guided by reference crowd data. The model successfully captures behaviors such as goal seeking, being part of consistent groups without the need to define explicit relationships and wandering around seemingly without a specific purpose. Two fundamental concepts are important in achieving these results: (a) the per agent state representation and (b) the reward function. The agent state is a temporal representation of the situation around each agent. The reward function is based on the idea that people try to move in situations/states in which they feel comfortable in. Therefore, in order for agents to stay in a comfortable state space, we first obtain a distribution of states extracted from real crowd data; then we evaluate states based on how much of an outlier they are compared to such a distribution. We demonstrate that our system can capture and simulate many complex and subtle crowd interactions in varied scenarios. Additionally, the proposed method generalizes to unseen situations, generates consistent behaviors and does not suffer from the limitations of other data-driven and reinforcement learning approaches.

Abstract Recent advancements in crowd simulation unravel a wide range of functionalities for virtual agents, delivering highly‐realistic, natural virtual crowds. Such systems are of particular importance to a variety of applications in fields such as: entertainment (e.g., movies, computer games); architectural and urban planning; and simulations for sports and training. However, providing their capabilities to untrained users necessitates the development of authoring frameworks. Authoring virtual crowds is a complex and multi‐level task, varying from assuming control and assisting users to realise their creative intents, to delivering intuitive and easy to use interfaces, facilitating such control. In this paper, we present a categorisation of the authorable crowd simulation components, ranging from high‐level behaviours and path‐planning to local movements, as well as animation and visualisation. We provide a review of the most relevant methods in each area, emphasising the amount and nature of influence that the users have over the final result. Moreover, we discuss the currently available authoring tools (e.g., graphical user interfaces, drag‐and‐drop), identifying the trends of early and recent work. Finally, we suggest promising directions for future research that mainly stem from the rise of learning‐based methods, and the need for a unified authoring framework.

Abstract The animation of user avatars plays a crucial role in conveying their pose, gestures, and relative distances to virtual objects or other users. Self‐avatar animation in immersive VR helps improve the user experience and provides a Sense of Embodiment. However, consumer‐grade VR devices typically include at most three trackers, one at the Head Mounted Display (HMD), and two at the handheld VR controllers. Since the problem of reconstructing the user pose from such sparse data is ill‐defined, especially for the lower body, the approach adopted by most VR games consists of assuming the body orientation matches that of the HMD, and applying animation blending and time‐warping from a reduced set of animations. Unfortunately, this approach produces noticeable mismatches between user and avatar movements. In this work we present a new approach to animate user avatars that is suitable for current mainstream VR devices. First, we use a neural network to estimate the user's body orientation based on the tracking information from the HMD and the hand controllers. Then we use this orientation together with the velocity and rotation of the HMD to build a feature vector that feeds a Motion Matching algorithm. We built a MoCap database with animations of VR users wearing a HMD and used it to test our approach on both self‐avatars and other users' avatars. Our results show that our system can provide a large variety of lower body animations while correctly matching the user orientation, which in turn allows us to represent not only forward movements but also stepping in any direction.

Achieving realistic virtual humans is crucial in virtual reality applications and video games. Nowadays there are software and game development tools, that are of great help to generate and simulate characters. They offer easy to use GUIs to create characters by dragging and drooping features, and making small modifications. Similarly, there are tools to create animation graphs and setting blending parameters among others. Unfortunately, even though these tools are relatively user friendly, achieving natural animation transitions is not straight forward and thus non-expert users tend to spend a large amount of time to generate animations that are not completely free of artefacts. In this paper we present a method to automatically generate animation blend spaces in Unreal engine, which offers two advantages: the first one is that it provides a tool to evaluate the quality of an animation set, and the second one is that the resulting graph does not depend on user skills and it is thus not prone to user errors.

This paper presents a virtual reality experiment in which two participants share both the virtual and the physical space while performing a collaborative task. We are interested in studying what are the differences in human locomotor behavior between the real world and the VR scenario. For that purpose, participants performed the experiment in both the real and the virtual scenarios. For the VR case, participants can see both their own animated avatar and the avatar of the other participant in the environment. As they move, we store their trajectories to obtain information regarding speeds, clearance distances and task completion times. For the VR scenario, we also wanted to evaluate whether the users were aware of subtle differences in the avatar's animations and foot steps sounds. We ran the same experiment under three different conditions: (1) synchronizing the avatar's feet animation and sound of footsteps with the movement of the participant; (2) synchronizing the animation but not the sound and finally (3) not synchronizing either one. The results show significant differences in user's presence questionnaires and also different trends in their locomotor behavior between the real world and the VR scenarios. However the subtle differences in animations and sound tested in our experiment had no impact on the results of the presence questionnaires, although it showed a small impact on their locomotor behavior in terms of time to complete their tasks, and clearance distances kept while crossing paths.

The challenge of path-finding in video games is to compute optimal or near optimal paths as efficiently as possible. As both the size of the environments and the number of autonomous agents increase, this computation has to be done under hard constraints of memory and CPU resources. Hierarchical approaches, such as HNA* can compute paths more efficiently, although only for certain configurations of the hierarchy. For other configurations, performance can drop drastically when inserting the start and goal position into the hierarchy. In this paper we present improvements to HNA* to eliminate bottlenecks. We propose different methods that rely on further memory storage or parallelism on both CPU and GPU, and carry out a comparative evaluation. Results show an important speed-up for all tested configurations and scenarios.

A navigation mesh is a representation of a 2D or 3D virtual environment that enables path planning and crowd simulation for walking characters. Various state-of-the-art navigation meshes exist, but there is no standardized way of evaluating or comparing them. Each implementation is in a different state of maturity, has been tested on different hardware, uses different example environments, and may have been designed with a different application in mind.

Abstract In this survey we review, classify and compare existing approaches for real‐time crowd rendering. We first overview character animation techniques, as they are highly tied to crowd rendering performance, and then we analyze the state of the art in crowd rendering. We discuss different representations for level‐of‐detail (LoD) rendering of animated characters, including polygon‐based, point‐based, and image‐based techniques, and review different criteria for runtime LoD selection. Besides LoD approaches, we review classic acceleration schemes, such as frustum culling and occlusion culling, and describe how they can be adapted to handle crowds of animated characters. We also discuss specific acceleration techniques for crowd rendering, such as primitive pseudo‐instancing, palette skinning, and dynamic key‐pose caching, which benefit from current graphics hardware. We also address other factors affecting performance and realism of crowds such as lighting, shadowing, clothing and variability. Finally we provide an exhaustive comparison of the most relevant approaches in the field.

There are two frequent artifacts in crowd simulation, the first one appears when all agents attempt to traverse the navigation mesh sharing the same way point over portals, increasing the probability of collision against other agents and lining up towards portals; the second one is caused by way points being assigned at locations where clearance is not guaranteed which causes the agents to walk too close to the static geometry, slide along walls or even get stuck. In this work we propose a novel method for dynamically calculating way points based on current trajectory, destination, and clearance while using the full length of the portal, thus guaranteeing that agents in a crowd will have different way points assigned. To guarantee collision free paths we propose two novel techniques: the first one provides the computation of paths with clearance for cells of any shape (even with concavities) and the second one presents a new method for calculating portals with clearance, so that the dynamically assigned way points will always guarantee collision free paths. We evaluate our results with a variety of scenarios, and compare our results against traditional way points at the center of portals to show that our technique offers a better use of the space by the agents, as well as a reduction in the number of collisions.

ABSTRACT In this paper, we present a new impostor‐based representation for 3D animated characters supporting real‐time rendering of thousands of agents. We maximize rendering performance by using a collection of pre‐computed impostors sampled from a discrete set of view directions. Our approach differs from previous work on view‐dependent impostors in that we use per‐joint rather than per‐character impostors. Our characters are animated by applying the joint rotations directly to the impostors, instead of choosing a single impostor for the whole character from a set of pre‐defined poses. This offers more flexibility in terms of animation clips, as our representation supports any arbitrary pose, and thus, the agent behavior is not constrained to a small collection of pre‐defined clips. Because our impostors are intended to be valid for any pose, a key issue is to define a proper boundary for each impostor to minimize image artifacts while animating the agents. We pose this problem as a variational optimization problem and provide an efficient algorithm for computing a discrete solution as a pre‐process. To the best of our knowledge, this is the first time a crowd rendering algorithm encompassing image‐based performance, small graphics processing unit footprint, and animation independence is proposed. Copyright © 2012 John Wiley & Sons, Ltd.

Abstract Rendering detailed animated characters is a major limiting factor in crowd simulation. In this paper we present a new representation for 3D animated characters which supports output‐sensitive rendering. Our approach is flexible in the sense that it does not require us to pre‐define the animation sequences beforehand, nor to pre‐compute a dense set of pre‐rendered views for each animation frame. Each character is encoded through a small collection of textured boxes storing colour and depth values. At runtime, each box is animated according to the rigid transformation of its associated bone and a fragment shader is used to recover the original geometry using a dual‐depth version of relief mapping. Unlike competing output‐sensitive approaches, our compact representation is able to recover high‐frequency surface details and reproduces view‐motion parallax effectively. Our approach drastically reduces both the number of primitives being drawn and the number of bones influencing each primitive, at the expense of a very slight per‐fragment overhead. We show that, beyond a certain distance threshold, our compact representation is much faster to render than traditional level‐of‐detail triangle meshes. Our user study demonstrates that replacing polygonal geometry by our impostors produces negligible visual artefacts.