International audience

This paper focuses on the problem of online action detection for interactive systems, with a special emphasis on earliness. Online Action Detection (OAD) refers to the challenging task of recognizing gestures in untrimmed, streaming videos where the actions occur in unpredictable orders and durations. To address these challenges, we present a skeleton-based system for OAD incorporating a decision mechanism to accurately detect ongoing gestures. This allows us to provide instance-level output, achieving a high level of stream understanding. This mechanism relies on a novel Connectionist Temporal Classification (CTC) loss design that restricts the path possibilities according to the action boundaries. We also present a mechanism to tune the trade-off between accuracy and earliness according to the needs of the interactive system using a weighted label prior. This system includes a 3D CNN network, referred to as DOLT-C3D, exploiting the spatial–temporal information provided by the euclidean skeleton representation. We extensively evaluate our approach on eight publicly available datasets, demonstrating its superior performance compared to state-of-the-art methods in terms of both accuracy and earliness. We also successfully applied our approach to early 2D gestures detection. Furthermore, our system shows real-time performance, making it a suitable choice for interactive systems.

Abstract Virtual reality (VR) has recently seen significant development in interaction with computers and the visualization of information. More and more people are using virtual and immersive technologies in their daily lives, especially for entertainment, fitness, and socializing purposes. This paper presents a qualitative evaluation of a large sample of users using a VR platform for dancing ( $$N=292$$ N = 292 ); we study the users’ motivations, experiences, and requirements for using VR as an inclusive platform for dancing, mainly as a social or physical activity. We used an artificial intelligence platform (OpenAI) to extract categories or clusters of responses automatically. We organized the data into six user motivation categories: fun, fitness, social activity, pandemic, escape from reality, and professional activities. Our results indicate that dancing in virtual reality is a different experience than in the real world, and there is a clear distinction in the user’s motivations for using VR platforms for dancing. Our survey results suggest that VR is a tool that can positively impact physical and mental well-being through dancing. These findings complement the related work, help in identifying the use cases, and can be used to assist future improvements of VR dance applications.

Early recognition of untrimmed handwritten gestures is the task of recognizing as soon as possible gestures drawn in a continuous stream, one after another. This is particularly challenging for multi-touch gestures because it is impossible to know when the gesture has started and finished. For mono-stroke gestures, in an application context where the finger is never removed from the device between gestures, the recognition is even more complex. In this work we present an extension of the Online Long-Term Convolutional 3D (OLT-C3D) network to address the task of early recognition of untrimmed gestures which have been addressed by very few works. To evaluate our approach, we created two synthetic datasets using freely available benchmarks, MTGSetB and ILGDB, simulating the streaming data in two different application scenarios. Furthermore, we propose a new evaluation metric for this specific task. Our approach achieves good performances on the two new datasets and will be a baseline for future works on this challenging task.

People generally coordinate their action to be more effective. However, in some cases, interference between them occur, resulting in an inefficient collaboration. The main goal of this study is to explore the way two persons regulate their actions when performing a cooperative task of ball interception, and how interference between them may occur. Starting face to face, twenty-four participants (twelve teams of two) had to physically intercept balls moving down from the roof to the floor in a virtual room. To this end, they controlled a virtual paddle attached to their hand moving along the anterior-posterior axis, and were not allowed to communicate. Results globally showed participants were often able to intercept balls without collision by dividing the interception space in two equivalent parts. However, an area of uncertainty (where many trials were not intercepted) appeared in the center of the scene highlighting the presence of interference between participants. The width of this area increased when situation became more complex and when less information was available. Moreover, participants often interpreted balls starting above them as balls they should intercept, even when these balls were in fine intercepted by their partner. Overall, results showed that team coordination emerges from between-participants interactions in this ball interception task and that interference between them depends on task complexity (uncertainty on partner's action and visual information available).

The recent development of virtual reality (VR) devices such as head mounted displays (HMDs) increases opportunities for applications at the confluence of physical activity and gaming. Recently, the fields of sport and fitness have turned to VR, including for locomotor activities, to enhance motor and energetic resources, as well as motivation and adherence. For example, VR can provide visual feedbacks during treadmill running, thereby reducing monotony and increasing the feeling of movement and engagement with the activity. However, the relevance of using VR tools during locomotion depends on the ability of these systems to provide natural immersive feelings, specifically a coherent perception of speed. The objective of this study is to estimate the error between actual and perceived locomotor speed in VE using an enactive approach, i.e. allowing an active control of the environment. Sixteen healthy individuals participated in the experiment, which consisted in walking and running on a motorized treadmill at speeds ranging from 3 to 11 km/h with 0.5 km/h increments, in a randomized order while wearing a HMD device (HTC Vive) displaying a virtual racetrack. Participants were instructed to match VE speed with what they perceived was their ac-tuallocomotion speed (LS), using a handheld Vive controller. They were able to modify the optic flow speed (OFS) with a 0.02 km/h increment/decrement accuracy. An optic flow multiplier (OFM) was computed based on the error between OFS and LS. It represents the gain that exists between the visually perceived speed and the real locomotion speed experienced by participants for each trial. For all conditions, the average of OFM was 1.00±.25 to best match LS. This finding is at odds with previous works reporting an underestimation of speed perception in VR. It could be explained by the use of an enactive approach allowing an active and accurate matching of visually and proprioceptively perceived speeds by participants. But above all, our study showed that the perception of speed in VR is strongly individual, with some participants always overestimating and others constantly underestimating. Therefore, a general OFM should not be used to correct speed in VE to ensure congruence in speed perception, and we propose the use of individual models as recommendations for setting up locomotion-based VR applications.

The selection of relevant features in large databases is one of the most important and challenging problems in data mining.Samples forming a given database are generally described by a predefined set of features, and the situation where not all such features can be used for classification purposes needs very often to be faced in real applications.This situation is very typical when the database is related to a phenomenon whose characteristics are not well known.In this context, the extraction of relevant features can therefore also provide additional information on the studied phenomena.We tackle the feature selection problem from an optimization point of view, by reducing it to the problem of finding a maximal consistent "clustering" grouping together the samples and the features of the database.In this work, we extend this approach to dynamical databases, where features are not represented by only one real value, but they are rather given as sequences of a predefined number of real values.Our main contribution consists in proposing an alternative representation of the database so that it fits with a tridimensional matrix with no missing entries, from which a consistent triclustering can be obtained.

This paper presents a study performed in virtual reality on the effect of gaze interception during collision avoidance between two walkers. In such a situation, mutual gaze can be considered as a form of nonverbal communication. Additionally, gaze is believed to detail future path intentions and to be part of the nonverbal negotiation to achieve avoidance collaboratively. We considered an avoidance task between a real subject and a virtual human character and studied the influence of the character's gaze direction on the avoidance behaviour of the participant. Virtual reality provided an accurate control of the situation: seventeen participants were immersed in a virtual environment, instructed to navigate across a virtual space using a joystick and to avoid a virtual character that would appear from either side. The character would either gaze or not towards the participant. Further, the character would either perform or not a reciprocal adaptation of its trajectory to avoid a potential collision with the participant. The findings of this paper were that during an orthogonal collision avoidance task, gaze behaviour did not influence the collision avoidance behaviour of the participants. Further, the addition of reciprocal collision avoidance with gaze did not modify the collision behaviour of participants. These results suggest that for the duration of interaction in such a task, body motion cues were sufficient for coordination and regulation. We discuss the possible exploitation of these results to improve the design of virtual characters for populated virtual environments and user interaction.

Daily activities require agents to interact with each other, such as during collision avoidance. The nature of visual information that is used for a collision free interaction requires further understanding. We aim to manipulate the nature of visual information in two forms, global and local information appearances. Sixteen healthy participants navigated towards a target in an immersive computer-assisted virtual environment (CAVE) using a joystick. A moving passive obstacle crossed the participant's trajectory perpendicularly at various pre-defined risks of collision distances. The obstacle was presented with one of five virtual appearances, associated to global motion cues (i.e., a cylinder or a sphere), or local motion cues (i.e., only the legs or the trunk). A full body virtual walker, showing both local and global motion cues, used as a reference condition. The final crossing distance was affected by the global motion appearances, however, appearance had no qualitative effect on motion adaptations. These findings contribute towards further understanding what information people use when interacting with others.

Navigating in virtual environments requires using some locomotion interfaces, especially when the dimensions of the environment exceed the ones of the Virtual Reality system. Locomotion interfaces induce some biases both in the perception of the self-motion or in the formation of virtual locomotion trajectories. These biases have been mostly evaluated in the context of static environments, and studies need to be revisited in the new context of populated environments where users interact with virtual characters. We focus on a situation of collision avoidance between a real participant and a virtual character, and compared it to previous studies on real walkers. Our results show that, as in reality, the risk of future collision is accurately anticipated by participants, however with delay. We also show that collision avoidance trajectories formed in VR have common properties with real ones, with some quantitative differences in avoidance distances. More generally, our evaluation demonstrates that reliable results can be obtained for qualitative analysis of small scale interactions in VR. We discuss these results in the perspective of a VR platform for large scale interaction applications, such as in a crowd, for which real data are difficult to gather.

Exergames involve using the fullbody to interact with an immersive world, which raises the challenge of capturing, processing and recognizing the action of the user even for cheap mocap systems such as the Microsoft Kinect. In fact, these recent technological advances have renewed interest in skeleton-based action recognition. Our review of related literature reveals that the issues encountered are not the result of random processes, which could simply be studied by using statistical tools, but are instead due to the fact that the pattern to be recognized, i.e. an action, was produced by a human being. 2D hand-drawn symbols are further examples of patterns resulting from a human motion. Therefore, the main contribution of this paper is to examine the validity of transferring the expertise of hand-drawn symbol representation to better recognize actions based on skeleton data. Principally, we propose a new action representation, namely the 3DMM, as an initial case-study illustrating how such transfer could be conducted. The experimental results, obtained over two benchmarks, confirm the soundness of our approach and encourage more thorough examination of the transfer.

Action recognition based on human skeleton structure represents nowadays a prosper research field. This is mainly due to the recent advances in terms of capture technologies and skeleton extraction algorithms. In this context, we observed that 3D skeleton-based actions share several properties with handwritten symbols since they both result from a human performance. We accordingly hypothesize that the action recognition problem can take advantage of trial and error already carried out on handwritten patterns. Therefore, inspired by one of the most efficient and compact handwriting feature-set, we propose in this paper a skeleton descriptor referred to as Handwriting-Inspired Features (HIF3D). First of all a data preprocessing is applied to joint trajectories in order to handle the variabilities among actor's morphologies. Then we extract the HIF3D features from the processed joint locations according to a time partitioning scheme so as to additionally encode the temporal information over the sequence. Finally, we selected the Support Vector Machine (SVM) to achieve the classification step. Evaluations conducted on two challenging datasets, namely HDM05 and UTKinect, testify the soundness of our approach as the obtained results outperform the state-of-the-art algorithms that rely on skeleton data.

A typical crowd engine pipeline animates numerous moving characters according to a two-step process: global trajectories are generated by a crowd simulator, whereas full body motions are generated by animation engines. Because interactions are only considered at the first stage, animations sometimes lead to residual collisions and/or characters walking as if they were alone, showing no sign to the influence of others. In this paper, we investigate the value of adding shoulder motions to characters passing at close distances on the perceived visual quality of crowd animations (i.e., perceived residual collisions and animation naturalness). We present two successive perceptual experiments exploring this question where we investigate first, local interactions between two isolated characters, and second, crowd scenarios. The first experiment shows that shoulder motions have a strong positive effect on both perceived residual collisions and animation naturalness. The second experiment demonstrates that the effect of shoulder motions on animation naturalness is preserved in the context of crowd scenarios, even though the complexity of the scene is largely increased. Our general conclusion is that adding secondary motions in character interactions has a significant impact on the visual quality of crowd animations, with a very light impact on the computational cost of the whole animation pipeline. Our results advance crowd animation techniques by enhancing the simulation of complex interactions between crowd characters with simple secondary motion triggering techniques.

Recognition of natural gestures is a key issue in many applications including videogames and other immersive applications. Whatever is the motion capture device, the key problem is to recognize a motion that could be performed by a range of different users, at an interactive frame rate. Hidden Markov Models (HMM) that are commonly used to recognize the performance of a user however rely on a motion representation that strongly affects the overall recognition rate of the system. In this paper, we propose to use a compact motion representation based on Morphology-Independent features and we evaluate its performance compared to classical representations. When dealing with 15 very similar upper limb motions, HMM based on Morphology-Independent features yield significantly higher recognition rate (84.9%) than classical Cartesian or angular data (70.4% and 55.0%, respectively). Moreover, when the unknown motions are performed by a large number of users who have never contributed to the learning process, the recognition rate of Morphology-Independent input feature only decreases slightly (down to 68.2% for a HMM trained with the motions of only one subject) compared to other features (25.3% for Cartesian features and 17.8% for angular features in the same conditions). The method is illustrated through an interactive demo in which three virtual humans have to interactively recognize and replay the performance of the user. Each virtual human is associated with a HMM recognizer based on the three different input features.

Abstract While walking through a crowd, a pedestrian experiences a large number of interactions with his neighbors. The nature of these interactions is varied, and it has been observed that macroscopic phenomena emerge from the combination of these local interactions. Crowd models have hitherto considered collision avoidance as the unique type of interactions between individuals, few have considered walking in groups. By contrast, our paper focuses on interactions due to the following behaviors of pedestrians. Following is frequently observed when people walk in corridors or when they queue. Typical macroscopic stop‐and‐go waves emerge under such traffic conditions. Our contributions are, first, an experimental study on following behaviors, second, a numerical model for simulating such interactions, and third, its calibration, evaluation and applications. Through an experimental approach, we elaborate and calibrate a model from microscopic analysis of real kinematics data collected during experiments. We carefully evaluate our model both at the microscopic and the macroscopic levels. We also demonstrate our approach on applications where following interactions are prominent.

Annotating unlabeled motion captures plays an important role in Computer Animation for motion analysis and motion edition purposes. Locomotion is a difficult case study as all the limbs of the human body are involved whereas a low-dimensional global motion is performed. The oscillatory nature of the locomotion makes difficult the distinction between straight steps and turning ones, especially for subtle orientation changes. In this paper we propose to geometrically model the center of mass trajectory during locomotion as a C-continuous circular arcs sequence. Our model accurately analyzes the global motion into the velocity-curvature space. An experimental study demonstrates that an invariant law links curvature and velocity during straight walk. We finally illustrate how the resulting law can be used for annotation purposes: any unlabeled motion captured walk can be transformed into an annotated sequence of straight and turning steps. Several examples demonstrate the robustness of our approach and give comparison with classical threshold-based techniques. Copyright © 2011 John Wiley & Sons, Ltd.

The performance of an interactive virtual crowd system for entertainment purposes can be greatly improved by setting a level-of-details (LOD) strategy: in distant areas, collision avoidance can even be stealthy disabled to drastically speed-up simulation and to handle huge crowds. The greatest difficulty is then to select LODs to progressively simplify simulation in an imperceptible but efficient manner. The main objective of this work is to experimentally evaluate spectators' ability to detect the presence of collisions in simulations. Factors related to the conditions of observation and simulation are studied, such as the camera angles, distance to camera, level of interpenetration or crowd density. Our main contribution is to provide a LOD selection function resulting from two perceptual studies allowing crowd system designers to scale a simulation by relaxing the collision avoidance constraint in a least perceptible manner. The relaxation of this constraint is an important source for computational resources savings. Our results reveal several misconceptions in previously used LOD selection functions and suggest yet unexplored variables to be considered. We demonstrate our function efficiency over several evaluation scenarios.

Abstract This paper presents a novel, parameteric framework for synthesizing new character motions from existing motion capture data. Our framework can conduct morphological adaptation as well as kinematic and physically‐based corrections. All these solvers are organized in layers in order to be easily combined together. Given locomotion as an example, the system automatically adapts the motion data to the size of the synthetic figure and to its environment; the character will correctly step over complex ground shapes and counteract with external forces applied to the body. Our framework is based on a frame‐based solver. This ensures animating hundreds of humanoids with different morphologies in real‐time. It is particularly suitable for interactive applications such as video games and virtual reality where a user interacts in an unpredictable way. Copyright © 2009 John Wiley & Sons, Ltd.

This paper aims at proposing a framework for animating virtual humans that can efficiently interact with real users in virtual reality (VR). If the user's order can be modeled as targets and commands, the system searches a database for the most convenient behavior. In order to avoid using a huge database that can deal with any kind of situation, we propose to associate this searching process to an adaptation module. Hence, even if the selected motion is not perfectly suited with the situation, it can be adapted in order to reach accurately the target specified by the user. This framework is illustrated with a kung-fu fighter example. Two people are involved in this example: the user and the supervisor. The user is displacing in the real environment while the position of his head is tracked in real-time thanks to reflective markers. The virtual opponent follows the displacement of the user to stay close to him. At any time, the supervisor can ask the virtual character to kick or punch the user. Our system automatically searches for the convenient motion in an average-size database (75 motions compared to hundreds of motions required for motion graphs) and adapts it to the current situation.

This paper aims at proposing a new method to animate aerial motions in interactive environments while taking dynamics into account. Classical approaches are based on spacetime constraints and require a complete knowledge of the motion. However, in Virtual Reality, the user's actions are unpredictable so that such techniques cannot be used. In this paper, we deal with the simulation of gymnastic aerial motions in virtual reality. A user can directly interact with the virtual gymnast thanks to a real-time motion capture system. The user's arm motions are blended to the original aerial motions in order to verify their consequences on the virtual gymnast's performance. Hence, a user can select an initial motion, an initial velocity vector, an initial angular momentum, and a virtual character. Each of these choices has a direct influence on mechanical values such as the linear and angular momentum. We thus have developed an original method to adapt the character's poses at each time step in order to make these values compatible with mechanical laws: the angular momentum is constant during the aerial phase and the linear one is determined at take-off. Our method enables to animate up to 16 characters at 30hz on a common PC. To sum-up, our method enables to solve kinematic constraints, to retarget motion and to correct it to satisfy mechanical laws. The virtual gymnast application described in this paper is very promising to help sports-men getting some ideas which postures are better during the aerial phase for better performance.

International audience

Using motion capture systems to animate humanlike figures still remains difficult when the movements are complex or need to be adapted to geometric constraints. We propose a new method to blend several captured movements while adapting the trajectories to new skeletons and to unknown environments. For each body part (considered as resources), a priority is defined for each movement (considered as consumers). The trajectories applied to the skeleton consist of a weighted sum of the motions trajectories. A new technique to compute the weights is proposed. Finally, the system adapts the resulting trajectories to the synthetic skeleton and to the environment. The results enabled to animate up to one hundred actors in interactive environments.

We present a generic animation and simulation platform which integrates the different animation models: descriptive, generative and behavioural models. The integration of these models in the same platform offers to each dynamic entity a more realistic and a richer environment, and thereby increases possible interactions between an actor and its environment. Therefore we describe the kernel of the platform, then we explain how it is used from the programmer's point of view and we illustrate its use in the field of driving simulation.