This demo paper introduces a unique mobile application for real-time monitoring of physical therapy exercises. The application learns from therapist-guided exercise sessions and provides personalized feedback during at-home sessions. Each captured session is analyzed with state-of-the-art pose estimation, segmented into individual repetitions, and at-home exercises are evaluated against a personalized template. The application also offers intuitive real-time feedback during the at-home exercise in mirrored video and a comprehensive summary of the entire session. The demo video is available at: https://youtu.be/ew1gCw9nJNk.

Pose-estimation methods enable extracting human motion from common videos in the structured form of 3D skeleton sequences. Despite great application opportunities, effective content-based access to such spatio-temporal motion data is a challenging problem. In this article, we focus on the recently introduced text-motion retrieval tasks, which aim to search for database motions that are the most relevant to a specified natural language textual description ( text-to-motion ) and vice-versa ( motion-to-text ). Despite recent efforts to explore these promising avenues, a primary challenge remains the insufficient data available to train robust text-motion models effectively. To address this issue, we propose to investigate joint-dataset learning—where we train on multiple text-motion datasets simultaneously—together with the introduction of a Cross-Consistent Contrastive Loss (CCCL) function, which regularizes the learned text-motion common space by imposing uni-modal constraints that augment the representation ability of the trained network. To learn a proper motion representation, we also introduce a transformer-based motion encoder, called MoT++, which employs spatio-temporal attention to process skeleton data sequences. We demonstrate the benefits of the proposed approaches on the widely used KIT Motion Language and HumanML3D datasets, including also some results on the recent Motion-X dataset. We perform detailed experimentation on joint-dataset learning and cross-dataset scenarios, showing the effectiveness of each introduced module in a carefully conducted ablation study and, in turn, pointing out the limitations of state-of-the-art methods. The code for reproducing our results is available here: https://github.com/mesnico/MOTpp .

Due to recent advances in pose-estimation methods, human motion can be extracted from a common video in the form of 3D skeleton sequences. Despite wonderful application opportunities, effective and efficient content-based access to large volumes of such spatio-temporal skeleton data still remains a challenging problem. In this paper, we propose a novel content-based text-to-motion retrieval task, which aims at retrieving relevant motions based on a specified natural-language textual description. To define baselines for this uncharted task, we employ the BERT and CLIP language representations to encode the text modality and successful spatio-temporal models to encode the motion modality. We additionally introduce our transformer-based approach, called Motion Transformer (MoT), which employs divided space-time attention to effectively aggregate the different skeleton joints in space and time. Inspired by the recent progress in text-to-image/video matching, we experiment with two widely-adopted metric-learning loss functions. Finally, we set up a common evaluation protocol by defining qualitative metrics for assessing the quality of the retrieved motions, targeting the two recently-introduced KIT Motion-Language and HumanML3D datasets. The code for reproducing our results is available here: https://github.com/mesnico/text-to-motion-retrieval.

Digitization of human motion using 2D or 3D skeleton representations offers exciting possibilities for many applications but, at the same time, requires scalable content-based retrieval techniques to make such data reusable. Although a lot of research effort focuses on extracting content-preserving motion features, there is a lack of techniques that support efficient similarity search on a large scale. In this paper, we introduce a new indexing scheme for organizing large collections of spatio-temporal skeleton sequences. Specifically, we apply the motion-word concept to transform skeleton sequences into structured text-like motion documents, and index such documents using an extended inverted-file approach. Over this index, we design a new similarity search algorithm that exploits the properties of the motion-word representation and provides efficient retrieval with a variable level of approximation, possibly reaching constant search costs disregarding the collection size. Experimental results confirm the usefulness of the proposed approach.

With the increasing availability of human motion data captured in the form of 2D/3D skeleton sequences, more complex motion recordings need to be processed. In this paper, we focus on the similarity-based retrieval of motion episodes - medium-sized skeleton sequences that consist of multiple semantic actions and correspond to some logical motion unit (e.g., a figure skating performance). We examine two orthogonal approaches to the episode-matching task: (1) the deep learning approach that is traditionally used for processing short motion actions, and (2) the motion-word technique that transforms skeleton sequences into a text-like representation. Since the second approach is more promising, we propose a two-phase retrieval scheme that combines mature text-processing techniques with application-specific refinement methods. We demonstrate that this solution achieves promising results in both effectiveness and efficiency, and can be further indexed to implement scalable episode retrieval.

Large volumes of RGB video data are recorded and processed every day. One of the embedded data modality within these videos is the information about human motions. Up to now, this information has been almost unfeasible to extract, and thus human-motion understanding research has been mainly limited to 3D skeleton data captured by dedicated hardware only. However, with recent advances in computer vision, it is possible to estimate 2D skeleton sequences from ordinary videos quite accurately. Such 2D skeleton data possess an excellent potential for future motion understanding applications. In this paper, we adopt a state-of-the-art bidirectional LSTM network to analyze the accuracy gap in the expressive power of 2D and 3D skeleton data recorded simultaneously on a high number of 20k human actions. We further examine how the missing depth information and fluctuations in 2D skeleton sizes influence the recognition rate. We also demonstrate the suitability of 2D skeleton data for general daily activity recognition by reporting baselines on the PKU-MMD dataset.

Action recognition is a fundamental operation in 3D human motion analysis. Existing deep learning classifiers achieve a high recognition accuracy if large amounts of training data are provided. However, such data are difficult to obtain in a variety of application scenarios, mainly due to the high costs of motion capture technologies and an absence of suitable actors. In this paper, we propose augmentation techniques to artificially enlarge existing collections of 3D human skeleton sequences. The proposed techniques are especially useful for datasets distinguishing in a high number of classes, each of them characterized by only a limited number of action samples. We experimentally demonstrate that the augmented data help to significantly increase the recognition accuracy even using a standard deep learning architecture.

Motion capture data are digital representations of human movements in form of 3D trajectories of multiple body joints. To understand the captured motions, similarity-based processing and deep learning have already proved to be effective, especially in classifying pre-segmented actions. However, in real-world scenarios motion data are typically captured as long continuous sequences, without explicit knowledge of semantic partitioning. To make such unsegmented data accessible and reusable as required by many applications, there is a strong requirement to analyze, search, annotate and mine them automatically. However, there is currently an absence of datasets and benchmarks to test and compare the capabilities of the developed techniques for continuous motion data processing. In this paper, we introduce a new large-scale LSMB19 dataset consisting of two 3D skeleton sequences of a total length of 54.5 hours. We also define a benchmark on two important multimedia retrieval operations: subsequence search and annotation. Additionally, we exemplify the usability of the benchmark by establishing baseline results for these operations.

Motion capture technologies can digitize human movements into a discrete sequence of 3D skeletons. Such spatio-temporal data have a great application potential in many fields, ranging from computer animation, through security and sports to medicine, but their computerized processing is a difficult problem. The objective of this tutorial is to explain fundamental principles and technologies designed for searching, subsequence matching, classification and action detection in the 3D human motion data. These operations inherently require the concept of similarity to determine the degree of accordance between pairs of 3D skeleton sequences. Such similarity can be modeled using a generic approach of metric space by extracting effective deep features and comparing them by efficient distance functions. The metric-space approach also enables applying traditional index structures to efficiently access large datasets of skeleton sequences. We demonstrate the functionality of selected motion-processing operations by interactive web applications.

Motion capture technologies digitize human movements by tracking 3D positions of specific skeleton joints in time. Such spatio-temporal multimedia data have an enormous application potential in many fields, ranging from computer animation, through security and sports to medicine, but their computerized processing is a difficult problem. In this paper, we focus on an important task of recognition of a user-defined motion, based on a collection of labelled actions known in advance. We utilize current advances in deep feature learning and scalable similarity retrieval to build an effective and efficient k-nearest-neighbor recognition technique for 3D human motion data. The properties of the technique are demonstrated by a web application which allows a user to browse long motion sequences and specify any subsequence as the input for probabilistic recognition based on 130 predefined classes.

This paper explores the possibility to use grapheme-based word and sub-word models in the task of spoken term detection (STD). The usage of grapheme models eliminates the need for expert-prepared pronunciation lexicons (which are often far from complete) and/or trainable grapheme-to-phoneme (G2P) algorithms that are frequently rather inaccurate, especially for rare words (words coming from a different language). Moreover, the G2P conversion of the search terms that need to be performed on-line can substantially increase the response time of the STD system. Our results show that using various grapheme-based models, we can achieve STD performance (measured in terms of ATWV) comparable with phoneme-based models but without the additional burden of G2P conversion.

Motion capture technologies digitize human movements by tracking 3D positions of specific skeleton joints in time. Such spatio-temporal data have an enormous application potential in many fields, ranging from computer animation, through security and sports to medicine, but their computerized processing is a difficult problem. The recorded data can be imprecise, voluminous, and the same movement action can be performed by various subjects in a number of alternatives that can vary in speed, timing or a position in space. This requires employing completely different data-processing paradigms compared to the traditional domains such as attributes, text or images. The objective of this tutorial is to explain fundamental principles and technologies designed for similarity comparison, searching, subsequence matching, classification and action detection in the motion capture data. Specifically, we emphasize the importance of similarity needed to express the degree of accordance between pairs of motion sequences and also discuss the machine-learning approaches able to automatically acquire content-descriptive movement features. We explain how the concept of similarity together with the learned features can be employed for searching similar occurrences of interested actions within a long motion sequence. Assuming a user-provided categorization of example motions, we discuss techniques able to recognize types of specific movement actions and detect such kinds of actions within continuous motion sequences. Selected operations will be demonstrated by on-line web applications.

Current motion-capture technologies produce continuous streams of 3D human joint trajectories. One of the challenges is to automatically annotate such streams of complex spatio-temporal data in real time. In this paper, we propose an efficient approach to label motion stream data in real time with a limited usage of main memory. Based on a set of user-defined motion profiles, each of them specified by multiple representative samples, the currently visible part of an input motion stream is processed by identifying a moderate number of segments of various lengths. These segments are compared to the profiles to measure their similarity. The segments having a high similarity to a given motion profile are annotated with the corresponding label. The proposed approach performs fast, allows profiles to be dynamically changed at runtime, and does not require any learning procedure, in comparison with existing solutions evaluated on real-life data.

Computer-aided analyses of motion capture data require an effective and efficient concept of motion similarity. Traditional methods generally compare motion sequences by applying time-warping techniques to high-dimensional trajectories of joints. An increasing effectiveness of machine-learning techniques, such as deep convolutional neural networks, brings new possibilities for similarity comparison. Inspired by recent advances in neural networks and image processing, we propose new variants of transformation of motion sequences into 2D images. The generated images are used to fine-tune a neural network from which 4,096D features are extracted and compared by a modified Euclidean distance. The proposed concept is not only efficient but also very effective and beats existing methods on a challenging dataset with 130 categories.

The subsequence-matching operation applied to motion capture data searches in long motion sequences to locate their parts that are similar to a query example. An effective and efficient implementation of such operation is valuable to increase reusability and findability of expensively recorded data in the past. This demonstration paper builds on recent advances in the field of motion-data processing and implements them into a single web application that allows users to discover query-similar subsequences. The proposed application does not require any textual annotations nor explicit knowledge of the data and can deal with spatio-temporal variances of human movements. Efficiency and effectiveness can be verified by searching a 12-hour database of motion sequences in real time.

Abstract The development of motion capturing devices poses new challenges in the exploitation of human‐motion data for various application fields, such as computer animation, visual surveillance, sports, or physical medicine. Recently, a number of approaches dealing with motion data have been proposed, suggesting characteristic motion features to be extracted and compared on the basis of similarity. Unfortunately, almost each approach defines its own set of motion features and comparison methods; thus, it is hard to fairly decide which similarity model is the most suitable for a given kind of human‐motion retrieval application. To cope with this problem, we propose the human motion model evaluator, which is a generic framework for assessing candidate similarity models with respect to the purpose of the target application. The application purpose is specified by a user in form of a representative sample of categorized motion data. Respecting such categorization, the similarity models are assessed from the effectiveness and efficiency points of view using a set of space‐complexity, information‐retrieval, and performance measures. The usability of the framework is demonstrated by case studies of three practical examples of retrieval applications focusing on recognition of actions, detection of similar events, and identification of subjects. Copyright © 2015 John Wiley & Sons, Ltd.

Exploiting the concepts of social networking represents a novel approach to the approximate similarity query processing. We present a metric social network where relations between peers, giving similar results, are established on per-query basis. Based on the universal law of generalization, a new query forwarding algorithm is proposed. The same principle is used to manage query histories of individual peers with the possibility to tune the tradeoff between the extent of the history and the level of the query-answer approximation. All algorithms are tested on real data and real network of computers.

We propose a self-organized and self-adapting system for content-based search in multimedia data. In particular, we build a semantic overlay over an existing peer-to-peer network. The self-organization of the overlay is obtained by using the social-network paradigm. The connections between peers are formed on the basis of a query-answer principle. The knowledge about answers to previous queries is exploited to route queries efficiently. At the same time, a randomized mechanism is used to explore new and unvisited parts of the network. In this way, the self-adaptable and robust system is built. Moreover, the metric space data model is used to achieve extensibility. The proposed concepts are verified on a network consisting of 2,000 peers and indexing 10 million images.