In the current era of AI technology, where systems increasingly rely on big data to process vast amounts of societal information, efficient methods for integrating and utilizing diverse datasets are essential. This article presents a novel approach for transforming the feature space of different datasets through singular value decomposition (SVD) to extract common and hidden features as using the prior domain knowledge. Specifically, we apply this method to two datasets: 1) one related to physical and cognitive frailty in the elderly; and 2) another focusing on identifyingIKIGAI(happiness, self-efficacy, and sense of contribution) in volunteer staff of a civic health promotion activity. Both datasets consist of multiple sub-datasets measured using different modalities, such as facial expressions, sound, activity, and heart rates. By defining feature extraction methods for each subdataset, we compare and integrate the overlapping data. The results demonstrated that our method could effectively preserve common characteristics across different data types, offering a more interpretable solution than traditional dimensionality reduction methods based on linear and nonlinear transformation. This approach has significant implications for data integration in multidisciplinary fields and opens the door for future applications to a wide range of datasets.

In this paper, we propose a novel graph convolutional network architecture, Graph Stacked Hourglass Networks, for 2D-to-3D human pose estimation tasks. The proposed architecture consists of repeated encoder-decoder, in which graph-structured features are processed across three different scales of human skeletal representations. This multi-scale architecture enables the model to learn both local and global feature representations, which are critical for 3D human pose estimation. We also introduce a multi-level feature learning approach using different-depth intermediate features and show the performance improvements that result from exploiting multi-scale, multi-level feature representations. Extensive experiments are conducted to validate our approach, and the results show that our model outperforms the state-of-the-art.

This article describes a stochastic framework for integrating human whole-body motions with natural language. Human whole-body motions in daily life are measured by inertial measurement units (IMU) and subsequently encoded into motion primitives. Sentences are manually attached to the human motion primitives for their descriptions. Two aspects of semantics and syntactics are represented by stochastic modules. One stochastic module trains the linking of motion primitives to words, and the other module represents word order in the sentence structure. These two modules are helpful toward converting human whole-body motions into descriptions, where multiple words are generated from the human motions by the first module, and the second module searches for syntactically consistent sentences consisting of the generated words. The proposed framework is tested on a large dataset of human whole-body motions and their descriptive sentences. The linking of human motions to natural language enables robots to understand observations of human behavior as sentences.

Humans perceive continuous high-dimensional information by dividing it into significant segments such as words and units of motion. We believe that such unsupervised segmentation is also important for robots to learn topics such as language and motion. To this end, we previously proposed a hierarchical Dirichlet process-Gaussian process-hidden semi-Markov model (HDP-GP-HSMM). However, an important drawback to this model is that it cannot divide high-dimensional time-series data. Further, low-dimensional features must be extracted in advance. Segmentation largely depends on the design of features, and it is difficult to design effective features, especially in the case of high-dimensional data. To overcome this problem, this paper proposes a hierarchical Dirichlet process-variational autoencoder-Gaussian process-hidden semi-Markov model (HVGH). The parameters of the proposed HVGH are estimated through a mutual learning loop of the variational autoencoder and our previously proposed HDP-GP-HSMM. Hence, HVGH can extract features from high-dimensional time-series data, while simultaneously dividing it into segments in an unsupervised manner. In an experiment, we used various motion-capture data to show that our proposed model estimates the correct number of classes and more accurate segments than baseline methods. Moreover, we show that the proposed method can learn latent space suitable for segmentation.

In this paper, we investigate the relationship between the depth perception and an approaching motion toward an object. We propose the depth perception dependency test, which is the combination of tests of a motion and depth perception based on the situation that an object is placed on the human's pathway. Firstly, as the motion test, we set the ball approach motion and asked elderly participants to approach and contact a ball by their foot, because this motion is easy to measure and requires localization skill and motion planning skill. The swing/support legs positions at the toeoff time of the swing leg just before contacting the ball was analyzed. Secondly, as the depth perception test, the pseudo3D image test was proposed. The coordinate transformation model for the calculation of the depth perception ability was also proposed. Through the proposed test and the proposed model, it was clarified that the participants who are regarded as perceiving the strong visual illusion perceive the objects closer than its real position, and their swing leg toe off positions in the ball approaching motion were significantly farther than other participants. Thus, it can be concluded that there was a relationship between the depth perception and the approaching motion that is thought to be a higher risk of stumbles.

This paper discusses video motion capture, namely, 3D reconstruction of human motion from multi-camera images. After the Part Confidence Maps are computed from each camera image, the proposed spatiotemporal filter is applied to deliver the human motion data with accuracy and smoothness for human motion analysis. The spatiotemporal filter uses the human skeleton and mixes temporal smoothing in two-time inverse kinematics computations. The experimental results show that the mean per joint position error was 26.1mm for regular motions and 38.8mm for inverted motions.

In this paper, we propose a motion model that focuses on the discriminative parts of the human body related to target motions to classify human motions into specific categories, and apply this model to multi-class daily motion classifications. We extend this model to a motion recognition system which generates multiple sentences associated with human motions. The motion model is evaluated with the following four datasets acquired by a Kinect sensor or multiple infrared cameras in a motion capture studio: UCF-kinect; UT-kinect; HDM05-mocap; and YNL-mocap. We also evaluate the sentences generated from the dataset of motion and language pairs. The experimental results indicate that the motion model improves classification accuracy and our approach is better than other state-of-the-art methods for specific datasets, including human–object interactions with variations in the duration of motions, such as daily human motions. We achieve a classification rate of 81.1% for multi-class daily motion classifications in a non cross-subject setting. Additionally, the sentences generated by the motion recognition system are semantically and syntactically appropriate for the description of the target motion, which may lead to human–robot interaction using natural language.

Gesture recognition is used for many practical applications such as human-robot interaction, medical rehabilitation and sign language. In this paper, we apply a hybrid generative-discriminative approach by using the Fisher Vector to improve the recognition performance. The strategy is to merge the generative approach of Hidden Markov Model dealing with spatio-temporal motion data with the discriminative approach of Support Vector Machine focusing on the classification task. The motion segments are encoded into HMMs, and each segment is converted to FV, whose elements can be obtained as the derivative of the probability of the segment being generated by the HMMs with respect to their parameters. SVM is subsequently trained by the FVs. An input gesture can be classified to corresponding gesture category by SVM. In the experiments, we test our approach by comparing three HMM chain models and four categorization methods on dataset provided by the ChaLearn Looking at People Challenge 2014 (LAP 2014). The results show that similar gesture patterns are clustered closely in several categories. Our approach based left-to-right HMMs outperforms other gesture recognition methods. More specifically, the hybrid generative-discriminative approach overcomes the standard HMM approach and the generative kernel approach overcomes the generative embedding approach. For these results, our approach is effective to improve the recognition performance.

This paper presents a novel robot pose measure for human movement imitation based entirely on the Euclidean distance information between any two links of a robot and any link and object in the robot's environment in a Cartesian task space. A Hidden Markov Model is used to encode the spatio-temporal information of multiple demonstrations. In combination with Gaussian Mixture Regression for extracting the important task properties, feasible full-body motion adaption can be achieved. The method is suited for use with a humanoid robot by considering additional constraints like balance control and collision avoidance. In order to tackle modeling errors occurring due to the human movement demonstration and the robotic reproduction, a manipulability based weighting scheme is proposed. Complexity reduction of the otherwise redundant pose measure is performed based upon a mechanical analogy of an interconnected spring system. Experiments are conducted using a HRP-4 robot and display the applicability of the presented methods for robotic full-body motion imitation tasks.

Motion recognition is an essential technology for social robots in various environments such as homes, offices and shopping center, where the robots are expected to understand human behavior and interact with them. In this paper, we present a system composed of three models: motion language model, natural language model and integration inference model, and achieved to generate sentences from motions using large high-order N-grams. We confirmed not only that using higher-order N-grams improves precision in generating long sentences but also that the computational complexity of the proposed system is almost the same as our previous one. In addition, we improved the precision by aligning the graph structure representing generated sentences into confusion network form. This means that simplifying and compacting word sequences affect the precision of sentence generation.

The language is a symbolic system unique to human being. The acquisition of language, which has its meanings in the real world, is important for robots to understand the environment and communicate with us in our daily life. This paper proposes a novel approach to establish a fundamental framework for the robots which can understand language through their whole body motions. The proposed framework is composed of three modules: “motion symbol”, “motion language model”, and “natural language model”. In the motion symbol module, motion data are symbolized by Hidden Markov Models (HMMs). Each HMM represents abstract motion patterns. Then the HMMs are defined as motion symbols. The motion language model is stochastically designed for links between motion symbols and words. This model consists of three layers of motion symbols, latent states and words. The connections between the motion symbol and the latent state, and between the latent state and the words are denoted by two kinds of probabilities respectively. One connection is represented by the probability that the motion symbol generates the latent state, and the other connection is represented by the probability that the latent state generates the word. Therefore, the motion language model can connect the motion symbols to the words through the latent state. The natural language model stochastically represents sequences of words. In this paper, a bigram, which is a special case of N-gram model, is adopted as the natural language model. This model has the words as nodes and transitions between two words as edges. Therefore sentence structure is expressed as transitions among words. The integration of the motion language model and natural language model can be implemented by the search computation for sentences corresponding to motions and for motions corresponding to sentences. Especially, the usage of the bigram as the natural language model provides a simple search computation so that appropriate and fast bidirectional computation between the motions and language can be achieved. Our approach makes it possible for humanoid robots not only to interpret motions as sentences but also to generate motions from sentences. The tests by using various motions and words validate our framework for the language acquisition of humanoid robots.

Natural language interaction between humans and robots is a very challenging topic, especially when it refers to motion descriptions in a certain environment. This problem is particularly relevant during physical human-robot interaction, e.g. in cooperative transportation tasks, where the partners' physical coupling requires an agreement on the way to follow. Understanding in depth the link between sentences, words, environmental properties and motions can deeply enhance the interaction between humans and robots. In this work, we propose a novel approach for learning relations and dependencies between motion, natural language and environmental properties using parameterized left-to-right time-based Hidden Markov Models. A natural language model represents the link between language and motion symbols while the HMMs parameterization corresponds to the explicit influence on motions of both words and environmental features. The proposed PHMM approach parameterizes the output and the transition probabilities using a non-linear dependency estimation. The method is validated by learning and generating navigation primitives in a 2 Degrees-Of-Freedom (DoF) virtual scenario.

This paper proposes an approach to construct a system which allows humanoid robots to recognize human behaviors and predict his or her future behaviors. The system consists of two modules : "motion symbol tree" and "motion symbol graph", Human demonstrator motion patterns are stored as motion symbols, which abstract the motion data by using Hidden Markov Models. The stored motion patterns are organized into a hierarchical tree structure, which represents the similarity among the motion patterns and provides abstracted motion patterns. The formed hierarchical structure is the motion symbol tree. Concatenated sequences of motion patterns are stochastically represented as transitions between the motion patterns by using an Ngram Model, and the causality among the human behaviors are extracted. This structure is the motion symbol graph. The behavioral hierarchy and transition model make it possible to predict human behaviors during observation and to generate sequences of motion patterns automatically while maintaining a natural motion stream, as if the system is a "crystal ball" to reflect future behaviors. The experiments demonstrate the validity of the proposed framework on a large scale motion data.

Captured human motion data are used so that humanoid robots or computer graphics (CG) characters can behave naturally. Because the motion capture system is expensive, and time-consuming process is needed to acquire motion data, technology that enables to reuse existing motion data efficiently is required. This paper proposes motion retrieval method with natural language word based on stochastic correlation between motion and language. We construct a space which has maximum correlation with motion pattern features and word features, and we use this space as search space for motion retrieval. Proto-symbol space, which represents the relationship of each symbolized motion patterns, is used as motion feature space. And as word feature, binary features are used which represent whether a word label is attached or not. Because the constructed search space has correlation between motion patterns and words, associative motion retrieval considering similarity of motion pattern or closeness of word meaning becomes possible. We validate proposed motion retrieval method by constructing motion database with captured human motion data.

This paper describes an novel approach towards linguistic processing for robots through integration of a motion language model and a natural language model. The motion language model works for association of words from motion symbols. The natural language model is one used for a morphological analysis, which has been developed in natural language community. The natural language model is optimized using a enormous amount of words. So this model is scalable architecture. The motion language model and the natural language model can be integrated since both models are represented graphically. The integration of the motion language model and the natural language model allows robots not only to interpret motion patterns as sentences but also to generate motions from sentences. This paper demonstrates the validity of our proposed framework even in the case that large-scale word corpus is needed processing through experiments of interpreting motion patterns as sentences and generating motion patterns from sentences.

This paper proposes an approach to hierarchy formation of human behaviors, extraction of the behavioral transitions, and their application to prediction and automatic generation of behaviors. Human demonstrator motion patterns are stored as motion symbols, which abstract the motion data by using Hidden Markov Models. The stored motion patterns are organized into a hierarchical tree structure, which represents the similarity among the motion patterns and provides abstracted motion patterns. Concatenated sequences of motion patterns are stochastically represented as transitions between the abstracted motion patterns by using an Ngram Model, and the transitional relationships of the human behaviors are extracted. The behavioral hierarchy and transition model make it possible to predict human behaviors during observation and to generate sequences of motion patterns automatically while maintaining a natural motion stream, as if the system is a “crystal ball” to reflect future behaviors. The experiments validates the proposed framework by using a developed visualization system, which shows the demonstrator or the operator the established hierarchical tree and the transition network of the motion patterns, predicted behaviors and generated sequences of the motion patterns.

This paper describes the linguistic model based on symbolization of motion patterns for humanoid robots. The model consists of two kinds of stochastic models : the motion language model and the natural language model. The motion language model stochastically connects the symbols of motion patterns to the morpheme words through the latent states which represent the underlying linguistic structure such as semantic contents. The natural language model represents the dynamics of the word classes. The motion language model and the natural language model correspond to semantics and syntax respectively. The integration of the motion language model and the natural language model allows robots not only to linguistically interpret the motion patterns as sentences but also to generate the motions from the sentences. The two kinds of linguistic processes of the interpretation and the generation can be obtained by solving search problems: search for a sequence of morpheme words and a symbol of motion pattern. The proposed approach to interpretation of motion patterns as sentences and generation of motion patterns from the sentences through integration of the motion language model and the natural language model is validated by the experiment on the human behavioral data.

This paper describes an novel approach towards linguistic processing for robots through integration of a motion language module and a natural language module. The motion language module represents association between symbolized motion patterns and words. The natural language module models sentences. The motion language module and the natural language module are graphically integrated. The integration allows robots not only to interpret observed motion as a sentence but also to generate motion with a sentence. This paper proposes incremental learning algorithm of association between symbolized motion patterns and words. The incremental learning is required for robot to autonomously develop the linguistic skill. The algorithm can be derived from optimization of the motion language module under stochastic constraints such that the associative probability of a new training pair composed of symbolized motion pattern and sentence becomes larger. Test of interpreting observed motion as sentences demonstrates the validity of the proposed incremental learning algorithm.

This paper describes a novel approach for incremental learning of human motion pattern primitives through online observation of human motion. The observed time series data stream is first stochastically segmented into potential motion primitive segments, based on the assumption that data belonging to the same motion primitive will have the same underlying distribution. The motion segments are then abstracted into a stochastic model representation and automatically clustered and organized. As new motion patterns are observed, they are incrementally grouped together into a tree structure, based on their relative distance in the model space. The tree leaves, which represent the most specialized learned motion primitives, are then passed back to the segmentation algorithm so that as the number of known motion primitives increases, the accuracy of the segmentation can also be improved. The combined algorithm is tested on a sequence of continuous human motion data that are obtained through motion capture, and demonstrates the performance of the proposed approach.

This paper describes a novel approach for incremental learning of human motion pattern primitives through on-line observation of human motion. The observed motion time series data stream is first stochastically segmented into potential motion primitive segments, based on the assumption that data belonging to the same motion primitive will have the same underlying distribution. The motion segments are then abstracted into a stochastic model representation, and automatically clustered and organized. As new motion patterns are observed, they are incrementally grouped together based on their relative distance in the model space. The resulting representation of the knowledge domain is a tree structure, with specialized motions at the tree leaves, and generalized motions closer to the root. The tree leaves, which represent the most specialized learned motion primitives, are then passed back to the segmentation algorithm, so that as the number of known motion primitives increases, the accuracy of the segmentation can also be improved. The combined algorithm is tested on a sequence of continuous human motion data obtained through motion capture, and demonstrates the performance of the proposed approach.

This paper describes an imitative learning of driving time series data for intellectual cognition toward future automobiles. The driving pattern primitives consisting of states of the environment, vehicle and driver are symbolized by hidden Markov models (HMMs), which can be used for both recognition and generation of the driving patterns. The relationship among the HMMs can be represented by locating the HMMs in a multidimensional space. The contribution of each variable to the HMM space can be analyzed such that important variables can be selected out of the driving data in order to reduce the size of the HMMs. Moreover, this paper presents a hierarchical model with the HMMs abstracting the primitive driving patterns in the lower layer, and another HMMs abstracting the longterm contextual driving patterns which are representation in the HMM space. Tests with a driving simulator and a actual vehicle demonstrate not only the validity of symbolization of driving pattern primitives, recognition and generation, but also availability of key feature selection. The extended hierarchical model is also proved to have a potential to predict the driving data appropriately.

Motion capture systems are used to obtain motion data such that humanoid robots or computer graphics (CG) characters can behave naturally. However, it has proven to be hard not only to modify the capture data without losing its reality but also to search for the required capture data in a lot of capture data. In this paper, we provide a solution to these problems based on our previous work on symbolization of motion patterns for developing humanoid intelligence. Similar motion sequences in the database are abstracted as a symbol, which will be applied to searching motion patterns in the database similar to a given motion. This paper also introduces a method for building a stochastic symbol-word mapping model utilizing the word labels provided by the operator during motion capture sessions. This model converts a input sequence of words into a sequence of symbols, and then allows the capture database both to be searched for capture data corresponding to the input (a sequence of words) and to provide the users with new motion data generated by the symbols. Finally, we apply analogy of symbols to establishing the database in order to provide an appropriate motion data in response to an unsupervised sequence of words and then demonstrate the validity of analogy theory.

This paper describes an improved methodology for human motion recognition and imitation based on Factorial Hidden Markov Models (FHMM). Unlike conventional Hidden Markov Models (HMMs), FHMMs use a distributed state representation, which allows for more efficient representation of each time sequence. Once the FHMMs are trained with exemplar motion data, they can be used to generate sample trajectories for motion production, and produce significantly more accurate trajectories compared to single Hidden Markov chain models. Due to the additional information encoded in FHMMs models, FHMM models have a higher Kullback- Leibler distance compared to single Markov chain models, making it easier to distinguish between similar models. The efficacy of using FHMMs is tested on a database of human motions obtained through motion capture. The results show that FHMMs provide better generalization to new data when compared to conventional HMMs during motion recognition, as well as providing a better fit for generated data.

hierarchical model incorporating motion patterns, proto symbols and words is proposed. The proto symbols abstract motion patterns, while the words are associated with the proto symbols stochastically. This paper describes the construction of a word space, where words are located in a multidimensional space based on dissimilarities among the words. The dissimilarity between two words can be calculated by using association probabilities that the words generate motion proto symbols. The word space encapsulates relations among the words such as similar or dissimilar pairs of words. The word space also allows motion recognition based on words. The validity of the constructed word space is demonstrated on a motion capture database. Moreover, the addition of the word associations is found to change the conventional proto symbol space so that the discrimination among the proto symbols is improved.

This paper describes a novel algorithm for autonomous and incremental learning of motion pattern primitives by observation of human motion. Human motion patterns are abstracted into a Hidden Markov Model representation, which can be used for both subsequent motion recognition and generation, analogous to the mirror neuron hypothesis in primates. As new motion patterns are observed, they are incrementally grouped together using hierarchical agglomerative clustering based on their relative distance in the HMM space. The clustering algorithm forms a tree structure, with specialized motions at the tree leaves, and generalized motions closer to the root. The generated tree structure will depend on the type of training data provided, so that the most specialized motions will be those for which the most training has been received. Tests with motion capture data for a variety of motion primitives demonstrate the efficacy of the algorithm.

Communication skill is essential for social robots in various environments such as homes, offices, and hospitals, where the robots are expected to interact with humans. In this paper, we model the primitive nonverbal communication between two persons by mimetic communication model. The model consists of three groups of hidden Markov models (HMMs) hierarchically combined to recognize motions of the human and to generate the interactive motions of the robot. HMMs in the lower layer abstract the motion patterns and HMMs in the upper layer represent the interaction patterns. We demonstrate the validity of this model through kick boxing match between a motion-captured human and humanoid robot, where the robot can autonomously generate its motion in response to attacks by the human