ABSTRACT Large‐scale human motion datasets are essential for training robust motion models for analysis, synthesis, and understanding. While marker‐based motion capture provides precise data, it is costly and limited in scale and diversity. Recent advances in monocular motion capture and video‐language understanding open the way to extract plausible motion from unconstrained online videos. We present a scalable pipeline for constructing in‐the‐wild human motion datasets. From a few keywords, the system retrieves videos, extracts 3D body and facial motion, and generates high‐level textual descriptions. The pipeline is flexible, enabling targeted collection of various motions, multi‐person interactions, or expressive behaviors. We demonstrate its quality by training motion reconstruction and motion generation models, showing performance approaching that of models trained on traditional motion capture datasets, along with strong cross‐dataset generalization. Code and motion data are available at https://github.com/NatSalaz/xmopipe .

For interactivity and cost-efficiency purposes, both biological and artificial agents (e.g., robots) usually rely on sets of complementary sensors. Each sensor samples information from only a subset of the environment, with both the subset and the precision of signals varying through time depending on the agent-environment configuration. Agents must therefore perform multimodal fusion to select and filter relevant information by contrasting the shortcomings and redundancies of different modalities. For that purpose, we propose to combine a classical off-the-shelf manifold learning algorithm with dynamic neural fields (DNF), a training-free bio-inspired model of competition amid topologically-encoded information. Through the adaptation of DNF to irregular multimodal topologies, this coupling exhibits interesting properties, promising reliable localizations enhanced by the selection and attentional capabilities of DNF. In particular, the application of our method to audiovisual datasets (with direct ties to either psychophysics or robotics) shows merged perceptions relying on the spatially-dependent precision of each modality, and robustness to irrelevant features.

Multimodal merging encompasses the ability to localize stimuli based on imprecise information sampled through individual senses such as sight and hearing. Merging decisions are standardly described using Bayesian models that fit behaviors over many trials, encapsulated in a probability distribution. We introduce a novel computational model based on dynamic neural fields able to simulate decision dynamics and generate localization decisions, trial by trial, adapting to varying degrees of discrepancy between audio and visual stimulations. Neural fields are commonly used to model neural processes at a mesoscopic scale-for instance, neurophysiological activity in the superior colliculus. Our model is fit to human psychophysical data of the ventriloquist effect, additionally testing the influence of retinotopic projection onto the superior colliculus and providing a quantitative performance comparison to the Bayesian reference model. While models perform equally on average, a qualitative analysis of free parameters in our model allows insights into the dynamics of the decision and the individual variations in perception caused by noise. We finally show that the increase in the number of free parameters does not result in overfitting and that the parameter space may be either reduced to fit specific criteria or exploited to perform well on more demanding tasks in the future. Indeed, beyond decision or localization tasks, our model opens the door to the simulation of behavioral dynamics, as well as saccade generation driven by multimodal stimulation.

Recent works in the domain of deep learning for object recognition on common image classification benchmarks often address the representation learning problem under the assumption of i.i.d. input data. Although achieving satisfying results, this assumption seems not realistic when agents have to learn autonomously. An autonomous agent receives a continual visual flow of objects which is far from an i.i.d. distribution of objects. Moreover, agents have to construct their representations of the world and adapt to unknown environments, without relying on external sources of information such as labels that would be provided post-classification and are unavoidable when an over-segmentation is done. Then, in order to exploit the learned representation effectively for object recognition, a clear and meaningful relationship w.r.t. real object categories is required, which has been largely neglected in existing unsupervised algorithms.In this paper, we propose a novelty detection method for continual and unsupervised object recognition, as an extension for the recent CURL model, which allows to moderate over-segmentation while preserving accuracy, in order to meet the requirements for autonomy. We experimentally validated our approach on two standard image classification benchmarks, MNIST and Fashion-MNIST, in this unsupervised and continual learning setting and improve the state of the art in terms of cluster purity, which is crucial for subsequent object recognition, since it facilitates clustering when information on ground truth labels is not available for free.

Deep neural networks are difficult to train when applied to tasks that can be expressed as algorithmic procedures. In this article, we propose to study how the explicit guidance of a network through all steps of the algorithm, using external memory and active choice of inputs, can improve its learning capability. The idea is to take inspiration from a child's learning and running through a procedure via interaction with an external support such as a paper. We show that this mechanism applied to a simple multilayer perceptron can significantly improve its performance when learning either a multi-digit addition or multiplication, which are simple but yet challenging operations to learn.

To be accepted in our everyday life and to be valuable interaction partners, robots should be able to display emotional and empathic behaviors. That is why there has been a great focus on developing empathy in robots in recent years. However, there is no consensus on how to measure how much a robot is considered to be empathic. In this context, we decided to construct a questionnaire which specifically measures the perception of a robot's empathy in human-robot interaction (HRI). Therefore we conducted pretests to generate items. These were validated by experts and will be further validated in an experimental setting.

PROPRE is a generic and cortically inspired framework that provides online input/output relationship learning. The input data flow is projected on a self-organizing map that provides an internal representation of the current stimulus. From this representation, the system predicts the value of the output target. A predictability measure, based on the monitoring of the prediction quality, modulates the projection learning so that to favor learning of representations that are helpful to predict the output. In this article, we study PROPRE when the input/output relationship is only defined in a small subspace of the input space, that we define as a partially learnable environment. This problem, which is not typical of the machine learning field, is however crucial for the robotic developmental field. Indeed, robots face high dimensional sensory-motor environments where large areas of these sensory-motor spaces are not learnable since a motor action cannot have a consequence on every perception each time. We show that the use of the predictability measure in PROPRE leads to an autonomous gathering of local representations where the input data are related to the output value, thus providing good classification performance as the system will learn the input/output function only where it is defined.

PROPRE is a generic and modular unsupervised neural learning paradigm that extracts meaningful concepts of multimodal data flows based on predictability across modalities. It consists on the combination of three modules. First, a topological projection of each data flow on a self-organizing map. Second, a decentralized prediction of each projection activity from each others map activities. Third, a predictability measure that compares predicted and real activities. This measure is used to modulate the projection learning so that to favor the mapping of predictable stimuli across modalities. In this article, we use Kohonen map for the projection module, linear regression for the prediction one and we propose multiple generic predictability measures. We illustrate the properties and performances of PROPRE paradigm on a challenging supervised classification task of visual pedestrian data. The modulation of the projection learning by the predictability measure improves significantly classification performances of the system independently of the measure used. Moreover, PROPRE provides a combination of interesting functional properties, such as a dynamical adaptation to input statistic variations, that is rarely available in other machine learning algorithms.

SOMMA (Self Organizing Maps for Multimodal Association) consists on cortically inspired paradigms for multimodal data processing. SOMMA defines generic cortical maps - one for each modality - composed of 3-layers cortical columns. Each column learns a discrimination to a stimulus of the input flow with the BCMu learning rule [26]. These discriminations are self-organized in each map thanks to the coupling with neural fields used as a neighborhood function [25]. Learning and computation in each map is influenced by other modalities thanks to bidirectional topographic connections between all maps. This multimodal influence drives a joint self-organization of maps and multimodal perceptions of stimuli. This work takes place after the design of a self-organizing map [25] and of a modulation mechanism for influencing its self-organization [26] oriented towards a multimodal purpose. In this paper, we introduce a way to connect these self-organizing maps to obtain a multimap multimodal processing, completing our previous work. We also give an overview of the architectural and functional properties of the resulting paradigm SOMMA.