Autism Spectrum Disorder (ASD) is a diverse collection of neurobiological conditions marked by challenges in social communication and reciprocal interactions, as well as repetitive and stereotypical behaviors. Atypical behavior patterns in a long, untrimmed video can serve as biomarkers for children with ASD. In this paper, we propose a video-based weakly-supervised method that takes spatio-temporal features of long videos to learn typical and atypical behaviors for autism detection. On top of that, we propose a shallow TCN-MLP network, which is designed to further categorize the severity score. We evaluate our method on actual evaluation videos of children with autism collected and annotated (for severity score) by clinical professionals. Experimental results demonstrate the effectiveness of behaviors biomarkers that could help clinicians in autism spectrum analysis.

The dermatological concept of the Ugly-Duckling Sign (UDS) emphasizes the importance of comparing skin lesions within the same patient for enhanced diagnostic accuracy in melanoma detection, stating that atypical lesions are more likely to be cancers. However this concept is still underutilized in research, as most work on melanoma detection rely on classification ConvNets which lack the capacity to compare images together. Addressing this research gap, we introduce ReSeT (Residual Set-Transformer), a framework designed to compare skin lesions within patients during prediction. ReSeT comprises an encoder that takes individual images as input to generate embeddings, and a Set-Transformer with a residual prediction layer that compares these embeddings while predicting. We demonstrate that our architecture ReSeT significantly enhances performance compared to ConvNets and we highlighting the necessity of residual connections in the context of multi-output Transformers. We also observe that self-supervised encoders are able to generate embeddings of comparable quality to those of supervised models, showing their robustness and impact on image comparison tasks.

Bayesian Neural Networks (BNNs) have long been considered an ideal, yet unscalable solution for improving the robustness and the predictive uncertainty of deep neural networks. While they could capture more accurately the posterior distribution of the network parameters, most BNN approaches are either limited to small networks or rely on constraining assumptions, e.g., parameter independence. These drawbacks have enabled prominence of simple, but computationally heavy approaches such as Deep Ensembles, whose training and testing costs increase linearly with the number of networks. In this work we aim for efficient deep BNNs amenable to complex computer vision architectures, e.g., ResNet-50 DeepLabv3+, and tasks, e.g., semantic segmentation and image classification, with fewer assumptions on the parameters. We achieve this by leveraging variational autoencoders (VAEs) to learn the interaction and the latent distribution of the parameters at each network layer. Our approach, called Latent-Posterior BNN (LP-BNN), is compatible with the recent BatchEnsemble method, leading to highly efficient (in terms of computation and memory during both training and testing) ensembles. LP-BNNs attain competitive results across multiple metrics in several challenging benchmarks for image classification, semantic segmentation, and out-of-distribution detection.

This paper presents our contribution to facial expression recognition using images obtained from the Light Field Face Dataset(LF). We compare several variants of neural network architectures to demonstrate the potential benefits of using this relatively new optical system in the field of facial expression recognition. We propose the use of the EfficientNetV2-S convolutional neural network as the base architecture, combined with various recurrent neural networks (LSTM, GRU, BiLSTM, and BiGRU) in our experiments. Furthermore, we investigate different sets of sub-aperture images, each varying in terms of the number of images and virtual position. The results demonstrate a significant improvement in accuracy for two specific configurations, depending on the sets of sub-aperture images used. The first configuration involves using the EfficientNetV2-S model in a two-branch configuration combined with an LSTM. The second configuration uses a single branch model with a BiLSTM.

Tracking an entire high-density crowd composed of more than five hundred individuals is a difficult task that has not yet been accomplished. In this article, we propose to track pedestrians using a model composed of a Particle Filter (PF) and three Deep Convolutional Neural Networks (DCNN). The first network is a detector that learns to localize the persons. The second one is a pretrained network that estimates the optical flow, and the last one corrects the flow. Our contribution resides in the way we train this last network by PF supervision, and in Markov Random Field linking the different tracks.

In this work, we perform an in-depth analysis of the specific difficulties a crowded scene dataset raises for tracking algorithms. Starting from the standard characteristics depicting the crowd and their limitations, we introduce six entropy measures related to the motion patterns and to the appearance variability of the individuals forming the crowd, and one appearance measure based on Principal Component Analysis. The proposed measures are discussed on synthetic configurations and on multiple real datasets. These criteria are able to characterize the crowd behavior at a more detailed level and may be helpful for evaluating the tracking difficulty of different datasets. The results are in agreement with the perceived difficulty of the scenes.

Automatic facial expression classification (FER) from videos is a critical problem for the development of intelligent human-computer interaction systems. Still, it is a challenging problem that involves capturing high-dimensional spatio-temporal patterns describing the variation of one's appearance over time. Such representation undergoes great variability of the facial morphology and environmental factors as well as head pose variations. In this paper, we use Conditional Random Forests to capture low-level expression transition patterns. More specifically, heterogeneous derivative features (e.g., feature point movements or texture variations) are evaluated upon pairs of images. When testing on a video frame, pairs are created between this current frame and previous ones and predictions for each previous frame are used to draw trees from Pairwise Conditional Random Forests (PCRF) whose pairwise outputs are averaged over time to produce robust estimates. Moreover, PCRF collections can also be conditioned on head pose estimation for multi-view dynamic FER. As such, our approach appears as a natural extension of Random Forests for learning spatio-temporal patterns, potentially from multiple viewpoints. Experiments on popular datasets show that our method leads to significant improvements over standard Random Forests as well as state-of-the-art approaches on several scenarios, including a novel multi-view video corpus generated from a publicly available database.

Being able to produce facial expressions (FEs) that are adequate given a social context is key to harmonious social development, particularly in the case of children plagued with autism spectrum disorder (ASD). In this paper, we introduce JEMImE, a serious game solution that aims at teaching children how to produce FEs. JEMImE is based on a FE recognition module that is learned on a large video corpus of children performing FEs. This module is validated and incorporated through multiple scenarios of gradual difficulty, ranging from a training phase where children have to perform the FEs on request, with or without an avatar model, to an in-context phase that involves many emotion-eliciting social situations with virtual characters.

Stress is a complex phenomenon that impacts the body and the mind at several levels. It has been studied for more than a century from different perspectives, which result in different definitions and different ways to assess the presence of stress. This paper introduces a methodology for analyzing multimodal stress detection results by taking into account the variety of stress assessments. As a first step, we have collected video, depth and physiological data from 25 subjects in a stressful situation: a socially evaluated mental arithmetic test. As a second step, we have acquired three different assessments of stress: self-assessment, assessments from external observers and assessment from a physiology expert. Finally, we extract 101 behavioural and physiological features and evaluate their predictive power for the three collected assessments using a classification task. Using multimodal features, we obtain average F1 scores up to 0.85. By investigating the composition of the best selected feature subsets and the individual feature classification performances, we show that several features provide valuable information for the classification of the three assessments: features related to body movement, blood volume pulse and heart rate. From a methodological point of view, we argue that a multiple assessment approach provide more robust results.

International audience

Resorting to crowdsourcing platforms is a popular way to obtain annotations. Multiple potentially noisy answers can thus be aggregated to retrieve an underlying ground truth. However, it may be irrelevant to look for a unique ground truth when we ask crowd workers for opinions, notably when dealing with subjective phenomena such as stress. In this paper, we discuss how we can better use crowdsourced annotations with an application to automatic detection of perceived stress. Towards this aim, we first acquired video data from 44 subjects in a stressful situation and gathered answers to a binary question using a crowdsourcing platform. Then, we propose to integrate two measures derived from the set of gathered answers into the machine learning framework. First, we highlight that using the consensus level among crowd worker answers substantially increases classification accuracies. Then, we show that it is suitable to directly predict for each video the proportion of positive answers to the question from the different crowd workers. Hence, we propose a thorough study on how crowdsourced annotations can be used to enhance performance of classification and regression methods.

This paper focuses on recognizing and understanding social dimensions (the personality traits and social impressions) during small group interactions. We extract a set of audio and visual features, which are divided into three categories: intra-personal features (i.e. related to only one participant), dyadic features (i.e. related to a pair of participants) and one vs all features (i.e. related to one participant versus the other members of the group). First, we predict the personality traits (PT) and social impressions (SI) by using these three feature categories. Then, we analyse the interplay be- tween groups of features and the personality traits/social impressions of the interacting participants. The prediction is done by using Support Vector Machine and Ridge Regression which allows to determine the most dominant features for each social dimension. Our experiments show that the combination of intra-personal and one vs all features can greatly improve the prediction accuracy of personality traits and social impressions. Prediction accuracy reaches 81.37% for the social impression named ’Rank of Dominance’. Finally, we draw some interesting conclusions about the relationship between personality traits/social impressions and social features.

Facial expression can be seen as the dynamic variation of one's appearance over time. Successful recognition thus involves finding representations of high-dimensional spatiotemporal patterns that can be generalized to unseen facial morphologies and variations of the expression dynamics. In this paper, we propose to learn Random Forests from heterogeneous derivative features (e.g. facial fiducial point movements or texture variations) upon pairs of images. Those forests are conditioned on the expression label of the first frame to reduce the variability of the ongoing expression transitions. When testing on a specific frame of a video, pairs are created between this frame and the previous ones. Predictions for each previous frame are used to draw trees from Pairwise Conditional Random Forests (PCRF) whose pairwise outputs are averaged over time to produce robust estimates. As such, PCRF appears as a natural extension of Random Forests to learn spatio-temporal patterns, that leads to significant improvements over standard Random Forests as well as state-of-the-art approaches on several facial expression benchmarks.

In the context of text detection evaluation, it is essential to use protocols that are capable of describing both the quality and the quantity aspects of detection results. In this paper we propose a novel visual representation and evaluation tool that captures the whole nature of a detector by using histograms. First, two histograms (coverage and accuracy) are generated to visualize the different characteristics of a detector. Secondly, we compare these two histograms to a so called optimal one to compute representative and comparable scores. To do so, we introduce the usage of the Earth Mover's Distance as a reliable evaluation tool to estimate recall and precision scores. Results obtained on the ICDAR 2013 dataset show that this method intuitively characterizes the accuracy of a text detector and gives at a glance various useful characteristics of the analyzed algorithm.

Sensorimotor assessment aims at evaluating sensorial and motor capabilities of children who are likely to present a pervasive developmental disorder, such as autism. It relies on playful activities which are proposed by a psychomotrician expert to the child, with the intent of observing how the latter responds to various physical and cognitive stimuli. Each session is recorded so that the psychomotrician can use the video as a support for reviewing in-session impressions and drawing final conclusions. These recordings carry a wealth of information that could be exploited for research purposes and contribute to a better understanding of autism spectrum disorders. However, the systematic inspection of these data by clinical professionals would be time-consuming and impracticable. In order to make these analyses feasible, we discuss a computer vision approach to prospect precise behavior information from the available visual data acquired throughout assessment sessions.

We propose an original particle filtering-based approach combining optimization and decomposition techniques for sequential non-parametric density estimation defined in high-dimensional state spaces. Our method relies on Annealing to focus on the correct distributions and on probabilistic conditional independences defined by Dynamic Bayesian Networks to focus samples on their modes. After proving its theoretical correctness and showing its complexity, we highlight its ability to track single and multiple articulated objects both on synthetic and real video sequences. We show that our approach is particularly effective, both in terms of estimation errors and computation times.

In this paper, we propose a novel approach to speed-up the computation of the histograms of multiple overlapping non rotating regions of a single image. The idea is to exploit the overlaps between regions to minimize the number of redundant computations. More precisely, once the histogram of a region has been computed, this one can be used to compute part of the histogram of another overlapping region. For this purpose, an optimal computation order of the regions needs to be determined and we show how this can be obtained as the solution of a minimum spanning tree of a graph modeling the overlaps between regions. This tree is computed using Kruskal's algorithm and parsing it in a depth-first search manner determines precisely how the histogram of a region can be computed efficiently from that of its parent in the tree. We show that, in practical situations, this approach can outperform the well-known integral histogram both in terms of computation times and in terms of memory consumption.

In this paper, we propose a novel method to introduce spatial information in particle filters. This information may be expressed as spatial relations (orientation, distance, etc.), velocity, scaling, or shape information. Spatial information is modeled in a generic fuzzy-set framework. The fuzzy models are then introduced in the particle filter and automatically define transition and prior spatial distributions. We also propose an efficient importance distribution to produce relevant particles, which is dedicated to the proposed fuzzy framework. The fuzzy modeling provides flexibility both in the semantics of information and in the transitions from one instant to another one. This allows one to take into account situations where a tracked object changes its direction in a quite abrupt way and where poor prior information on dynamics is available, as demonstrated on synthetic data. As an illustration, two tests on real video sequences are performed in this paper. The first one concerns a classical tracking problem and shows that our approach efficiently tracks objects with complex and unknown dynamics, outperforming classical filtering techniques while using only a small number of particles. In the second experiment, we show the flexibility of our approach for modeling: Fuzzy shapes are modeled in a generic way and allow the tracking of objects with changing shape.

Online target tracking requires to solve two problems: data association and online dynamic estimation. Usually, association effectiveness is based on prior information and observation category. However, problems can occur for tracking quite similar targets under the constraints of missing data and complex motions. The lack in prior information limits the association performance. To remedy, we propose a novel method for data association inspired from the evolution of target's dynamic model and given by a global minimization of an energy. The concept amounts to measure the absolute geometric accuracy between features. The main advantage of our approach is that it is parameterless. We also integrate our method into the classical particle filter, that leads to what we call the energetic particle filter (EPF).

In this paper, we present a method based on tangent distance to estimatemotion in image sequences. Tangent distance combines an intuitive understanding and effective modeling of differences between patterns. This tool was first introduced and successfully applied in character recognition. It allows to compare patterns according to small transformations (translations, rotations, etc.). We show, how to take advantages of some properties of tangent distances to perform a robust motion estimation algorithm. Particularly, the presented algorithm can easily be adapted and optimized to various types of movements and can also be used to estimate optical flow in image sequences. Moreover, and despite a time of computation a bit long, this algorithm can be massively paralleled.

In this paper, we propose a method to track multiple deformable objects in video sequences using a recursive clustering scheme. In a first step, a set of Gabor filter banks is used to filter the difference image between two consecutive frames. Then, the moving areas are sampled by randomly positioning particles in high magnitude area of the filtered image. Finally, these points are clustered to obtain one class for each moving object. The novelty in our method is in using cluster information for the previous frame to classify new particles in the current frame. This makes our method robust to occlusions, objects entering and leaving the field of view, objects stopping and starting, and moving objects getting really close to each other.

In this article, we present a novel fast algorithm dedicated to the visualization of scalar and vectorial information lying on elliptical surfaces. As the geometry of the data is simple and not relevant, and inside structures are meaningless, standard visualization methods appear to be not well adapted to our context. Our algorithm is based on map projections, good candidates to unroll elliptical surface around a region of interest, without being computationally prohibitive. It has the merit to project, on a 2D cartographic reference, different views of scalar, as well as, vectorial 3D information lying on the surface. We display results for synthetic and cell wall simulations data.

We present the partitioning of grey-scale images using rectangular partitions, and its use for image prediction by motion compensation. Generally, a good prediction of a video frame can be made if an accurate motion estimation is computed, most often by using block matching algorithms (BMA). We propose to use an adaptive rectangular partition in order to improve the motion estimation and compensation steps. Rectangular partitioning presents the advantage of being well adapted to the image texture and to generate variable size blocks. Taking into account these properties, we test different adaptations of the classical BMA to the rectangular partitioning, in order to improve the motion compensation performance. We consider the simple translational BMA and extend this algorithm by using spatial transformations in order to cope with rotations and scalings. The last algorithm is furthermore optimized by taking into account the shape of the rectangles in the partition. Simulations show that the last method provides higher PSNR than the two other.