Facial expression recognition (FER) systems frequently suffer significant performance degradation when confronted with head pose variations, a pervasive challenge in real-world applications ranging from healthcare monitoring to human–computer interaction. While existing frontal view synthesis (FVS) methods attempt to address this issue, they predominantly operate in the appearance domain, often introducing artifacts that distort the subtle motion patterns crucial for accurate expression analysis. We present eMotion-GAN, a two-stage generative motion-domain framework that fundamentally rethinks frontalization by decomposing facial dynamics into two distinct components: (1) expression-related motion stemming from muscle activity, and (2) pose-related motion acting as noise. We conducted extensive evaluations using several widely recognized dynamic FER datasets, which encompass sequences exhibiting various degrees of head pose variations in both intensity and orientation. Our results demonstrate the effectiveness of our approach in significantly reducing the FER performance gap between frontal and non-frontal faces. Specifically, we achieved a FER improvement of up to +5% for small pose variations and up to +20% improvement for larger pose variations. Code and pre-trained models are available at: https://github.com/o-ikne/eMotion-GAN.git . • Treats head pose as structured noise in optical flow for robust frontalization. • Needs no landmarks; not affected by inaccurate facial landmark detection. • Splits facial motion into pose and expression; gains 20% FER accuracy for poses. • Enables expression transfer for animation and facial data augmentation. • Reduces artifacts and outperforms appearance-based frontalization methods.

International audience

Facial expression spotting is an effective metric for categorizing human behavior changes. It refers to the precise localization of the temporal intervals in a sequence where a visual event occurs in a face. In this paper, we propose an innovative framework, which relies on the consistency in terms of orientation and intensity of the local facial motions. First, we build local facial motion consistency maps to differentiate expression-related facial motion from facial noise. Then, these maps are fed into a recurrent neural network to precisely delineate the temporal progression of facial expression activation. Extensive evaluations were undertaken on SNAP-2DFE dataset demonstrating the effectiveness of the proposed framework in temporally segmenting expression activation in presence of low or high head pose variations

Physical adversarial attacks pose a significant practical threat as it deceives deep learning systems operating in the real world by producing prominent and maliciously designed physical perturbations. Emphasizing the evaluation of naturalness is crucial in such attacks, as humans can easily detect unnatural manipulations. To address this, recent work has proposed leveraging generative adversarial networks (GANs) to generate naturalistic patches, which may seem visually suspicious and evade human’s attention. However, these approaches suffer from a limited latent space which leads to an inevitable trade-off between naturalness and attack efficiency. In this paper, we propose a novel approach to generate naturalistic and inconspicuous adversarial patches. Specifically, we redefine the optimization problem by introducing an additional loss term to the total loss. This term works as a semantic constraint to ensure that the generated camouflage pattern holds semantic meaning rather than arbitrary patterns. It leverages similarity metrics-based loss that we optimize within the global adversarial objective function. Our technique is based on directly manipulating the pixel values in the patch, which gives higher flexibility and larger space compared to the GAN-based techniques that are based on indirectly optimizing the patch by modifying the latent vector. Our attack achieves superior success rate of up to $91.19 \%$ and $72 \%$, respectively, in the digital world and when deployed in smart cameras at the edge compared to the GAN-based approach.

Spike Timing-Dependent Plasticity (STDP) is a promising substitute to backpropagation for local training of Spiking Neural Networks (SNNs) on neuromorphic hardware. STDP allows SNNs to address classification tasks by combining unsupervised STDP for feature extraction and supervised STDP for classification. Unsupervised STDP is usually employed with Winner-Takes-All (WTA) competition to learn distinct patterns. However, WTA for supervised STDP classification faces unbalanced competition challenges. In this paper, we propose a method to effectively implement WTA competition in a spiking classification layer employing first-spike coding and supervised STDP training. We introduce the Neuronal Competition Group (NCG), an architecture that improves classification capabilities by promoting the learning of various patterns per class. An NCG is a group of neurons mapped to a specific class, implementing intra-class WTA and a novel competition regulation mechanism based on two-compartment thresholds. We incorporate our proposed architecture into spiking classification layers trained with state-of-the-art supervised STDP rules. On top of two different unsupervised feature extractors, we obtain significant accuracy improvements on image recognition datasets such as CIFAR-10 and CIFAR-100. We show that our competition regulation mechanism is crucial for ensuring balanced competition and improved class separation.

Although facial landmark localization (FLL) approaches are becoming increasingly accurate in identifying facial components, one question remains unanswered: what is the impact of these approaches on subsequent, related tasks? In this paper, we focus on facial expression recognition (FER), where facial landmarks are used for face registration, which is a common usage. Since the common datasets for facial landmark localization do not allow for a proper measurement of performance according to the different difficulties (e.g., pose, expression, illumination, occlusion, motion blur), we also quantify the performance of recent approaches in the presence of head pose variations and facial expressions. Finally, we conduct a study of the impact of these approaches on FER. We show that the landmark accuracy achieved so far by optimizing the euclidean distance does not necessarily guarantee a gain in performance for FER. To deal with this issue, we propose a new evaluation metric for FLL that is more relevant to FER.

Current advances in technology have highlighted the importance of video analysis in the domain of computer vision. However, video analysis has considerably high computational costs with traditional artificial neural networks (ANNs). Spiking neural networks (SNNs) are third generation biologically plausible models that process the information in the form of spikes. Unsupervised learning with SNNs using the spike timing dependent plasticity (STDP) rule has the potential to overcome some bottlenecks of regular artificial neural networks, but STDP-based SNNs are still immature and their performance is far behind that of ANNs. In this work, we study the performance of SNNs when challenged with the task of human action recognition, because this task has many real-time applications in computer vision, such as video surveillance. In this paper we introduce a multi-layered 3D convolutional SNN model trained with unsupervised STDP. We compare the performance of this model to those of a 2D STDP-based SNN when challenged with the KTH and Weizmann datasets. We also compare single-layer and multi-layer versions of these models in order to get an accurate assessment of their performance. We show that STDP-based convolutional SNNs can learn motion patterns using 3D kernels, thus enabling motion-based recognition from videos. Finally, we give evidence that 3D convolution is superior to 2D convolution with STDP-based SNNs, especially when dealing with long video sequences.

In this paper, we develop a new method that recognizes facial expressions, on the basis of an innovative Local Motion Patterns (LMP) feature. The LMP feature analyzes locally the motion distribution in order to separate consistent mouvement patterns from noise. Indeed, facial motion extracted from the face is generally noisy and without specific processing, it can hardly cope with expression recognition requirements especially for micro-expressions. Direction and magnitude statistical profiles are jointly analyzed in order to filter out noise. This work presents three main contributions. The first one is the analysis of the face skin temporal elasticity and face deformations during expression. The second one is a unified approach for both macro and micro expression recognition leading the way to supporting a wide range of expression intensities. The third one is the step forward towards in-the-wild expression recognition, dealing with challenges such as various intensity and various expression activation patterns, illumination variations and small head pose variations. Our method outperforms state-of-the-art methods for micro expression recognition and positions itself among top-ranked state-of-the-art methods for macro expression recognition.

Video facial expression recognition is useful for many applications and received much interest lately. Although some methods give good results in controlled environments (no occlusion), recognition in the presence of partial facial occlusion remains a challenging task. To handle facial occlusions, methods based on the reconstruction of the occluded part of the face have been proposed. These methods are mainly based on the texture or the geometry of the face. However, the similarity of the face movement between different persons doing the same expression seems to be a real asset for the reconstruction. In this paper we exploit this asset and propose a new method based on an auto-encoder with skip connections to reconstruct the occluded part of the face in the optical flow domain. To the best of our knowledge, this is the first work that directly reconstructs the movement for facial expression recognition. We validated our approach in the controlled CK+ datasets on which different occlusions were generated. Our experiments show that the proposed method reduces the gap in the recognition accuracy between occluded and unoccluded situations. We also compare our approach with existing state-of-the-art approaches. In order to lay the basis of a reproducible and fair comparison in the future, we also propose a new experimental protocol that includes occlusion generation and reconstruction evaluation.

There has been an increasing interest in spiking neural networks in recent years. SNNs are seen as hypothetical solutions for the bottlenecks of ANNs in pattern recognition, such as energy efficiency [1]. But current methods such as ANN-to-SNN conversion and back-propagation do not take full advantage of these networks, and unsupervised methods have not yet reached a success comparable to advanced artificial neural networks. It is important to study the behavior of SNNs trained with unsupervised learning methods such as spike-timing dependent plasticity (STDP) on video classification tasks, including mechanisms to model motion information using spikes, as this information is critical for video understanding. This paper presents multiple methods of transposing temporal information into a static format, and then transforming the visual information into spikes using latency coding. These methods are paired with two types of temporal fusion known as early and late fusion, and are used to help the spiking neural network in capturing the spatio-temporal features from videos. In this paper, we rely on the network architecture of a convolutional spiking neural network trained with STDP, and we test the performance of this network when challenged with action recognition tasks. Understanding how a spiking neural network responds to different methods of movement extraction and representation can help reduce the performance gap between SNNs and ANNs. In this paper we show the effect of the similarity in the shape and speed of certain actions on action recognition with spiking neural networks, we also highlight the effectiveness of some methods compared to others.

The use of e-service platforms has become essential for many applications (administrative documents, online shopping, reservations). Although these platforms have improved significantly the user experience, unexpected and stressful situations can occur. Navigation problems (latency, missing information, poor ergonomics) are not always reported to the designers. To address this problem, we propose a multimodal dataset (video, audio, and physiological data) to help implicitly quantify the impact of navigation problems on users when using an e-service platform. A scenario has been designed to generate various navigation problems which can lead to changes in user behaviour. A baseline is proposed to spot changes in user behaviour, opening the way towards automatically qualifying user experiences while using e-service platforms.

In recent years, spiking neural networks (SNNs) emerge as an alternative to deep neural networks (DNNs). SNNs present a higher computational efficiency - using low-power neuromorphic hardware - and require less labeled data for training - using local and unsupervised learning rules such as spike timing-dependent plasticity (STDP). SNNs have proven their effectiveness in image classification on simple datasets such as MNIST. However, to process natural images, a pre-processing step is required. Difference-of-Gaussians (DoG) filtering is typically used together with on-center/off-center coding, but it results in a loss of information that decreases the classification performance. In this paper, we propose to use whitening as a pre-processing step before learning features with STDP. Experiments on CIFAR-10 show that whitening allows STDP to learn visual features that are visually closer to the ones learned with standard neural networks, with a significantly increased classification performance as compared to DoG filtering. We also propose an approximation of whitening as convolution kernels that is computationally cheaper to learn and more suited to be implemented on neuromorphic hardware. Experiments on CIFAR-10 show that it performs similarly to regular whitening. Cross-dataset experiments on CIFAR-10 and STL-10 also show that it is stable across datasets, making it possible to learn a single whitening transformation to process different datasets.

Spiking neural networks (SNNs) are good candidates to produce ultra-energy-efficient hardware. However, the performance of these models is currently behind traditional methods. Introducing multi-layered SNNs is a promising way to reduce this gap. We propose in this paper a new threshold adaptation system which uses a timestamp objective at which neurons should fire. We show that our method leads to state-of-the-art classification rates on the MNIST dataset (98.60%) and the Faces/Motorbikes dataset (99.46%) with an unsupervised SNN followed by a linear SVM. We also investigate the sparsity level of the network by testing different inhibition policies and STDP rules.

Face alignment remains difficult under uncontrolled conditions due to the many variations that may considerably impact facial appearance. Recently, video-based approaches have been proposed, which take advantage of temporal coherence to improve robustness. These new approaches suffer from limited temporal connectivity. We show that early, direct pixel connectivity enables the detection of local motion patterns and the learning of a hierarchy of motion features. We integrate local motion to the two predominant models in the literature, coordinate regression networks and heatmap regression networks, and combine it with late connectivity based on recurrent neural networks. The experimental results on two datasets, 300VW and SNaP-2DFe, show that local motion improves video-based face alignment and is complementary to late temporal information. Despite the simplicity of the proposed architectures, our best model provides competitive performance with more complex models from the literature.

This paper introduces an original video dataset containing dynamic index finger gestures in a mobile context. The dataset consists of 746 video sequences of 6 index finger gestures: left, right, up, down, tap, circle. The video sequences are obtained from a mobile phone's rear camera equipped with a wide angle lens, thus the dataset features particular challenges due to background motion and the distorted field of view. We present a baseline method that uses optical flow features in the form of a histogram to represent the motion information in the video sequences.

In uncontrolled settings occlusions occur and interfere with facial expressions recognition task. It is interesting to limit the number of regions required for face expression recognition task in order to moderate the occlusion interference. We propose a weighting scheme that ranks the facial regions needed to recognize expressions. Weights are calculated based on the contribution of each region to boost recognition in presence of various occlusions. Intelligent facial frameworks, based on region ranks are computed in presence of static occlusions (such as glasses, hair, hand on the face). Evaluations conducted using motion information as the underlying descriptor show that our approach maintains, per expression, very good recognition rates under various static occlusions occurring in uncontrolled settings.

Image recognition tasks require multi-layer networks to achieve good performance on complex data. However, building multi-layer spiking neural networks (SNN) still remains unreachable. One cause is that the learning mechanism of these models decreases the spiking activity throughout the layers. We propose three mechanisms to solve this issue without impacting the performance of the network: target frequency threshold adaptation, which forces neurons to reach a desired frequency, binary coding, which improves the performance of the network at high levels of activity, and mirrored STDP, which improves the convergence of the training. Experiments on single layer networks show that these mechanisms preserve both the recognition rate and the level of spiking activity.

This paper presents cross-database evaluations of automatic appearance-based gender recognition methodology using normalized raw pixels and SVM classifier under unconstrained settings. Proposed method uses both histogram specification and feature space normalization on automatically aligned faces to achieve reliable recognition rate for real scenarios. Using a web based unconstrained training database, we applied local window search to increase generalization ability of the proposed method. Our contribution is two-fold. First we showed that aligned and normalized raw pixel intensities are providing the best performance in case of unconstrained cross-database tests than feature-based studies on unaligned faces. Second, we showed that histogram specification provides better normalization than that of histogram equalization for automatically aligned faces in large databases for gender recognition. Variety of cross-database experiments performed on uncontrolled Image of Groups (88.16%), Genki-4K (91.07%) and LFW databases (91.87%) showed that proposed method provides superior generalization ability than that of the state-of-the-art methods.

Recognizing human facial expression and emotion by computer is an interesting and challenging problem. In this paper, we propose a method for recognizing negative emotions through an appropriate representation of facial features from relevant face regions displayed in video streams and still images. A measure that is sensitive to facial movements is used in predefined regions of interest to detect the negative emotions. The experimentation has been performed on a standard dataset and live video streams and has showed promising results.

The recent growth of multimedia requires an extensive use of metadata for their management. However, a uniform access to metadata is necessary in order to take advantage of them. In this context, several techniques for achieving metadata interoperability have been developed. Most of these techniques focus on matching schemas defined by using one schema description language. The few existing matching systems that support schemas from different languages present some limitations. In this paper we present a new integration system supporting schemas from different description languages. Moreover, the proposed matching process makes use of several types of information (linguistic, semantic and structural) in a manner that increases the matching accuracy.

The recent growth of multimedia in our lives requires an extensive use of metadata for multimedia management. Consequently, many metadata standards have appeared. Using these standards has become very complicated since they have been developed by independent communities. The content and context are usually described using several metadata standards. Accordingly, a multimedia user must be able to interpret all these standards. In this context, several metadata integration techniques have been proposed in order to deal with this challenge. These integrations are made by domain experts which is costly and time-consuming. This paper presents a new system for a semi-automatic integration of multimedia metadata. This system will automatically map between metadata needed by the user and those encoded in different formats. The integration process makes use of several types of information: XML Schema entity names, their corresponding comments as well as the hierarchical features of XML Schema. Our experimental results demonstrate the integration benefits of the proposed system.

The progress and the continuous evolution of computer capacities, as well as the emergence of the X3D standard have recently boosted the 3D domain. Associating some semantics with 3D contents becomes a major issue specially for reusing such contents or pieces of content after having extracted them from existing 3D scenes. In this paper, we address this issue by proposing a generic semantic annotation model for 3D, called 3DSEAM (3D semantics annotation model). 3DSEAM aims at indexing 3D contents considering visual, geometric and semantic aspects. 3DSEAM is instantiated using MPEG 7 that we have extended with 3D specific locators. These locators link some visual, geometric and semantic features to the corresponding X3D fragments. These features can then be used for indexing and querying.

The evolving desktop computer capacities and the emergence of the X3D standard offer a new boost to 3D domain. Giving sense to 3D content becomes a major issue specially for reusing such a content extracted from existing 3D scenes. In this paper, we address this issue by proposing a generic semantic annotation model for 3D called 3DSEAM (3D SEmantics Annotation Model). 3DSEAM aims at indexing 3D content considering visual, geometric and semantic aspects. 3DSEAM is instantiated using MPEG-7 extended with 3D specific locators. These locators link the visual, geometric and semantic features of a 3D content to the corresponding X3D fragment.

This paper addresses the adaptation of dynamic and synchronized multimedia presentations built by querying XML compatible data sources. We provide WIS designers with facilities for describing presentations whose content is not known at design time in terms of quantity, but only after the execution of queries. Our approach relies on the definition of a template. A template consists of a model that aims at automatically adapting the multimedia content of a presentation to both the user's profile and the characteristics of her/his access device. We show here how a template is built and how adaptations of the presentation are performed when the quantity of information and/or the material capabilities of the access devices (e.g. display size), do not match the template's spatiotemporal specifications.