Predicting values of Particulate Matter concentration presents an undeniable interest, as it is key in preventing their adversarial impact on human health. Physics-based methods rely on several different variables and equations to perform this task. They are reliable, as they are based on known physical laws and models. Neural networks rely on high computing power and an important volume of data. They are faster than physics-based models, but sometimes considered as black boxes and potentially less reliable. There is therefore a need for a fast and reliable solution. Physics-Informed Neural Networks make use of all four (variables, equations, computing power and a high volume of data), which makes them more reliable than classical neural networks, and faster than physics-based methods. This paper proposes a model that predicts Particulate Matter concentration values using a variety of meteorological and optical variables as well as a Physics-Inspired loss function. The impact of different factors on the performance of this model, such as the amount of available ground truth, the use of Boundary Conditions, and the prediction time frame, is discussed in this paper as well. The model shows satisfying performance. More precisely, when compared to the best baseline method presented, our model shows a diminution of the MAE from 3.74 to 2.71 .

This comprehensive review delves deeply into the various methodologies applied to facial expression recognition (FER) through the lens of graph representation learning (GRL). Initially, we introduce the task of FER and the concepts of graph representation and GRL. Afterward, we discuss some of the most prevalent and valuable databases for this task. We explore promising approaches for graph representation in FER, including graph diffusion, spatio-temporal graphs, and multi-stream architectures. Finally, we identify future research opportunities and provide concluding remarks.

Facial Expression Recognition (FER) is an active research domain that has shown great progress recently, notably thanks to the use of large deep learning models. However, such approaches are particularly energy intensive, which makes their deployment difficult for edge devices. To address this issue, Spiking Neural Networks (SNNs) coupled with event cameras are a promising alternative, capable of processing sparse and asynchronous events with lower energy consumption. In this paper, we establish the first use of event cameras for FER, named "Event-based FER", and propose the first related benchmarks by converting popular video FER datasets to event streams. To deal with this new task, we propose "Spiking-FER", a deep convolutional SNN model, and compare it against a similar Artificial Neural Network (ANN). Experiments show that the proposed approach achieves comparable performance to the ANN architecture, while consuming less energy by orders of magnitude (up to 65.39x). In addition, an experimental study of various event-based data augmentation techniques is performed to provide insights into the efficient transformations specific to event-based FER.

Often, physical phenomena are difficult to model by a simple equation and require a lot of processing resources. Studies on Physics-Informed Neural Networks (PINNs) have repeatedly shown the interest of leveraging the information contained in context-relevant physics equations in order to guide the training, as well the ability of this type of networks to reduce the need for labeled data. Some of these analysis have also demonstrated the interest of additional knowledge through Initial and Boundary Conditions (I/BCs) in this type of context. This knowledge can take a variety of forms and shapes, among which is the one of sparse ground truths, and more precisely sparse matrices, as matrices are often well fitted to represent the spatial aspect of this type of problem. The popularity of Computer Vision techniques is partly due to their ability to take into account the spatial aspect of a given problem. The combined use of methods from these two fields therefore seems natural. This paper introduces a method for the use of Boundary Conditions for Generative Adversarial Networks (GANs), and outside the context of PINNs. The interest of leveraging the BCs with a GAN is evaluated in terms of performance, and various BC configuration and quantities are tested to discuss their impact on obtained performance.

Although facial landmark localization (FLL) approaches are becoming increasingly accurate in identifying facial components, one question remains unanswered: what is the impact of these approaches on subsequent, related tasks? In this paper, we focus on facial expression recognition (FER), where facial landmarks are used for face registration, which is a common usage. Since the common datasets for facial landmark localization do not allow for a proper measurement of performance according to the different difficulties (e.g., pose, expression, illumination, occlusion, motion blur), we also quantify the performance of recent approaches in the presence of head pose variations and facial expressions. Finally, we conduct a study of the impact of these approaches on FER. We show that the landmark accuracy achieved so far by optimizing the euclidean distance does not necessarily guarantee a gain in performance for FER. To deal with this issue, we propose a new evaluation metric for FLL that is more relevant to FER.

This paper presents "Bina-Rep", a simple representation method that converts asynchronous streams of events from event cameras to a sequence of sparse and expressive event frames. By representing multiple binary event images as a single frame of N-bit numbers, our method is able to obtain sparser and more expressive event frames thanks to the retained information about event orders in the original stream. Coupled with our proposed model based on a convolutional neural network, the reported results achieve state-of-the-art performance and repeatedly outperforms other common event representation methods. Our approach also shows competitive robustness against common image corruptions, compared to other representation techniques.

In this paper, we develop a new method that recognizes facial expressions, on the basis of an innovative Local Motion Patterns (LMP) feature. The LMP feature analyzes locally the motion distribution in order to separate consistent mouvement patterns from noise. Indeed, facial motion extracted from the face is generally noisy and without specific processing, it can hardly cope with expression recognition requirements especially for micro-expressions. Direction and magnitude statistical profiles are jointly analyzed in order to filter out noise. This work presents three main contributions. The first one is the analysis of the face skin temporal elasticity and face deformations during expression. The second one is a unified approach for both macro and micro expression recognition leading the way to supporting a wide range of expression intensities. The third one is the step forward towards in-the-wild expression recognition, dealing with challenges such as various intensity and various expression activation patterns, illumination variations and small head pose variations. Our method outperforms state-of-the-art methods for micro expression recognition and positions itself among top-ranked state-of-the-art methods for macro expression recognition.

Video facial expression recognition is useful for many applications and received much interest lately. Although some methods give good results in controlled environments (no occlusion), recognition in the presence of partial facial occlusion remains a challenging task. To handle facial occlusions, methods based on the reconstruction of the occluded part of the face have been proposed. These methods are mainly based on the texture or the geometry of the face. However, the similarity of the face movement between different persons doing the same expression seems to be a real asset for the reconstruction. In this paper we exploit this asset and propose a new method based on an auto-encoder with skip connections to reconstruct the occluded part of the face in the optical flow domain. To the best of our knowledge, this is the first work that directly reconstructs the movement for facial expression recognition. We validated our approach in the controlled CK+ datasets on which different occlusions were generated. Our experiments show that the proposed method reduces the gap in the recognition accuracy between occluded and unoccluded situations. We also compare our approach with existing state-of-the-art approaches. In order to lay the basis of a reproducible and fair comparison in the future, we also propose a new experimental protocol that includes occlusion generation and reconstruction evaluation.

Many research works focus on leveraging the complementary geometric information of indoor depth sensors in vision tasks performed by deep convolutional neural networks, notably semantic segmentation. These works deal with a specific vision task known as "RGB-D Indoor Semantic Segmentation". The challenges and resulting solutions of this task differ from its standard RGB counterpart. This results in a new active research topic. The objective of this paper is to introduce the field of Deep Convolutional Neural Networks for RGB-D Indoor Semantic Segmentation. This review presents the most popular public datasets, proposes a categorization of the strategies employed by recent contributions, evaluates the performance of the current state-of-the-art, and discusses the remaining challenges and promising directions for future works.

With the advent of neuromorphic hardware, spiking neural networks can be a good energy-efficient alternative to artificial neural networks. However, the use of spiking neural networks to perform computer vision tasks remains limited, mainly focusing on simple tasks such as digit recognition. It remains hard to deal with more complex tasks (e.g. segmentation, object detection) due to the small number of works on deep spiking neural networks for these tasks. The objective of this paper is to make the first step towards modern computer vision with supervised spiking neural networks. We propose a deep convolutional spiking neural network for the localization of a single object in a grayscale image. We propose a network based on DECOLLE, a spiking model that enables local surrogate gradient-based learning. The encouraging results reported on Oxford-IIIT-Pet validates the exploitation of spiking neural networks with a supervised learning approach for more elaborate vision tasks in the future.

This paper overviews two interdependent issues important for mining remote sensing data (e.g. images) obtained from atmospheric monitoring missions. The first issue relates to building new public datasets and benchmarks, which are hot priority of the remote sensing community. The second issue is to investigate deep learning methodologies for atmospheric data classification based on vast amount of data and without ground truth or with very limited ground truth. The targeted application is air quality assessment and prediction. Air quality is defined as the pollution level linked with several atmospheric constituent such as gases and aerosols. Low levels of air quality and thus high levels of air pollution led to increase in public health issues. The target application is the development of a fast prediction model for local and regional air quality assessment and tracking. The results of mining data will have significant implication for citizen and decision makers by providing a fast prediction and reliable air quality monitoring system able to cover the local and regional scale through intelligent extrapolation of sparse ground-based in situ measurement networks.

Human action recognition in videos is an important issue in computer vision. We propose an approach based on the integration of partial or global human body segmentation in the classification process to deal with partial movements and immobility. Experimentation on UCF101 public dataset output competitive recognition accuracy related state of the art.

Face alignment remains difficult under uncontrolled conditions due to the many variations that may considerably impact facial appearance. Recently, video-based approaches have been proposed, which take advantage of temporal coherence to improve robustness. These new approaches suffer from limited temporal connectivity. We show that early, direct pixel connectivity enables the detection of local motion patterns and the learning of a hierarchy of motion features. We integrate local motion to the two predominant models in the literature, coordinate regression networks and heatmap regression networks, and combine it with late connectivity based on recurrent neural networks. The experimental results on two datasets, 300VW and SNaP-2DFe, show that local motion improves video-based face alignment and is complementary to late temporal information. Despite the simplicity of the proposed architectures, our best model provides competitive performance with more complex models from the literature.

In uncontrolled settings occlusions occur and interfere with facial expressions recognition task. It is interesting to limit the number of regions required for face expression recognition task in order to moderate the occlusion interference. We propose a weighting scheme that ranks the facial regions needed to recognize expressions. Weights are calculated based on the contribution of each region to boost recognition in presence of various occlusions. Intelligent facial frameworks, based on region ranks are computed in presence of static occlusions (such as glasses, hair, hand on the face). Evaluations conducted using motion information as the underlying descriptor show that our approach maintains, per expression, very good recognition rates under various static occlusions occurring in uncontrolled settings.

This paper presents a novel descriptor for face depth images, generalizing the well-known Local Binary Pattern (LBP), in order to enhance its discriminative power for smooth depth images. The proposed descriptor is based on detecting shape patterns from face surfaces and enables accurate and fast description of shape variation in depth images. It is in the same form as conventional LBP, so patterns can be readily combined to form joint histograms to represent depth faces. The descriptor is computationally very simple, rapid and it is totally training-free. When we associate our descriptor in a face recognition scheme based on nearest neighbor classifier, it shows its discriminative power in depth based face recognition comparing to the conventional LBP and other extensions proposed for 3D face recognition. Many experiments are conducted on different databases in order to evaluate our method.

This paper presents cross-database evaluations of automatic appearance-based gender recognition methodology using normalized raw pixels and SVM classifier under unconstrained settings. Proposed method uses both histogram specification and feature space normalization on automatically aligned faces to achieve reliable recognition rate for real scenarios. Using a web based unconstrained training database, we applied local window search to increase generalization ability of the proposed method. Our contribution is two-fold. First we showed that aligned and normalized raw pixel intensities are providing the best performance in case of unconstrained cross-database tests than feature-based studies on unaligned faces. Second, we showed that histogram specification provides better normalization than that of histogram equalization for automatically aligned faces in large databases for gender recognition. Variety of cross-database experiments performed on uncontrolled Image of Groups (88.16%), Genki-4K (91.07%) and LFW databases (91.87%) showed that proposed method provides superior generalization ability than that of the state-of-the-art methods.

Recognizing human facial expression and emotion by computer is an interesting and challenging problem. In this paper, we propose a method for recognizing negative emotions through an appropriate representation of facial features from relevant face regions displayed in video streams and still images. A measure that is sensitive to facial movements is used in predefined regions of interest to detect the negative emotions. The experimentation has been performed on a standard dataset and live video streams and has showed promising results.

In this paper, we introduce a novel approach for face stereo reconstruction in passive stereo vision system. Our approach is based on the generation of a facial disparity map, requiring neither expensive devices nor generic face models. It consists of incorporating face properties in the disparity estimation to enhance the 3D face reconstruction. An algorithm based on the Active Shape Model (ASM) is proposed to acquire 3D sparse estimation of the face with a high confidence. Using sparse estimation as guidance and considering the face symmetry and smoothness, the dense disparity is completed. Experimental results are presented to demonstrate the reconstruction accuracy of the proposed method.

Having effective methods to access the desired images is essential nowadays with the availability of a huge amount of digital images. We propose a higher-level visual representation that enhances the traditional part-based Bag of Visual Words (BOW) representation in two aspects. Firstly, we introduce a new multilayer semantic significance analysis (MSSA) model to select Semantically Significant Visual Words (SSVWs) from the classical visual words in order to overcome the noisiness of the feature quantization process. Secondly, we strengthen the discrimination power of SSVWs by constructing Semantically Significant Visual Phrases (SSVPs) from frequently co-occurring SSVWs in the same local context that are semantically coherent. Finally, the large-scale extensive experimental results show that the proposed higher-level visual representation outperforms the traditional part-based image representation in social image retrieval.

The recent growth of multimedia requires an extensive use of metadata for their management. However, a uniform access to metadata is necessary in order to take advantage of them. In this context, several techniques for achieving metadata interoperability have been developed. Most of these techniques focus on matching schemas defined by using one schema description language. The few existing matching systems that support schemas from different languages present some limitations. In this paper we present a new integration system supporting schemas from different description languages. Moreover, the proposed matching process makes use of several types of information (linguistic, semantic and structural) in a manner that increases the matching accuracy.

In this paper, we present a new approach to count the number of people that cross a counting line from monocular video images. The proposed approach accumulates image slices and estimates the optical flow on them. Then, it performs an online blob detection on these slices in order to extract the crossing persons. The number of persons associated to each blob is determined using a linear regression model applied to blob features which are the position, velocity, orientation and size. The proposed approach is validated on several datasets captured using either a vertical overhead or an oblique mounted camera. The real-time performance and the high counting accuracy of this approach in indoor and outdoor environments are also demonstrated.

In this paper, we demonstrate in the first that the amount (lengths and widths) of symmetrical parts on face are good and robust geometric features for head pose estimation. Secondly, we selected from all proposed algorithms of symmetry detection in the literature, the more suitable algorithm and the values of the proposed features are computed using some head poses which correspond to Yaw and Pitch motions. The obtained results demonstrate the validity of the proposed approach.

In this paper, we present an effective method for human action recognition using statistical models based on optical flow orientations. We compute a distribution mixture over motion orientations at each spatial location of the video sequence. The set of estimated distributions constitutes the direction model, which is used as a mid-level feature for the video sequence. We recognize human actions using a distance metric to compare the direction model of a query sequence with the direction models of training sequences. The experimentations have been performed on standard datasets and have showed promising results.

In this paper, we present an unconstrained visual gaze estimation system. The proposed method extracts the visual field of view of a person looking at a target scene in order to estimate the approximate location of interest (visual gaze). The novelty of the system is the joint use of head pose and eye location information to fine tune the visual gaze estimated by the head pose only, so that the system can be used in multiple scenarios. The improvements obtained by the proposed approach are validated using the Boston University head pose dataset, on which the standard deviation of the joint visual gaze estimation improved by 61:06% horizontally and 52:23% vertically with respect to the gaze estimation obtained by the head pose only. A user study shows the potential of the proposed system.

The recent growth of multimedia in our lives requires an extensive use of metadata for multimedia management. Consequently, many metadata standards have appeared. Using these standards has become very complicated since they have been developed by independent communities. The content and context are usually described using several metadata standards. Accordingly, a multimedia user must be able to interpret all these standards. In this context, several metadata integration techniques have been proposed in order to deal with this challenge. These integrations are made by domain experts which is costly and time-consuming. This paper presents a new system for a semi-automatic integration of multimedia metadata. This system will automatically map between metadata needed by the user and those encoded in different formats. The integration process makes use of several types of information: XML Schema entity names, their corresponding comments as well as the hierarchical features of XML Schema. Our experimental results demonstrate the integration benefits of the proposed system.

Having effective methods to access the desired images is essential nowadays with the availability of huge amount of digital images. The proposed approach is based on an analogy between content-based image retrieval and text retrieval. The aim of the approach is to build a meaningful mid-level representation of images to be used later for matching between a query image and other images in the desired database. The approach is based firstly on constructing different visual words using local patch extraction and fusion of descriptors. Secondly, we introduce a new method using multilayer pLSA to eliminate the noisiest words generated by the vocabulary building process. Thirdly, a new spatial weighting scheme is introduced that consists in weighting visual words according to the probability of each visual word to belong to each of the n Gaussian. Finally, we construct visual phrases from groups of visual words that are involved in strong association rules. Experimental results show that our approach outperforms the results of traditional image retrieval techniques.

In this paper we will present an approach for multimedia scenario management in a database system that considers: an object-oriented method for multimedia scenarios modeling; time-interval and causal relations models for the specification of known and unknown multimedia object playing duration; temporal specification language; time Petri net automatic generation based on temporal specifications; and finally automatic detection of user temporal specification errors and contradictions.

This paper proposes a new approach to detect exceptional motion frames from real videos irrespective of both static and dynamic backgrounds. The approach is based on the use of the spatiotemporal region of interest (ST-RoI) features obtained from ST-RoI, which is estimated using motion history image (MHI). Within ST-RoI, exceptional motion makes the motion vectors (e.g., directions) change a lot as compared to normal motion. The normalized continuous rank-increase measure (NCRIM) calculated from the ST-RoI features has been used as the judgement index for determining normal or exceptional motion frame. To demonstrate the interest of the proposed approach, the results based on the detection of exceptional motion frames in real videos obtained from a single camera placed on the escalator exit in an airport have been presented.

In this technical demonstration, we present a tool that projects on a target scene the visual gaze of the people passing in front of it. The target scene could be a large plasma screen, an advertising poster, a shelf or a shop window. The projection of the visual gaze corresponds to the coordinates of a region that represents the person's location of interest in the target scene. It is an important problem with many applications that try to understand human behavior in a controlled environment such as in security or customized marketing. The visual gaze projection is influenced by the parameters of the camera, the settings of the target scene, the method used for the estimation of the gaze (e.g. the information about the head pose, the location of eyes centers and corners, or any combination of them), and the visual gaze starting point.

Video-surveillance systems are becoming more and more autonomous in the detection and the reporting of abnormal events. In this context, this paper presents an approach to detect abnormal situations in crowded scenes by analyzing the motion aspect instead of tracking subjects one by one. The proposed approach estimates sudden changes and abnormal motion variations of a set of points of interest (POI). The number of tracked POIs is reduced using a mask that corresponds to hot areas of the built motion heat map. The approach detects events where local motion variation is important compared to previous events. Optical flow techniques are used to extract information such as density, direction and velocity. To demonstrate the interest of the approach, we present the results on the detection of collapsing events in real videos of airport escalator exits.

In this paper we present an application of computer vision techniques to obtain specific information about the behaviour of the people passing in front of a target scene. This is done by analyzing videos captured by cameras monitoring an area under surveillance. The target scene can be a large plasma screen, a projected image, an advertising poster or a shop window. An example of the type of information that can be extracted is the number of people passing in the area (possibly even making a stop), those who are interested in the target scene (i.e. looking in its direction), and the specific locations of interest inside the target scene. The person detection counts the number of persons subsequently followed by the person tracking which determines who is stopping and who is moving. The head pose estimation denotes whether they are looking or not at the target scene. Finally the projection of the visual field extracts the location of interest in the target scene. All these tasks are improved by taking in account the environmental information.

This paper presents the use of a template based method in order to make a head pose estimation. As an image classification problem the aim of this kind of techniques is to convert the input head image into a feature vector. The feature vectors of different persons taken at the same pose will serve to learn a head pose classifier. The aim of this work is to estimate the head pose of people looking at a target scene in order to extract the location of their gaze in the scene.

We present a tool that automatically detects abnormal situations in crowded scenes in real time. The followed approach analyzes the general motion aspect, instead of tracking subjects one by one, by detecting abnormal optical flow patterns of tracked KLT points. The number of tracked points is reduced by using a learned mask. We define a measure that describes the situation abnormality based on crowd density, direction variance and distribution, mean velocity and sometimes trajectory matching. To demonstrate the interest of this approach, we present the results on the detection of collapsing events in real videos of airport escalator exits.

Eye movements are arguably the most natural and repetitive movement of a human being. The most mundane activity, such as watching television or reading a newspaper, involves this automatic activity which consists of shifting our gaze from one point to another. Identification of the components of eye movements (fixations and saccades) is an essential part in the analysis of visual behavior because these types of movements provide the basic elements used by further investigations of human vision. However, many of the algorithms that detect fixations present a number of problems. In this paper, we present the results of a new fixation identification technique that is based on clustering of eye positions, using projections and a projection aggregation applied to static pictures. We also present results of a new method that computes dispersion of eye fixations in videos considering a multi-user environment.

International audience

International audience

Our demo focuses on eye tracking on web, image and video data. We use some state-of-the-art measurements, such as scan path, to determine how the user sees web documents, images and videos. Our approach is characterised by automatic eye/gaze tracking with non intrusive sensors, mainly infrared cameras of web, image and video documents. We analyse eye/gaze tracking concerns spatial regions of static documents (images and web pages) and spatial zones of dynamic documents (video, sequence of web pages hyperlinked). In the context of dynamic documents, the eye/gaze tracking is processed image-per -image in video documents, and page-per-page in web documents. The result is more rough on video, and more accurate on images and hyperlinked web pages. Eye/gaze tracking on video is relatively new and unexplored in the literature.

Extending beyond the boundaries of science, art, and culture, content-based multimedia information retrieval provides new paradigms and methods for searching through the myriad variety of media all over the world. This survey reviews 100+ recent articles on content-based multimedia information retrieval and discusses their role in current research directions which include browsing and search paradigms, user studies, affective computing, learning, semantic queries, new features and media types, high performance indexing, and evaluation techniques. Based on the current state of the art, we discuss the major challenges for the future.

Motivated by the needs for efficient indexing structures adapted to real applications in video database, we present a new indexing structure named Kpyr. In Kpyr, we use a clustering algorithm to partition the data space into sub-spaces on which we apply Pyramid technique (S. Berchtold, et al., 1998). We thus reduce the search space concerned by a query and improve the performances. We show that our approach provides interesting and performing experimental results for both K-nearest neighbors and window queries

Whereas the early frequent pattern mining methods admitted only relatively simple data and pattern formats (e.g., sets, sequences, etc.), there is nowadays a clear push towards the integration of ever larger portions of domain knowledge in the mining process in order to increase the precision and the abstraction level of the retrieved patterns and hence ease their interpretation. We present here a practically motivated study of a frequent pattern extraction from sequences of data objects that are described within a domain ontology. As the complexity of the descriptive structures is high, an entire framework for the pattern extraction process had to be defined. The key elements thereof are a pair of descriptive languages, one for individual data and another one for generic patterns, a generality relation between patterns, and an Apriori-like method for pattern mining.

In spite of important efforts in content-based indexing and retrieval during these last years, seeking relevant and accurate images remains a very difficult query. In the state-of-the-art approaches, the retrieval task may be efficient for some queries in which the semantic content of the query can be easily translated into visual features. For example, finding images of fires is simple because fires are characterized by specific colors (yellow and red). However, it is not efficient in other application fields in which the semantic content of the query is not easily translated into visual features. For example, finding images of birds during migrations is not easy because the system has to understand the query semantic. In the query, the basic visual features may be useful (a bird is characterized by a texture and a color), but they are not sufficient. What is missing is the generalization capability. Birds during migrations belong to the same repository of birds, so they share common associations among basic features (e.g., textures and colors) that the user cannot specify explicitly. We present an approach that discovers hidden associations among features during image indexing. These associations discriminate image repositories. The best associations are selected on the basis of measures of confidence. To reduce the combinatory explosion of associations, because images of the database contain very large numbers of colors and textures, we consider a visual dictionary that group together similar colors and textures.

This paper deals with the challenge of extending classical image retrieval by including visual rules. The visual rules are extracted automatically from classes of images. They contribute to making the retrieval process more accurate. The visual-rules extraction is based on symbolic representations of image descriptors. The symbolic representations are the results of color and texture clustering.

International audience

International audience