The widespread adoption of smartphones with high-resolution cameras has driven a surge in image capture, particularly for selfies, food, and landscapes, which dominate social media. Efficient image compression is essential to reduce storage and transmission requirements while maintaining visual quality. Traditional methods like JPEG and JPEG2000 have reached their limits, making learning-based image compression (LIC) a promising alternative. However, most LIC models, such as SegPIC [2], are trained on general-purpose datasets like COCO, limiting their effectiveness for smartphone-specific content.

Current deepfake detection methods are prone to overfitting to specific deepfake artifacts and often struggle with genuine images that have undergone compression and other image processing operations. These processes can obscure indicators of forgery, leading to inaccurate decisions. This paper aims to redefine the boundary between real and fake images by narrowing the definition of authentic samples to a stage closer to the radiance of the scene as captured by the sensor, prior to any transformations by an Image Signal Processor (ISP). Our proposed method bypasses ISP processing steps, such as denoising, white balance, and demosaicing, which are embedded in camera hardware. This unaltered preservation makes raw data an ideal starting point for deepfake detection. Given the scarcity of large-scale datasets designed for training on raw images, we propose a methodological approach to train our model on raw image data. Our method demonstrated state-of-the-art performance on the CDF dataset and showed competitive results across other RGB domain deepfake detection datasets. The model developed in this study is available at https://github.com/DeepFaux/Deepfake-Detection-with-RAW-Data.

Body worn cameras (BWCs) have grown in popularity over the last decade. They are becoming one of the most essential tools used by law enforcement for surveillance. Limited academic research has been conducted on image and video processing using BWCs. The number of datasets based on BWCs is incredibly few. For this objective, we introduce FALEB (Face, Action, License, Egocentric look using Body worn cameras): a multimodal dataset for image processing using BWCs. This work includes two distinct insights: (1) introduction of a dataset specific to body cameras with the applications of facial recognition, action recognition, license plate recognition, and egocentric look, and (2) baseline experiments on the dataset. We investigate the methodologies employed in extracting meaningful patterns from BWC footage, the effectiveness of deep learning models in recognizing faces and cat-egorizing actions, and the potential applications of these advancements. By focusing on events uniquely relevant to law enforcement scenarios, we ensure that our dataset meets the practical needs of the authorities and researchers aiming to enhance public safety through advanced video analysis technologies. The complete dataset is available for research purposes and can be accessed by contacting the authors1.

The rapid advancement of deepfake technology has raised significant concerns about the authenticity of digital media and its potential misuse. While much progress has been made in developing methods to detect whether a video is fake or not, a critical question remains: Can we go one step further? What additional information can be derived once a deepfake is identified? Beyond merely flagging manipulated content, understanding the source of the manipulation holds significant value for forensics and investigation. This paper addresses one aspect of this challenge by demonstrating how to recover information from the driving video, i.e., the input video guiding the deepfake generation, to identify the person acting in the driving video (suspected driver). By learning facial expressions and movements unique to a suspected driver, we can identify which deepfake has been generated using videos of the suspected driver in a pool of deepfakes. While the current limitation of this work implies having a large quantity of data concerning your suspected identity, this work proves the feasibility of deducing information on driving videos directly from the deepfakes. Code available at: https://github.com/Thiresias/BRT-driver-identitfication

Politicians and government leaders are critical targets for deepfake attacks. A single deepfake involving these individuals can severely damage their careers or, in extreme cases, pose a national security threat. Attackers can leverage vast amounts of publicly available audio and video recordings to train their models, making this threat even more pressing. In response, specialized deepfake detectors have been developed to focus on detecting deepfakes targeting a specific Person of Interest (POI). By learning facial expressions and movements unique to the POI, these detectors can identify inconsistencies in deepfakes where these authentic attributes are absent. However, previous methods relied on Facial Action Units, which offer an incomplete representation of the POI’s behavior. In this paper, we propose a novel approach to learning POI-specific movements without requiring deepfake samples during training, making it independent of any deepfake generation methods. Although our technique is speaker-dependent, it provides a robust solution for protecting high-profile individuals who are particularly exposed to deepfake threats.

In the field of action recognition, event cameras have marked a breakthrough by capturing motion dynamics beyond the capability of traditional cameras, thanks to their high temporal sensitivity. However, the asynchronous and sparse nature of event data challenges their use with traditional convolutional neural networks (CNNs). The E2VID model offers a solution by transforming event data into continuous video frames, enabling the use of standard CNNs for event-based data analysis. However, it struggles with accurately capturing motion speed variations and pauses, limiting its effectiveness in scenarios where temporal dynamics are crucial. In response, we introduce TIME-E2V, which integrates spatial embeddings from E2VID with LSTM-derived temporal embeddings from frame timestamps. This combination is processed by a modified 3D convolutional network (C3D), leveraging its inherent strengths in video analysis. Our proposed approach not only overcomes E2VID’s challenges but also delivers competitive performance across a wide range of dynamic scenes with the leading action recognition networks for event cameras, including those based on Spiking Neural Networks.

Artificial intelligence-based face recognition solutions are becoming increasingly popular. Therefore, it is crucial to fully understand and explain how these technologies work in order to make them more effective and acceptable to society. This is the goal of the CHIST-ERA project XAIface, the final results of which are reported in this article: a framework and toolkit for improving AI decision explainability, in the context of automated face recognition, through several novel methods are presented. These methods are integrated into an end-to-end face recognition demonstrator system, which facilitates studying the impact of various influencing factors and system processes on recognition performance. By doing so, we can visually explain the decisions made by the face verification pipeline for specific instances in our test set using heatmaps and locally interpretable features. Furthermore, we offer a comprehensive explanation of the end-to-end model by examining the relationship between verification failures and misclassifications of soft biometric facial traits.

In the field of face recognition and verification, the practice of face frontalization is conventionally regarded as a standard technique. However, traditional frontalization methods often manipulate original facial images, relying on symmetric cues or data distributions from machine learning model training, which may lead to the distortion of genuine facial features. To tackle these challenges, this paper presents AlignFace, a novel face normalization algorithm specifically designed for preprocessing in the context of face verification. Distinct from existing methods, AlignFace uniquely aligns head pose, expression, and illumination conditions between image pairs. This is achieved by estimating these parameters in one image and reconstructing the other to correspond, all while meticulously preserving each image’s distinct identity features. Such an approach not only ensures a more authentic representation of facial characteristics but also maintains the integrity of real features in one of the images. Our extensive experimental evaluations, conducted on benchmark datasets such as LFW, CFP, AgeDB, and IJB-B, underscore the effectiveness of AlignFace. The comparative analysis with existing methods demonstrates its state-of-the-art performance, highlighting substantial advancements in face verification accuracy. For further research and replication, the code for our method is accessible at: https://github.com/SaharHusseini/ALIGNFACE.

Weight is a soft biometric trait which estimation is useful in numerous health related applications such as remote estimation from a health professional or at-home daily monitoring. In scenarios when a scale is unavailable or the subject is unable to cooperate, i.e. road accidents, estimating a person's weight from face appearance allows for a contactless measurement. In this article, we define an optimal transfer learning protocol for a ResNet50 architecture obtaining better performances than the state-of-the-art thus moving one step forward in closing the gap between remote weight estimation and physical devices. We also demonstrate that gender-splitting, image cropping and hair occlusion play an important role in weight estimation which might not necessarily be the case in face recognition. We use up-to-date explainability tools to illustrate and validate our assumptions. We conduct extensive simulations on the most popular publicly available face dataset annotated by weight to ensure a fair comparison with other approaches and we aim to overcome its flaws by presenting our self-collected database composed of 400 new images.

In recent years we have assisted the proliferation of deepfakes. The progress concerning both creation and, to a certain extent, automatic detection is spectacular. Nevertheless, there is a lack of protocols concerning the objective evaluation of deepfakes. In this article, we focus on the quality of head motion replication by deepfake generators that use a pilot video of a particular person to animate a single source image of another person. We test several publicly available generators to reproduce particular head movements (rotation around yaw, pitch, and a combination of pitch and yaw). In order to measure how well the deepfake generators replicate head motion, a 3D head model is utilized to render video sequences with the known head pose. Then the generated head movements by deepfake are compared to an exact 3D simulation that can be used as ground-truth. Several measures, such as SSIM and average facial keypoint distance, are used to quantify results.

Event cameras generate data based on the amount of motion present in the captured scene, making them attractive sensors for solving object tracking tasks. In this paper, we present a framework for tracking humans using a single event camera which consists of three components. First, we train a Graph Neural Network (GNN) to recognize a person within the stream of events. Batches of events are represented as spatio-temporal graphs in order to preserve the sparse nature of events and retain their high temporal resolution. Subsequently, the person is localized in a weakly-supervised manner by adopting the well established method of Class Activation Maps (CAM) for our graph-based classification model. Our approach does not require the ground truth position of humans during training. Finally, a Kalman filter is deployed for tracking, which uses the predicted bounding box surrounding the human as measurement. We demonstrate that our approach achieves robust tracking results on test sequences from the Gait3 database, paving the way for further privacy-preserving methods in event-based human tracking. Code, pre-trained models and datasets of our research are publicly available1.

Recognition of the social network of origin of an image is a relatively recent topic that is part of the techniques that fall under the umbrella of digital image forensics. It consists of the classification of images according to the social network on which they were posted. In contrast with other topics of digital image forensics, there are no works addressing counter forensic for source social network identification. Thus, we analyse the impact of image manipulations on its performances. We focus our study on AI-based compression, which tends to become the new compression solution with the upcoming standard JPEG AI. To conduct a fair analysis, we compare the AI-based compression with the conventional legacy JPEG compression, and also include three other manipulations: median filtering, Gaussian blurring, and additional white Gaussian noise, which are often used to assess the robustness of digital image forensic methods. We define two sets of parameters based on the resulting image quality in terms of structural similarity, which correspond respectively to attacks with strong and limited image degradation. In the context of strong downgrade of the image quality, all the manipulations lead to similar decrease in performance, while for attacks that preserve image quality, AI-based compression is able to reach a drop in identification rate twice higher than the other manipulations.

The progress achieved in deepfake technology has been remarkable; however, evaluating the resulting videos and comparing different generators remains challenging. A primary concern arises from the lack of ground-truth data, except for self-reenactment scenarios. Additionally, available datasets may have inherent limitations, such as lacking expected animations or demonstrating inadequate subject diversity. Furthermore, there are ethical and privacy concerns when using real individuals' faces in such applications. This paper goes beyond the state-of-the-art dealing with the evaluation of deepfake generators by introducing an innovative dataset featuring MetaHumans. Our dataset ensures the availability of ground-truth data and encompasses diverse facial expressions, variations in pose and illumination conditions, and combinations of these factors. Additionally, we meticulously control and verify the expected animations within the dataset. The proposed dataset enables accurate evaluation of cross-reenactment generated images. By utilizing various established metrics, we demonstrate a high degree of correlation between the generator's scores obtained from deep-fake videos of Metahumans and those obtained from deepfake videos of real persons. The synthesized MetaHuman dataset can be accessed at: https://github.com/SaharHusseini/MMSP_2023

The detection of forged images is an important topic in digital image forensics. There are two main types of forgery: copy-move and splicing. These forgeries are created with image editors that apply JPEG compression by default, when saving the forged images. As a result, the authentic and falsified areas have different compression statistics, including histograms of DCT coefficients that show inconsistencies in the case of double JPEG compression. Therefore, the detection of double JPEG compression (DJPEG-C) is an important topic for JPEG-related image forensic detectors. Since the emergence of deep learning in image processing, AI-based compression methods have been proposed. This paper is the first to consider AI-based compression with digital image analysis tools. The objective is to understand whether AI-based compression can be a new unintended counter-attack for JPEG-related image forensic detectors. To verify our hypothesis, we selected the best detector to date, an AI-based compression method and the Casia v2 database that contains both splicing and copy-move (all publicly available). We focused our experiment on benign post-processing operations: AI-based and JPEG recompressions (with different quality levels). The evaluation is performed using different metrics (average precision, F1 score and accuracy, PSNR, SSIM) to take into account both the impact on detection and image quality. At similar image quality, AI-based recompression achieves a decrease in performance at least twice higher than JPEG, while preserving high visual image quality. Thus, AI-based compression is a new unintended counter-attack, which can no longer be ignored in future studies on image forensic detectors.

International audience

The identification of the source social network from an image is a relatively new research area in the image forensic domain. The classification of the source social network can be a crucial element for the growing number of cases of social-media related crimes, such as cyberbullying. This paper takes into consideration the state-of-the-art approaches addressing this problem and proposes a new methodology to improve the results obtained to date. Our identification technique is based on the idea that social networks perform some processing on the uploaded images, such as resizing or recompression, and leave some artifacts on them. We propose to use discrete cosine transform features and image noise residual analysis to detect such artifacts. A two-stream convolutional neural network, which combines the inputs from these two artifact domains, is trained to classify the source social network of images coming from three different datasets. This paper explores the two domains, proposes strategies for managing unbalanced datasets, provides details about the proposed two-stream convolutional neural network, and presents the results achieved by our method compared with the current state-of-the-art approaches.

Thermal imaging has substantially evolved, during the recent years, to be established as a complement, or even occasionally as an alternative to conventional visible light imaging, particularly for face analysis applications. Facial landmark detection is a crucial prerequisite for facial image processing. Given the upswing of deep learning based approaches, the performance of facial landmark detection has been significantly improved. However, this uprise is merely limited to visible spectrum based face analysis tasks, as there are only few research works on facial landmark detection in thermal spectrum. This limitation is mainly due to the lack of available thermal face databases provided with full facial landmark annotations. In this paper, we propose to tackle this data shortage by converting existing face databases, designed for facial landmark detection task, from visible to thermal spectrum that will share the same provided facial landmark annotations. Using the synthesized thermal databases along with the facial landmark annotations, two different models are trained using active appearance models and deep alignment network. Evaluating the models trained on synthesized thermal data on real thermal data, we obtained facial landmark detection accuracy of 94.59% when tested on low quality thermal data and 95.63% when tested on high quality thermal data with a detection threshold of 0.15×IOD.

Cross-spectrum face recognition, e.g. visible to thermal matching, remains a challenging task due to the large variation originated from different domains. This paper proposed a subspace projection hashing (SPH) to enable the cross-spectrum face recognition task. The intrinsic idea behind SPH is to project the features from different domains onto a common subspace, where matching the faces from different domains can be accomplished. Notably, we proposed a new loss function that can (i) preserve both inter-domain and intra-domain similarity; (ii) regularize a scaled-up pairwise distance between hashed codes, to optimize projection matrix. Three datasets, Wiki, EURECOM VIS-TH paired face and TDFace are adopted to evaluate the proposed SPH. The experimental results indicate that the proposed SPH outperforms the original linear subspace ranking hashing (LSRH) in the benchmark dataset (Wiki) and demonstrates a reasonably good performance for visible-thermal, visible-near-infrared face recognition, therefore suggests the feasibility and effectiveness of the proposed SPH.

Most existing works regarding facial demographic estimation are focused on still image datasets, although nowadays the need to analyze video content in real applications is increasing. We propose to tackle gender, age and ethnicity estimation in the context of video scenarios. Our main contribution is to use an attribute-specific quality assessment procedure to select most relevant frames from a video sequence for each of the three demographic modalities. Selected frames are classified with fine-tuned MobileNet models and a final video prediction is obtained with a majority voting strategy. Our validation on three different datasets and our comparison with state-of-the-art models, show the effectiveness of the proposed demographic classifiers and the quality pipeline, which allows to reduce both: the number of frames to be classified and the processing time in practical applications; and improves the soft biometrics prediction accuracy.

Access to technologies like mobile phones contributes to the significant increase in the volume of digital visual data (images and videos). In addition, photo editing software is becoming increasingly powerful and easy to use. In some cases, these tools can be utilized to produce forgeries with the objective to change the semantic meaning of a photo or a video (e.g. fake news). Digital image forensics (DIF) includes two main objectives: the detection (and localization) of forgery and the identification of the origin of the acquisition (i.e. sensor identification). Since 2005, many classical methods for DIF have been designed, implemented and tested on several databases. Meantime, innovative approaches based on deep learning have emerged in other fields and have surpassed traditional techniques. In the context of DIF, deep learning methods mainly use convolutional neural networks (CNN) associated with significant preprocessing modules. This is an active domain and two possible ways to operate preprocessing have been studied: prior to the network or incorporated into it. None of the various studies on the digital image forensics provide a comprehensive overview of the preprocessing techniques used with deep learning methods. Therefore, the core objective of this article is to review the preprocessing modules associated with CNN models.

Crowd density analysis is a crucial component in video surveillance mainly for security monitoring. This paper proposes a novel approach for crowd density classification, in which learned features substitute the commonly used handcrafted features. In particular, the approach consists of employing deep networks to extract useful crowd features that can further be manageable by a classifier. This process is favorable for crowd features extraction due to the large learning capability of deep networks compared to traditional methods based on handcrafted features. The proposed approach is evaluated on three challenging datasets, and the results demonstrate the effectiveness of learned features for crowd density classification. Furthermore, we include an extensive comparative study between different learned/hand-crafted features in order to investigate their discriminative power to handle such problems. Their performance is evaluated using different classifiers and strategies as well.

High dynamic range (HDR) imaging is able to capture and display significantly more colors when compared to the legacy imaging technology. Recently, HDR environment is gaining popularity and becoming common in our daily life, which resulted in the growing number of HDR images. In this work, a technique is put forward to first perceptually mask a HDR image, and then hide data into the masked HDR image. Specifically, pixels in the OpenEXR file, which are stored in half floating point precision format, are manipulated. First, a predictor is utilized to predict pixel values, where well predicted pixel locations are flagged and utilized as the venues to hide data. On the other hand, the ill predicted pixel locations are flagged as unusable. Next, each pixel is divided into segments of 5 bits, and the segments are XOR-ed and permuted to mask the perceptual semantic of the HDR image. Data hiding then takes place at locations flagged as usable, which further distorts the quality of the image. The proposed method is both reversible and separable. The basic performance of the proposed joint technique is evaluated by using 6 HDR images. In the best case scenario, 91% of pixels in the image can be utilized for data hiding purpose while the perceptual semantic of the original image is completely masked.

In this paper, we propose a secure visual-thermal fused face recognition system using non-linear hashing. To extract features from both thermal and visible facial images, a deep neural network model pre-trained by visible images, namely InsightFace, is utilized in extracting deep features from both thermal and visible images. Next, we investigate into the effectiveness of using nonlinear hashing in protecting deep features extracted from both thermal and visible face images. To further boost the accuracy performance of the facial recognition system under unfavorable environment, feature- and score-level fusion of thermal and visible images for face matching are studied. The performance of different application scenarios are tested on the EURECOM VIS-TH face dataset. Experiment results suggest that: 1) feature- and score-level fusion techniques are effective in achieving higher accuracy under unfavorable situation; 2) non-linear hashing offers additional layer of protection, namely, privacy preservation, to face image. We also found that the deep model trained by using visible images is applicable to thermal images for feature extraction, which is particularly useful because there is no large thermal dataset available to train deep neural network.

Light field cameras are becoming increasingly popular thanks to higher capabilities with respect to regular cameras in capturing information of a scene. Even though the principle associated with structured light sensors is quite different from the technology behind light field cameras, data provided by these technologies are similar in terms of depth map. With the aim of comparing the potential of Kinect and Lytro sensors on face recognition, two experiments are conducted on separate but publically available datasets and validated on a database acquired simultaneously with Lytro Illum camera and Kinect V1 sensor. The results obtained on RGB and depth maps are integrated with an experiment based on fusion at score level. The introduction of depth information in the RGB data is found more effective than standard bi dimensional imaging, especially in case of occlusions.

The paper explores how light fields captured by plenoptic cameras can increase the performance of face landmark detection. The idea is to exploit light fields geometrical constraints to correct the position of points detected by classical face landmark detectors. These geometric constraints are used to enforce landmark points angular coherency across the different views of the light field, and by doing so to correct the positions of the landmarks on all views. The corrected landmark points are compared with ground-truth manual annotations of a set of 400 images corresponding to the central views of 400 light fields of faces with different pose and expression.

Here, the authors propose a scalable scrambling algorithm operating in the discrete cosine transform (DCT) domain within the JPEG codec. The goal is to ensure that people are no more identifiable while keeping their actions still understandable regardless of the image size. For each 8 × 8 block, the authors encrypt the DCT coefficients to protect data information, and shift them towards the high frequencies to make the DC position available. Whereas encrypted coefficients appear as noise in the protected image, the DC position is dedicated to restitute some of the original information (e.g. the average colour associated with one or a group of blocks). The proposed approach automatically sets the value of each DC according to the region of interest size in order to keep the level of privacy protection strong enough. Comparing to existing methods, the proposed privacy protection framework provides flexibility concerning the appearance of the protected version which makes it stronger for protecting the privacy even during potential attacks. Moreover, the method does not cause excessive perturbation for the recognition of the actions and slightly decreases the efficiency of the JPEG standard.

Despite the tremendous progress in face verification performance as a result of Deep Learning, the sensitivity to human age variations remains an Achilles' heel of the majority of the contemporary face verification software. A promising solution to this problem consists in synthetic aging/rejuvenation of the input face images to some predefined age categories prior to face verification. We recently proposed [3] Age-cGAN aging/rejuvenation method based on generative adversarial neural networks allowing to synthesize more plausible and realistic faces than alternative non-generative methods. However, in this work, we show that Age-cGAN cannot be directly used for improving face verification due to its slightly imperfect preservation of the original identities in aged/rejuvenated faces. We therefore propose Local Manifold Adaptation (LMA) approach which resolves the stated issue of Age-cGAN resulting in the novel Age-cGAN+LMA aging/rejuvenation method. Based on Age-cGAN+LMA, we design an age normalization algorithm which boosts the accuracy of an off-the-shelf face verification software in the cross-age evaluation scenario.

It has been recently shown that Generative Adversarial Networks (GANs) can produce synthetic images of exceptional visual fidelity. In this work, we propose the first GAN-based method for automatic face aging. Contrary to previous works employing GANs for altering of facial attributes, we make a particular emphasize on preserving the original person's identity in the aged version of his/her face. To this end, we introduce a novel approach for “Identity-Preserving” optimization of GAN's latent vectors. The objective evaluation of the resulting aged and rejuvenated face images by the state-of-the-art face recognition and age estimation solutions demonstrate the high potential of the proposed method.

A novel approach for fast iris recognition on mobile devices is presented in this paper. Its key features are: (i) the use of a combination of classifiers exploiting the iris colour and texture information; (ii) its limited computational time, particularly suitable for fast identity checking on mobile devices; (iii) the high parallelism of the code, making this approach also appropriate for identity verification on large database. The proposed method has been submitted to the Mobile Iris CHallenge Evaluation II. The test set employed for the contest evaluation is made available on the contest web page. The latter has been used to assess the performance of the proposed method in terms of Recognition Rate (RR) and Area Under Receiver Operating Characteristic Curve (AUC).

Gender estimation has received increased attention due to its use in a number of pertinent security and commercial applications. Automated gender estimation algorithms are mainly based on extracting representative features from face images. In this work we study gender estimation based on information deduced jointly from face and body, extracted from single-shot images. The approach addresses challenging settings such as low-resolution-images, as well as settings when faces are occluded. Specifically the face-based features include local binary patterns (LBP) and scale-invariant feature transform (SIFT) features, projected into a PCA space. The features of the novel body-based algorithm proposed in this work include continuous shape information extracted from body silhouettes and texture information retained by HOG descriptors. Support Vector Machines (SVMs) are used for classification for body and face features. We conduct experiments on images extracted from video-sequences of the Multi-Biometric Tunnel database, emphasizing on three distance-settings: close, medium and far, ranging from full body exposure (far setting) to head and shoulders exposure (close setting). The experiments suggest that while face-based gender estimation performs best in the close-distance-setting, body-based gender estimation performs best when a large part of the body is visible. Finally we present two score-level-fusion schemes of face and body-based features, outperforming the two individual modalities in most cases.

This paper addresses the problem of image features selection for pedestrian gender recognition. Hand-crafted features (such as HOG) are compared with learned features which are obtained by training convolutional neural networks. The comparison is performed on the recently created collection of versatile pedestrian datasets which allows us to evaluate the impact of dataset properties on the performance of features. The study shows that hand-crafted and learned features perform equally well on small-sized homogeneous datasets. However, learned features significantly outperform hand-crafted ones in the case of heterogeneous and unfamiliar (unseen) datasets. Our best model which is based on learned features obtains 79% average recognition rate on completely unseen datasets. We also show that a relatively small convolutional neural network is able to produce competitive features even with little training data.

In this letter, a novel reversible data hiding (RDH) algorithm is proposed for digital images. Instead of trying to keep the PSNR value high, the proposed algorithm enhances the contrast of a host image to improve its visual quality. The highest two bins in the histogram are selected for data embedding so that histogram equalization can be performed by repeating the process. The side information is embedded along with the message bits into the host image so that the original image is completely recoverable. The proposed algorithm was implemented on two sets of images to demonstrate its efficiency. To our best knowledge, it is the first algorithm that achieves image contrast enhancement by RDH. Furthermore, the evaluation results show that the visual quality can be preserved after a considerable amount of message bits have been embedded into the contrast-enhanced images, even better than three specific MATLAB functions used for image contrast enhancement.

The pervasive adoption of video surveillance systems demands tools for protecting the privacy of the persons being monitored. Current solutions are either naïve or they lack of important characteristics, such as reversibility or visual quality preservation. In this paper, we propose a novel scrambling procedure for protecting privacy sensitive image regions, which encodes the sensitive data in a parametric form, exploiting the visual information in the remaining part of the image. The encoded data is encrypted with a secret key. Partial knowledge of encryption key gives a protected version of the original image at variable levels of scrambling, while the knowledge of the full key allows decryption to a quality level suitable for people identification. To evaluate the proposed approach, we apply our scrambling filter to the AT&T face recognition dataset and we measure the resulting quality with an objective metric.

The study of crowd behavior in public areas or during some public events is receiving a lot of attention in security community to detect potential risk and to prevent overcrowd. In this paper, we propose a novel approach for change detection and event recognition in human crowds. It consists of modeling time-varying dynamics of the crowd using local features. It also involves a feature tracking step which allows excluding feature points on the background and extracting long-term trajectories. This process is favourable for the later crowd event detection and recognition since the influence of features irrelevant to the underlying crowd is removed and the tracked features undergo an implicit temporal filtering. These feature tracks are further employed to extract regular motion patterns such as speed and flow direction. In addition, they are also used as an observation of a probabilistic crowd function to generate fully automatic crowd density maps. Finally, the variation of these attributes (local density, speed, and flow direction) in time is employed to determine the ongoing crowd behavior. The experimental results on two different crowd datasets demonstrate the effectiveness of our proposed approach for early detection of crowd change and accurate results for event recognition.

One of the most critical sources of variation in face recognition is facial expressions, especially in the frequent case where only a single sample per person is available for enrollment. Methods that improve the accuracy in the presence of such variations are still required for a reliable authentication system. In this paper, we address this problem with an analysis-by-synthesis-based scheme, in which a number of synthetic face images with different expressions are produced. For this purpose, an animatable 3D model is generated for each user based on 17 automatically located landmark points. The contribution of these additional images in terms of the recognition performance is evaluated with three different techniques (principal component analysis, linear discriminant analysis, and local binary patterns) on face recognition grand challenge and Bosphorus 3D face databases. Significant improvements are achieved in face recognition accuracies, for each database and algorithm.

The recent success of emerging RGB-D cameras such as the Kinect sensor depicts a broad prospect of 3-D data-based computer applications. However, due to the lack of a standard testing database, it is difficult to evaluate how the face recognition technology can benefit from this up-to-date imaging sensor. In order to establish the connection between the Kinect and face recognition research, in this paper, we present the first publicly available face database (i.e., KinectFaceDB1) based on the Kinect sensor. The database consists of different data modalities (well-aligned and processed 2-D, 2.5-D, 3-D, and video-based face data) and multiple facial variations. We conducted benchmark evaluations on the proposed database using standard face recognition methods, and demonstrated the gain in performance when integrating the depth data with the RGB data via score-level fusion. We also compared the 3-D images of Kinect (from the KinectFaceDB) with the traditional high-quality 3-D scans (from the FRGC database) in the context of face biometrics, which reveals the imperative needs of the proposed database for face recognition research.

In this paper we present a method of improving a human detector by means of crowd density information. Human detection is especially challenging in crowded scenes which makes it important to introduce additional knowledge into the detection process. We compute crowd density maps in order to estimate the spatial distribution of people in the scene and show how it is possible to enhance the detection results of a state-of-the-art human detector by this information. The proposed method applies a self-adaptive, dynamic parametrization and as an additional contribution uses scene-adaptive learning of the human aspect ratio in order to reduce false positive detections in crowded areas. We evaluate our method on videos from different datasets and demonstrate how our system achieves better results than the baseline algorithm.

There are several types of spoofing attacks to face recognition systems such as photograph, video or mask attacks. To the best of our knowledge, the impact of mask spoofing on face recognition has not been analyzed yet. The reason for this delay is mainly due to the unavailability of public mask attacks databases. In this study, we use a 2D+3D mask database which was prepared for a research project in which the authors are all involved. This paper provides new results by demonstrating the impact of mask attacks on 2D, 2.5D and 3D face recognition systems. The results show that face recognition systems are vulnerable to mask attacks, thus countermeasures have to be developed to reduce the impact of mask attacks on face recognition. The results also show that 2D texture analysis provides more information than 3D face shape analysis in order to develop a countermeasure against high-quality mask attacks.

Semi-supervised learning is attracting growing interest within the biometrics community. Almost all prior work focuses on closed-set scenarios, in which samples labelled automatically are assumed to belong to an enrolled class. This is often not the case in realistic applications and thus open-set alternatives are needed. This paper proposes a new approach to open-set, semi-supervised learning based on co-training, Linear Discriminant Analysis (LDA) subspaces and Sparse Representation Classifiers (SRCs). Experiments on the standard MOBIO dataset show how the new approach can utilize automatically labelled data to augment a smaller, manually labelled dataset and thus improve the performance of an open-set audio-visual person recognition system.

Crowd density analysis is a crucial component in visual surveillance for security monitoring. In this paper, we propose to estimate crowd density at patch level, where the size of each patch varies in such way to compensate the effects of perspective distortions. The main contribution of this paper is two-fold: First, we propose to learn a discriminant subspace of the high-dimensional Local Binary Pattern (LBP) instead of using raw LBP feature vector. Second, an alternative algorithm for multiclass SVM based on relevance scores is proposed. The effectiveness of the proposed approach is evaluated on PETS dataset, and the results demonstrate the effect of low-dimensional compact representation of LBP on the classification accuracy. Also, the performance of the proposed multiclass SVM algorithm is compared to other frequently used algorithms for multi-classification problem and the proposed algorithm gives good results while reducing the complexity of the classification.

The widespread growth in the adoption of digital video surveillance systems emphasizes the need for privacy preservation video analytics techniques. While these privacy aspects have shown big interest in recent years, little importance has been given to the concept of context-aware privacy protection filters. In this paper, we specifically focus on the dependency between privacy preservation and crowd density. We show that additional information about the crowd density in the scene can be used in order to adjust the level of privacy protection according to the local needs. This additional information cue consists of modeling time-varying dynamics of the crowd density using local features as an observation of a probabilistic crowd function. It also involves a feature tracking step which enables excluding feature points on the background. This process is favourable for the later density function estimation since the influence of features irrelevant to the underlying crowd density is removed. Then, the protection level of personal privacy in videos is adapted according to the crowd density. Afterwards, a framework for objective evaluation of the contextualized protection filters is proposed. The effectiveness of the proposed context-aware privacy filters has been demonstrated by assessing the intelligibility vs. privacy trade-off using videos from different crowd datasets.

Facial cosmetics, also called makeup, may change the appearance of a face that we could perceive. In order to contribute to studies in image processing related to facial cosmetics, a database is built which contains multiple images per person with and without applied cosmetics. Annotations provide detailed information about the amount and location of applied makeup for each picture. Furthermore, a classification approach is presented. The classification approach takes the altering effect of the applied cosmetics and the application area into account. Since automatic face recognition evolved to an important topic over the last decades and is affected by facial cosmetics, preliminary tests are done to evaluate their impact on automatic face recognition. The face as a whole as well as its most significant makeup application areas that are skin, eyes and mouth are investigated separately.

Crowd density analysis is a crucial component in visual surveillance mainly for security monitoring. This paper proposes a novel approach for crowd density measure, in which local information at pixel level substitutes a global crowd level or a number of people per-frame. The proposed approach consists of generating fully automatic and crowd density maps using local features as an observation of a probabilistic crowd function. It also involves a feature tracking step which allows excluding feature points belonging to the background. This process is favorable for the later density function estimation since the influence of features irrelevant to the underlying crowd density is removed. Our proposed approach is evaluated on videos from different datasets, and the results demonstrate the effectiveness of feature tracks for crowd estimation. Furthermore, we include a comparative study between different local features in order to investigate their discriminative power to the crowd.

3D shape data for face recognition is advantageous to its 2D counterpart for being invariant to illumination and pose. However, expression variations and occlusions still remain as major challenges since the shape distortions hinder accurate matching. Numerous algorithms developed to overcome this problem mainly propose region-based approaches, where similarity scores are calculated separately by local regional matchers and fused for recognition. In this paper, we present a regional confidence score assessment scheme that estimates the expression or occlusion induced distortions in different facial regions. Thereby, reliability scores are obtained which can be used in fusion step for recognition. For 7 regions of face, primitive shape distributions are extracted and the surface quality is measured automatically by an Iterative Closest Point (ICP) based method. Using these measurements, an Artificial Neural Network (ANN) is trained and utilized to estimate regional reliability scores. Experiments have been conducted on FRGC v2 3D face database and results demonstrate a high accuracy in surface quality estimation.

Information fusion in biometrics mostly relates to multi-biometric systems which attempt to improve the performance of individual matchers: multi-sensor, multi-algorithm, multimodal, etc. However, in addition to these scenarios, the need for methods to fuse the individual regional classifiers has also emerged, due to the increasing number of region-based methods proposed to overcome expression and occlusion problems in face recognition. In this paper, we present a combination approach by converting the regional match scores into probabilities with the help of estimated regional confidence measures. Initially, face is broken into several segments and similarity and confidence scores are obtained. Then, the posteriori probabilities of the user being genuine are calculated in each region given these two scores. For this calculation, the conditional densities are obtained on the training samples by applying non-parametric kernel density estimation separately for different intervals of confidence levels. Experimental results demonstrate that the inclusion of the regional confidence measures via probabilistic conversion is much more advantageous when compared to weighted sum of original scores.

Facial occlusion is a critical issue in many face recognition applications. Existing approaches of face recognition under occlusion conditions mainly focus on the conventional facial accessories (such as sunglasses and scarf) and thus presume that the occluded region is dense and contiguous. Yet due to the wide variety of natural sources which can occlude a human face in uncontrolled environments, methods based on the dense assumption are not robust to thin and randomly distributed occlusions. This paper presents the solution to a newly identified facial occlusion problem - sparse occlusion in the context of face biometrics in video surveillance. We show that the occluded pixels can be detected in the low-rank structure of a canonical face set under the Robust-PCA framework; and the occluded part can be inpainted solely based on the nonoccluded part and a Fields-of-Experts prior via spatial inference. Experiments demonstrate that the proposed approach significantly improve various face recognition algorithms in presence of complex sparse occlusions.

Client models used in Automatic Speaker Recognition (ASR) and Automatic Face Recognition (AFR) are usually trained with labelled data acquired in a small number of menthol sessions. The amount of training data is rarely sufficient to reliably represent the variation which occurs later during testing. Larger quantities of client-specific training data can always be obtained, but manual collection and labelling is often cost-prohibitive. Co-training, a paradigm of semi-supervised machine learning, which can exploit unlabelled data to enhance weakly learned client models. In this paper, we propose a co-LDA algorithm which uses both labelled and unlabelled data to capture greater intersession variation and to learn discriminative subspaces in which test examples can be more accurately classified. The proposed algorithm is naturally suited to audio-visual person recognition because vocal and visual biometric features intrinsically satisfy the assumptions of feature sufficiency and independency which guarantee the effectiveness of co-training. When tested on the MOBIO database, the proposed co-training system raises a baseline identification rate from 71% to 99% while in a verification task the Equal Error Rate (EER) is reduced from 18% to about 1%. To our knowledge, this is the first successful application of co-training in audio-visual biometric systems.

Numerous major challenges in face recognition, such as pose, illumination, expression and aging, have been investigated extensively. All those variations modify the texture and/or the shape of the face in a similar manner for different individuals. However, studies on alterations applied on face via plastic surgery or prosthetic make-up which can be in countless different ways and amounts, are still very limited. In this paper, we analyze how such changes on nose region affect the face recognition performances of several key techniques. For this purpose, a simulated face database is prepared using FRGC v1.0 in which nose in each sample is replaced with another randomly chosen one. Since this is a 3D database, the impact analysis is not limited to only 2D, which is one of the novelties of this study. Performance comparisons of three 2D and four 3D algorithms are provided. In addition, differently from previous works, baseline results for the original database are also reported. Hence, the impact which is purely due to the applied nose alterations can be measured. The experimental results indicate that with the introduction of alterations both modalities lose precision, especially 3D.

Extensive adoption of video surveillance, affecting many aspects of the daily life, alarms the concerned public about the increasing invasion into personal privacy. Therefore, to address privacy issues, many tools have been proposed for protection of personal privacy in image and video. However, little is understood regarding the effectiveness of such tools and especially their impact on the underlying surveillance tasks. In this paper, we propose a subjective evaluation methodology to analyze the tradeoff between the preservation of privacy offered by these tools and the intelligibility of activities under video surveillance. As an example, the proposed method is used to compare several commonly employed privacy protection techniques, such as blurring, pixelization, and masking applied to indoor surveillance video. The results show that, for the test material under analysis, the pixelization filter provides the best performance in terms of balance between privacy protection and intelligibility.

Despite recent advances, face-recognition algorithms are still challenged when applied in the setting of video surveillance systems which inherently introduce variations in the pose of subjects. The present work addresses this problem, and seeks to provide a recognition algorithm that is specifically suited for a frontal-to-side re-identification setting. Deviating from classical biometric approaches, the proposed method considers color- and texture- based soft biometric traits, specifically those taken from patches of hair, skin and clothes. The proposed method and the suitability of these patch-based traits are then validated both analytically and empirically.

Facial occlusions, due for example to sunglasses, hats, scarf, beards etc., can significantly affect the performance of any face recognition system. Unfortunately, the presence of facial occlusions is quite common in real-world applications especially when the individuals are not cooperative with the system such as in video surveillance scenarios. While there has been an enormous amount of research on face recognition under pose/illumination changes and image degradations, problems caused by occlusions are mostly overlooked. The focus of this paper is thus on facial occlusions, and particularly on how to improve the recognition of faces occluded by sunglasses and scarf. We propose an efficient approach which consists of first detecting the presence of scarf/sunglasses and then processing the non-occluded facial regions only. The occlusion detection problem is approached using Gabor wavelets, PCA and support vector machines (SVM), while the recognition of the non-occluded facial part is performed using block-based local binary patterns. Experiments on AR face database showed that the proposed method yields significant performance improvements compared to existing works for recognizing partially occluded and also non-occluded faces. Furthermore, the performance of the proposed approach is also assessed under illumination and extreme facial expression changes, demonstrating interesting results.

Face recognition algorithms based on linear discriminant analysis (LDA) generally give satisfactory performance but tend to require a relatively high number of samples in order to learn reliable projections. In many practical applications of face recognition there is only a small number of labelled face images and in this case LDA-based algorithms generally lead to poor performance. The contributions in this paper relate to a new semi-supervised, self-training LDA-based algorithm which is used to augment a manually labelled training set with new data from an unlabelled, auxiliary set and hence to improve recognition performance. Without the cost of manual labelling such auxiliary data is often easily acquired but is not normally useful for learning. We report face recognition experiments on 3 independent databases which demonstrate a constant improvement of our baseline, supervised LDA system. The performance of our algorithm is also shown to significantly outperform other semi-supervised learning algorithms.

Recently, local binary patterns (LBP) based descriptors and sparse representation based classification (SRC) become both eminent techniques in face recognition. Preliminary techniques of combining LBP and SRC have been proposed in the literature. However, the state-of-art method suffers from the “curse of dimensionality” for real world scenarios. In this paper, a novel face recognition algorithm of combining LBP with SRC is proposed; in which the dimensionality problem is resolved by divide-and-conquer and the discriminative power is strengthen via its pyramid architecture. The proposed face recognition method is evaluated on AR Face Database and yields very impressive results.

While there has been an enormous amount of research on face recognition under pose/illumination changes and image degradations, problems caused by occlusions are mostly overlooked. Moreover, most of the existing approaches of face recognition under occlusion conditions focus on overcoming facial occlusion problems due to sunglasses and scarf. To the best of our knowledge, occlusion due to cap has never been studied in the literature, but the importance of this problem should be emphasized since it is known that bank robbers and football hooligans take advantage of it for hiding their faces. This paper presents a solution to this newly identified face occlusion problem -- the time-variant occlusion due to cap in entrance surveillance, in the context of face biometrics in video surveillance. The proposed approach consists of two parts: detection and tracking of occluded faces in complex surveillance videos; detecting the presence of cap by exploiting temporal information. The detection and tracking part is based upon body silhouette and elliptical head tracker. The classification of cap/non-cap faces utilizes dynamic time warping (DTW) and agglomerative hierarchical clustering. The proposed algorithm is evaluated on several surveillance videos and yields good detection rates.

In this technical demonstration, we show the application of our research on body soft biometrics. Exploiting a 3D video sensor we are able to extract semantic information that describes subjects standing in front of a camera. Semantic analysis and tracking is performed, a series of anthropometric measures are extracted and used to compute subjects' height, weight, and gender information. Possible applications of such research fall in the medical domain for monitoring elderly people; in the gaming industry for automatic avatar creation, or in smart billboards which collects demographics of the public interested by the commercial. Our algorithm allows the estimation of all these parameters in real time without requiring the computational complexity of a 3D model fitting approach.

This work studies eye color as a soft biometric trait and provides a novel insight about the influence of pertinent factors in this context, like color spaces, illumination and presence of glasses. A motivation for the paper is the fact that the human iris color is an essential facial trait for Caucasians, which can be employed in iris pattern recognition systems for pruning the search or in soft biometrics systems for person re-identification. Towards studying iris color as a soft biometric trait, we consider a system for automatic detection of eye color, based on standard facial images. The system entails automatic iris localization, followed by classification based on Gaussian Mixture Models with Expectation Maximization. We finally provide related detection results on the UBIRIS2 database employable in a real time eye color detection system.

Digital visual media represent nowadays one of the principal means for communication. Lately, the reliability of digital visual information has been questioned, due to the ease in counterfeiting both its origin and content. Digital image forensics is a brand new research field which aims at validating the authenticity of images by recovering information about their history. Two main problems are addressed: the identification of the imaging device that captured the image, and the detection of traces of forgeries. Nowadays, thanks to the promising results attained by early studies and to the always growing number of applications, digital image forensics represents an appealing investigation domain for many researchers. This survey is designed for scholars and IT professionals approaching this field, reviewing existing tools and providing a view on the past, the present and the future of digital image forensics.

Biometric systems based on human traits like fingerprinting, face, iris, etc., even widely explored in literature, cannot be used in scenarios like video surveillance of crowded envinroments. A different class of biometry, usually referred as soft biometry, including individual's height, weight, skin color, clothes color, trajectory, etc., can be more easily extracted from cameras in a large network to provide some useful information about the users. In this work we focus our attention on the color of human clothes. We apply color based soft biometric to a specific scenario, i.e. to prevent hooligans fights, based on the color of the clothes they wear. A video surveillance system able to quickly identify the presence of rival teams fans in a specific location would be helpful in avoiding vandalism and destructive behaviour.

In this paper, a demonstrator called BIOFACE incorporating several facial biometric techniques is described. It includes the well established Eigenfaces and the recently published Tomofaces techniques, which perform face recognition based on facial appearance and dynamics, respectively. Both techniques are based on the space dimensionality reduction and the enrollment requires the projection of several positive face samples to the reduced space. Alternatively, BIOFACE also performs face recognition based on the matching of Scale Invariant Feature Transform (SIFT) features.

This work introduces the novel idea of using a bag of facial soft biometrics for person verification and identification. The novel tool inherits the non-intrusiveness and computational efficiency of soft biometrics, which allow for fast and enrolment-free biometric analysis, even in the absence of consent and cooperation of the surveillance subject. In conjunction with the proposed system design and detection algorithms, we also proceed to shed some light on the statistical properties of different parameters that are pertinent to the proposed system, as well as provide insight on general design aspects in soft-biometric systems, and different aspects regarding efficient resource allocation.

This paper presents a graph based scheme for color text recognition in images and videos, which is particularly robust to complex background, low resolution or video coding artifacts. This scheme is based on a novel method named the image text recognition graph (iTRG) composed of five main modules: an image text segmentation module, a graph connection builder module, a character recognition module, a graph weight calculator module and an optimal path search module. The first two modules are based on convolutional neural networks so that the proposed system automatically learns how to robustly perform segmentation and recognition. The proposed method is evaluated on the public ICDAR 2003 test word dataset.

Current object detection systems reach high detection rates, at the expense of requiring a large training database. This paper presents a new method for object detection, that gives state-of-the-art results, while using a reduced training database. The proposed system relies on a new local feature extraction approach inspired by Convolutional Neural Networks, Principal Component Analysis and Multilayer Perceptrons. We show that the proposed scheme improves robustness and generalization on the specific problem of face detection, with a very reduced set of exemplar face images.

Biometric systems have gained the attention of both the research community and the industry becoming an important topic in real application scenarios. Face recognition is, with fingerprint, among the most used techniques since it is natural for humans to recognize people from facial appearance, since the technology is mature, and because, unlike fingerprint, it is completely unintrusive. Existing systems only focus on the appearance of the subjects considering facial expressions as an obstacle to their aim. On the other hand such systems presents several limitations when dealing with variable illumination conditions, head pose, day-to-day variations (e.g. beard, glasses, or make-up), etc. Furthermore, most of the current techniques do not exploit dynamics to detect the liveness of the tested subjects. In this paper we present a study on person recognition from the dynamics of the facial feature points. The aim of this work is to demonstrate that dynamics of facial expressions could be seen as a biometric characteristic. Therefore, only dynamic characteristics are considered and the adopted features are purged of all appearance information. The results clearly show that relevant biometric information can be extracted from facial expressions and other dynamics of the face.

In this article we propose a novel spatio-temporal approach for person recognition using video information. By applying discrete video tomography, our algorithm summarises the head and facial dynamics of a sequence into a single image (called "video X-ray image"), which is subsequently analysed by an extended version of the eigenface approach. In the experimental part, we assess the discriminative power of our system and we compare it with an analogous one working on traditional facial appearance. Finally, we integrate the X-ray information with appearance in a multimodal system, which improves the recognition rates of standalone frameworks.

In this article we present a novel multimodal gender recognition system, which successfully integrates the head and mouth motion information with facial appearance by taking advantage of a unified probabilistic framework. In fact, we develop a temporal subsystem that has an extended feature space consisting of parameters related to head and mouth motion; at the same time, we introduce a complementary spatial subsystem based on a probabilistic extension of the eigenface approach. In the end, we implement an integration step to combine the similarity scores of the two parallel subsystems, using a suitable opinion fusion (or score fusion) strategy. The experiments show that not only facial appearance but also head and mouth motion possess a potentially relevant discriminatory power, and that the integration of different sources of biometric information from video sequences is the key strategy to develop more accurate and reliable recognition systems.

In this paper, a reversible watermarking algorithm is proposed for 3D mesh models based on prediction-error expansion. Firstly, we predict a vertex position by calculating the centroid of its traversed neighbors. Then the prediction error, i.e. the difference between the predicted and real positions, is expanded for data embedding. So only the vertex coordinates are modified to embed a watermark into the mesh content without changing the topology. We further reduce the distortion by adaptively choosing a threshold so that the prediction errors with too large magnitude will not be expanded. The chosen threshold value and critical location information should be saved in the watermarked mesh to guide the recovery process. The experiments show that the original mesh can be exactly recovered and consequently our algorithm can be used for symmetric or public key authentication of 3D mesh models.

Video based facial recognition is an appealing modality in biometrics due to its acceptability and ease of use but the associated recognition rate is not high enough due to multiple sources of variation and lack of constraints in real world applications. In this article we investigate the possible contribution of facial mimics extracted from low quality videos for person recognition. The initial results reported tend to validate this original proposal, thus opening some new perspectives for design of future hybrid and efficient system combining facial appearance and dynamics.

In this article we present a multimodal system to person recognition by integrating two complementary approaches that work with video data. The first module exploits the behavioural information: it is based on statistical features computed using the displacement signals of a head; the second one is dealing with the physiological information: it is a probabilistic extension of the classic Eigenface approach. For a consistent fusion, both systems share the same probabilistic classification framework: a Gaussian mixture model (GMM) approximation and a Bayesian classifier. We assess the performances of the multimodal system by implementing two fusion strategies and we analyse their evolution in presence of artificial noise.

In the last few years, a large number of 3D watermarking schemes have been proposed. We describe in this paper a possible benchmark to evaluate 3D watermarking algorithms. A list of objects and basic reproducible attacks against which 3D watermarking system could be evaluated are proposed as well as a way to compute a final score.

This paper describes a new approach for watermarking 3D objects via contour information. Unlike most conventional 3D object watermarking techniques, for which both insertion and extraction of the mark are performed on the object itself (3D/3D approach), we propose an asymmetric 3D/2D procedure. This procedure consists of watermarking 3D objects but retrieving the mark from represented views. In this paper we also propose an extension of 2D contour watermarking algorithm to 3D silhouette

In this paper, we describe a new framework for watermarking 3D objects via their contour information. Unlike most conventional 3D object watermarking techniques, for which both insertion and extraction of the mark are performed on the object itself (3D/3D approach), we propose an asymmetric 3D/2D procedure. The goal of our work is to retrieve information (originally hidden in the apparent 3D silhouette of the object) from resulting images or videos having used the synthetic object, thus protecting the visual representations of the object. After developing theoretical and practical key-points of this 3D object watermarking scheme, we present the results of some preliminary experiments

This paper describes a new approach for identity recognition using video sequences. While most image and video recognition systems discriminate identities using physical information only, our approach exploits the behavioral information of head dynamics; in particular the displacement signals of few head features directly extracted in the image plane. Due to the lack of standard video database, identification and verification scores have been obtained using a small collection of video sequences: the results for this new approach are nevertheless promising

Face is considered as an attractive biometric but because of multiple sources of variabilities, the associated recognition rate is not high enough, when working on appearance only, for most of real applications. Considering that most available visual data are videos and not still images, we investigate in this article the possible contribution of some dynamic parameters (head displacements and mouth motion) in person recognition. Some preliminary results tend to validate this original proposal that opens some new perspectives in the possible design of future hybrid and efficient system combining appearance and dynamics of faces

Geometrical distortions are the Achilles heel for many watermarking schemes. Most countermeasures proposed in the literature only address the problem of global affine transforms (e.g., rotation, scaling, and translation). In this paper, we propose an original blind watermarking algorithm robust to local geometrical distortions such as the deformations induced by Stirmark. Our method consists in adding a predefined additional information to the useful message bits at the insertion step. These additional bits are labeled as resynchronization bits or reference bits and they are modulated in the same way as the information bits. During the extraction step, the reference bits are used as anchor points to estimate and compensate for small local and global geometrical distortions. The deformations are approximated using a modified basic optical flow algorithm.

Digital watermarking has been introduced in the 1990s as a complementary technology for copyright protection. In an effort to anticipate hostile behavior of adversaries, the research community is constantly introducing new attacks to benchmark watermarking systems. In this paper, we present a generic attack strategy based on block replacement. As multimedia content is often highly repetitive, the attack exploits signal's self-similarities to replace each signal block with another, perceptually similar one. Guided by the principles of the proposed attack framework, we implemented three attack algorithms for different types of multimedia content: video shots, audio tracks and still images. Finally, considering the effectiveness of the proposed algorithms, we identify the properties that a watermark should have to counter this attacking strategy

This paper presents an original framework that analyzes the complete ocular (eye + eyebrow) expression on video sequences to reproduce it later on synthetic three-dimensional (3-D) models in real-time. We propose a two step process to develop robust techniques for facial feature analysis aimed at working without special illumination conditions or physical constraints on the user (markers, fixed frontal pose, etc.). First, simple and efficient image-processing methods based on motion models are designed over a frontal point of view of the face. Natural and realistic intra-feature and inter-feature constraints are applied to improve the analysis results. Then, the usability of these algorithms is extended to enable the analysis regardless of the person's pose in front of the camera. This is achieved by redefining the motion models involved over the speaker's highly realistic synthetic representation (clone), by using a suitable observation model, and by predicting the head pose in 3-D, frame by frame.

Security issues have almost been ignored during the first decade of digital watermarking. As a result, many released watermarking algorithms are weak against hostile intelligence. For instance, block replacement attacks defeat watermarking systems which do not consider the self-similarities of the host signal during embedding. Such attacks replace each signal block with another one, or a combination of other ones, taken at a different location. In this paper, a novel strategy will be presented to generate a signal coherent watermark to achieve immunity against block replacement attacks. The basic idea consists in imposing a linear relationship between watermark samples embedded at different locations, with respect to their local neighborhoods which are characterized with Gabor features.

The ability to verify automatically and with great accuracy the identity of a person has become crucial in everyday life. Biometrics is an emerging topic in the field of signal processing. Our research on biometrics aims at developing a complete framework useful to control access. This technical demo shows the latest image processing techniques for face detection developed at France Telecom and for face recognition developed at Eurécom. Using only one computer and one standard webcam, our biometric system detects the user face and the recognition algorithm uses this image to enable the access to a resource, a service or a location.

This paper proposes a new measure of "distance" between faces. This measure involves the estimation of the set of possible transformations between face images of the same person. The global transformation, which is assumed to be too complex for direct modeling, is approximated by a patchwork of local transformations, under a constraint imposing consistency between neighboring local transformations. The proposed system of local transformations and neighboring constraints is embedded within the probabilistic framework of a two-dimensional hidden Markov model. More specifically, we model two types of intraclass variabilities involving variations in facial expressions and illumination, respectively. The performance of the resulting method is assessed on a large data set consisting of four face databases. In particular, it is shown to outperform a leading approach to face recognition, namely, the Bayesian intra/extrapersonal classifier.

The security issue has been neglected for a long time in digital watermarking. Recent results for video watermarking have pointed out that existing watermarking schemes are not secure, i.e., a hostile intelligence succeeds in removing the hidden watermarks. In particular, for a given secret key, many watermarking schemes embed watermarks which lie in the same low-dimensional subspace whatever the host data is. We show that this subspace can be quite easily estimated with an efficient principal component analysis (PCA). For storage convenience, an online expectation-maximization (EM) algorithm is considered. Once this watermarking subspace has been estimated, an attacker only has to project incoming data onto the orthogonal of this subspace to remove the watermark.

We recently introduced a novel approximation of the intractable two-dimensional hidden Markov model (2D HMM), the turbo-HMM (T-HMM), which consists of a set of interconnected horizontal and vertical 1D HMMs. In this paper, we consider the extension of this framework to the continuous state HMM, generally referred to as the state-space model (SSM). We provide efficient approximate answers to the three following problems: (1) how to compute the likelihood of a set of observations; (2) how to find the sequence of states that best "explains" a set of observations; and (3) how to estimate the model parameters given a set of observations. The application of this work to the challenging problem of face recognition, in the presence of large illumination variations, illustrates the potential of our approach.

A recent application related to 3D watermarking has led to a specific registration problem: the registration of a 3D computer object with a computer-generated 2D view of it. So far, projective registration algorithms have focused on images of real objects because there was no interest in registering synthetic images with computer models. While those algorithms could also be directly applied to the case of synthetic images, they do not take advantage of some specificities of the synthetic/synthetic registration problem. This problem is addressed here and a dedicated registration algorithm is presented.

This paper presents an original data-hiding application where the payload is intimately related to the host data. On the other hand, we want to preserve this relationship so that each part of the payload is hidden in the related part of the host. On the other hand, we want to ensure that a degradation of the host data implies a proportional degradation of the related part of the hidden data.

While the hidden Markov model (HMM) has been extensively applied to one-dimensional problems, the complexity of its extension to two-dimensions grows exponentially with the data size and is intractable in most cases of interest. We introduce an efficient algorithm for approximate decoding of 2D HMMs, i.e., searching for the most likely state sequence. The basic idea is to approximate a 2D HMM with a turbo-HMM (T-HMM), which consists of horizontal and vertical 1D HMMs that "communicate", and allow iterated decoding (ID) of rows and columns by a modified version of the forward-backward algorithm. We derive the approach and its re-estimation equations. We then compare its performance to another algorithm designed for decoding 2D HMMs: the path constrained variable state Viterbi (PCVSV) algorithm (Li, J. et al., IEEE Trans. on Sig. Processing, vol.48, no.2, 2000). Finally, we combine our approach with PCVSV and show that the combination outperforms each algorithm taken separately.

This paper introduces a novel deformable model for face mapping and its application to automatic person identification. While most face recognition techniques directly model the face, our goal is to model the transformation between face images of the same person. As a global face transformation may be too complex to be modeled in its entirety, it is approximated by a set of local transformations with the constraint that neighboring transformations must be consistent with each other. Local transformations and neighboring constraints are embedded within the probabilistic framework of a two-dimensional hidden Markov model (2-D HMM). Experimental results on a face identification task show that the new approach compares favorably to the popular Fisherfaces algorithm.

This paper proposes a new video watermarking scheme based on a hierarchical motion analysis, that is robust against classical video processing (filtering, lossy compression...). This model works in the uncompressed domain and disturbs the motion vectors computed by an exhaustive BMA on blocks of size n*n in order to insert an invisible watermark enabling a protection against most attacks. Finally, to increase the robustness, our system inserts the mark by generating a hierarchy of motion vectors of size N*N with N=k*n to spread the mark on the lower level associated to block n*n.

Recent efforts for watermarking digital video extend the results obtained for still image watermarking. As a result, most of the proposed algorithms rely on a frame-by-frame approach. Such an adaptation leads to unreliable algorithms in terms of security. The goal of this article is to stress the problem of collusion when digital watermarked data is distributed at large scale and especially intra-video collusion in the context of video. Three simple collusion attacks are described before being evaluated on two alternative video watermarking algorithms based on the spread spectrum technique. Finally, some experimental results are presented confirming the danger of intra-video collusion and some perspectives are discussed.

We describe a novel framework for watermarking three-dimensional (3D) video objects via their texture information. Unlike classical algorithms dealing with 3D objects that operate on meshes in order to protect the object itself, the main goal of our work is to retrieve information originally hidden in the textured image of the object from resulting images or videos having used the 3D synthetic object, thus protecting the visual representations of the object. After developing the theory and practical details of this 3D object watermarking scheme, we present the results of several experiments carried out in various conditions, ranging from ideal conditions (e.g., known rendering parameters) to more realistic conditions (e.g, unknown rendering parameters, but estimated from a two-dimensional view) or within the context of attacks (e.g., mesh reduction).

Biometrics is an emerging topic in the field of signal processing. While technologies (e.g. audio, video) for biometrics have mostly been studied separately, ultimately, biometric technologies could find their strongest role as interwined and complementary pieces of a multi-modal authentication system. In this paper, a short overview of voice, fingerprint, and face authentication algorithms is provided.

During the last decade, a significant effort has been put into designing watermarking algorithms. The watermarking community now needs some advanced attacks and fair benchmarks in order to compare the performances of different watermarking technologies. Moreover, attacks permit the weaknesses of an algorithm to be found and consequently trigger further research in order to overcome the problem. These ideas motivated the creation of the European Certimark project. After a short definition of dewatermarking, we present an original attack based on self similarities. This attack is then put to the test with three different publicly available watermarking tools. Finally, we discuss briefly the feasibility of a generic attack i.e. a dewatermarking attack which should succeed in removing whatever watermarks have been inserted by whatever watermarking tools.

In the past few years, much research on watermarking has been focused on improving robustness to attacks. The starting point of the present study is an R&D algorithm which was not optimised for still images, but offered a good trade-off in terms of capacity, visibility and robustness, and worked in a full blind manner. We have focused on adapting it to video and on optimising it to the world of embedded terminals, such as digital still cameras, digital television and wireless terminals, where the computing power and storage resources of a Pentium are not available. Results of our adaptations are shown in the form of execution times versus platform capabilities. This work is the result of a close collaboration between the Eurecom research institute and STMicroelectronics.

Most face expression algorithms assume a front or 'near-to-front' head position. This assumption becomes an important limitation when studying input from real systems. We present a new approach to robustly determine face expressions independently of the head pose. Our analysis-synthesis cooperation, possible thanks to the use of a highly realistic 3D head model and the application of Kalman filtering to predict the user pose, permits us to correctly track the interesting face features. Adapting 'near-to-front' analysis techniques based on the predicted pose enables us to use such algorithms with moving speakers.

Future human-machine interaction interfaces will need a perfect understanding of the person's behavior so that machines can learn from it, react accordingly, synthetically reproduce this behavior afterwards, etc. The study of user's head action and face expression is fundamental to achieve this comprehension. This technical demo shows the latest image processing techniques for face tracking and expression analysis developed at Eurecom. Our research on face analysis aims at synthetically reproducing head movements in telecom applications. Users will be able to test themselves on how our system can track and analyze their face expressions with just one webcamera only under any unconstrained environment.

Multimedia data security is very important for multimedia commerce on the Internet such as "pay-per-view" services. Thus, watermarking algorithms for data security appear. These algorithms describe methods and technologies that allow information to be hidden, for example a number or text, into a media, such as images, video, audio files... In this paper, we propose a new application of watermarking based on a scrambling process. In this application, we develop a waterscrambling technique in which video data are scrambled efficiently by disturbing a subset of motion vectors. The interest of this approach is its ability to scramble a video while maintaining a certain visibility. Effectively it permits the use of considerable levels of security in order to choose the level of perceptibility of the video. By using watermarking techniques we will be able to combine this approach with a classical copyright watermarking system.

This article presents an efficient approach to eye movement estimation by combining color and energy based image analysis algorithms. The movement is first analyzed and then described in terms of action units. A temporal state diagram is used to control the behavior of the analysis over time so that the movements of the eye can be synthesized from the former description, after translating them into face animation parameters.

This paper applies ideas from fractal compression and optimization theory to attack the problem of efficient content-based image indexing and retrieval. Similarity of images is measured by block matching after optimal (geometric, photometric, etc.) transformation. Such block matching which, by definition, consists of localized optimization, is further governed by a global dynamic programming technique (Viterbi algorithm) that ensures continuity and coherence of the localized block matching results. Thus, the overall optimal transformation relating two images is determined by a combination of local block-transformation operations subject to a regularization constraint. Experimental results on a sample of seventy five binary images from the MPEG-7 database demonstrate the power and potential of the proposed approach.

The aim of this demonstration is to present the ongoing performance of our research and development watermarking scheme software for owner and image authentication. The proposed illustrations cover a large panel of original images (in grey levels and colors), watermarks and attacks. Evaluation is performed according to ratio, visibility and robustness.

We present a novel view-based approach to quantify and reproduce facial expressions, by systematically exploiting the degrees of freedom allowed by a realistic face model, which embeds efficient mesh morphing and texture animations to synthesize facial expressions. For this purpose, we propose to use eigenfeatures, built from synthetic images, and design a linear estimator to interpret the responses of the eigenfeatures on a facial expression in terms of animation parameters.

This paper proposes an algorithm for content-based image indexing whose formulation of similarity is borrowed from methods for fractal compression. Unlike most traditional block-based image indexing algorithms, the proposed method employs dynamic programming to exploit inter-block dependencies. A regularization constraint is globally imposed, and the overall optimal transformation relating two images is efficiently determined by application of the Viterbi algorithm. Preliminary experimental results on a sample of eighty binary images from the MPEG-7 database are presented.

The aim is to present the ongoing performance of our R and D watermarking scheme software. The proposed illustrations cover a large panel of original images (in grey levels and colors), signatures and attacks. Evaluation is performed according to ratio, visibility and robustness.

A novel paradigm for fractal coding selectively corrects the fractal code for selected domain blocks with an image-adaptive VQ codebook. The codebook is generated from the initial uncorrected fractal code and is therefore available at the decoder. An efficient trade-off results between incremental performance and bit rate.

We study the efficiency of the image sequence coding scheme based on 3-D iterated function systems (IFS). This coding scheme is an extension of the still image coding scheme proposed by Jacquin (1992). After a brief presentation of the IFS based coding and in particular the 3-D IFS scheme for coding image sequences, we simulate and discuss the already published schemes. We propose some improvements based on the choice of isometries and the construction of domain cubes during the coding.

This paper presents, a new technique for compression and secure image coding that is based on the iterated function system (IFS). A multi-resolution access control algorithm based on the convergence control of an iterative image reconstruction process is investigated. For multimedia applications an icon representation of low image quality is extracted directly from the encrypted data flow thanks to the partial self-affinity properties inherent to the local-IFS. The robustness of our scheme is due to the inherent high correlation of image data and to the non-linear decoding process.