Automatic face recognition systems are widely used in different applications which require authentication. Among various types of attacks against face recognition systems, morphing attacks have become a major concern, where face images of two subjects are combined into a face morph image which is submitted for enrolment. In a successful attack, both contributing subjects can then authenticate against the morph reference. In this work, we propose a new method to generate face morphs based on inversion of the optimal morph embeddings. To this end, we first find the optimal morph embeddings using the face embeddings of two source face images and then use state-of-the-art template inversion techniques to generate the morph. We use three different template inversion methods: the first one exploits a fully self-contained embedding-to-image inversion model, while the second and third leverage the realistic image generation of a pretrained StyleGAN network and a foundation model based on diffusion models, respectively. Furthermore, we use optimization methods to improve the performance of template inversion methods in the generation of face morph images from optimal morph embeddings. In our experiments, we evaluate the performance of generated face morph images and compare them with state-of-the-art morph generation methods, showing the superiority of our method. We showcase that our method can outperform state-of-the-art deep-learning-based morph generation methods, both in white-box and black-box attack scenarios, and compete with state-of-the-art landmark-based morph generation methods. Moreover, we perform a practical print-scan attack to simulate a real-world scenario and compare our method with previous methods in the literature, demonstrating the effectiveness and superiority of our method. The source code of our proposed method and all experiments are publicly available.

Face recognition datasets are often collected by crawling Internet and without individuals' consents, raising ethical and privacy concerns. Generating synthetic datasets for training face recognition models has emerged as a promising alternative. However, the generation of synthetic datasets remains challenging as it entails adequate inter-class and intra-class variations. While advances in generative models have made it easier to increase intra-class variations in face datasets (such as pose, illumination, etc.), generating sufficient inter-class variation is still a difficult task. In this paper, we formulate the dataset generation as a packing problem on the embedding space (represented on a hypersphere) of a face recognition model and propose a new synthetic dataset generation approach, called HyperFace. We formalize our packing problem as an optimization problem and solve it with a gradient descent-based approach. Then, we use a conditional face generator model to synthesize face images from the optimized embeddings. We use our generated datasets to train face recognition models and evaluate the trained models on several benchmarking real datasets. Our experimental results show that models trained with HyperFace achieve state-of-the-art performance in training face recognition using synthetic datasets. Project page: https://www.idiap.ch/paper/hyperface

Face recognition models are trained on large-scale datasets, which have privacy and ethical concerns. Lately, the use of synthetic data to complement or replace genuine data for the training of face recognition models has been proposed. While promising results have been obtained, it still remains unclear if generative models can yield diverse enough data for such tasks. In this work, we introduce a new method, inspired by the physical motion of soft particles subjected to stochastic Brownian forces, allowing us to sample identities distributions in a latent space under various constraints. We introduce three complementary algorithms, called Langevin, Dispersion, and DisCo, aimed at generating large synthetic face datasets. With this in hands, we generate several face datasets and benchmark them by training face recognition models, showing that data generated with our method exceeds the performance of previously GAN-based datasets and achieves competitive performance with state-of-the-art diffusion-based synthetic datasets. While diffusion models are shown to memorize training data, we prevent leakage in our new synthetic datasets, paving the way for more responsible synthetic datasets. Project page: https://www.idiap.ch/paper/synthetics-disco

This paper provides an overview of the recent advancements in foundation models and discusses potential applications of these models in the field of biometrics. Foundation models (such as large language models, vision language models, audio-language models, and large multi-modal models) are based on large neural networks which are trained with massive amounts of data and enable robust feature extraction for transfer learning. These models allow efficient zero-shot and few-shot learning, achieving state-of-the-art performance in downstream tasks. Foundation models have been studied and used in different domains, including natural language processing, computer vision, audio processing, and multi-modal processing. Biometrics is also an active field of research, which involves various research problems, ranging from robust recognition to security and privacy in biometric systems. In this paper, we present an in-depth analysis of state-of-the-art methodologies regarding foundation multi-modal models, their advancements, and their applicability to biometrics tasks. We also highlight current limitations and provide insights into potential future research directions in the applications of foundation models in biometrics. To our knowledge, this paper is the first survey which investigates the applications of foundation models in biometrics.

Large-scale face recognition datasets are collected by crawling the Internet and without individuals' consent, raising legal, ethical, and privacy concerns. With the recent advances in generative models, recently several works proposed generating synthetic face recognition datasets to mitigate concerns in web-crawled face recognition datasets. This paper presents the summary of the Synthetic Data for Face Recognition (SDFR) Competition held in conjunction with the 18th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2024) and established to investigate the use of synthetic data for training face recognition models. The SDFR competition was split into two tasks, allowing participants to train face recognition systems using new synthetic datasets and/or existing ones. In the first task, the face recognition backbone was fixed and the dataset size was limited, while the second task provided almost complete freedom on the model backbone, the dataset, and the training pipeline. The submitted models were trained on existing and also new synthetic datasets and used clever methods to improve training with synthetic data. The submissions were evaluated and ranked on a diverse set of seven benchmarking datasets. The paper gives an overview of the submitted face recognition models and reports achieved performance compared to baseline models trained on real and synthetic datasets. Furthermore, the evaluation of submissions is extended to bias assessment across different demography groups. Lastly, an outlook on the current state of the research in training face recognition models using synthetic data is presented, and existing problems as well as potential future directions are also discussed.

Face recognition systems tend toward ubiquity and are commonly utilized for security purposes. These systems operate based on facial representations, called templates, extracted by a deep neural network from each face image. However, it has been shown that face recognition templates can be inverted to reconstruct underlying face images, posing new security and privacy threats to face recognition systems. To mitigate such attacks against face recognition systems, several biometric template protection schemes have been proposed in the literature. The ISO/IEC 24745 standard requires each biometric template protection scheme to fulfill several requirements, among which non-invertibility is of the utmost importance. Therefore, each of the proposed template protection schemes in the literature used an ad-hoc approach to investigate the invertibility of the protected templates. In this paper, we consider a scenario where an adversary gains knowledge of a template protection scheme as well as its secrets, and tries to reconstruct a face image using a leaked protected template. We consider different template protection schemes, including Bio-Hashing, MLP-Hashing, and Homomorphic Encryption (HE), and reconstruct face images from protected templates. We also use different state-of-the-art face recognition models in both whitebox and blackbox scenarios. To our knowledge, this is the first work on learning-based reconstruction of face images from protected facial templates.

Coded aperture imaging is an emerging technique allowing thin form factor cameras that can be cheaply constructed. Many applications benefit from using such lensless cameras, such as face recognition. We propose a method for face recognition using coded aperture images that does not require retraining any component of the face recognition pipeline, but instead applies post-processing to the images with deep learning refinement so that they are compatible with existing face recognition for RGB images. We generate training data with a simulation process, based on the convolutional model of a lensless camera, and train a neural network to reconstruct face images. We train our network with a multi-term loss function to refine identity information in the reconstructed face image. We provide extensive experiments on different face recognition datasets, including LFW, CA-LFW, CP-LFW, AgeDB, FERET, and FRGC, showing the effectiveness and generalization of our proposed method. Our source code will be made available publicly to facilitate the reproducibility of our work.

Face recognition systems are widely used in different applications. In such systems, some features (called templates) are extracted from each face image and stored in the system’s database. In this paper, we propose an attack against face recognition systems where the adversary gains access to a portion of facial templates and aims to reconstruct the underlying face image. To this end, we train a face reconstruction network to invert partially leaked templates. In our experiments, we evaluate the vulnerability of state-of-the-art face recognition systems on different datasets, including MOBIO, LFW, and AgeDB. Our experiments demonstrate the vulnerability of face recognition systems to template inversion based on a portion of leaked templates. For example, with only 20% of facial templates, our experiments show that an adversary can achieve a success attack rate of 87% on a system based on ArcFace on the LFW dataset configured at the false match rate of 0.1%. To our knowledge, this paper is the first work on the inversion of partially leaked facial templates, and paves the way for future studies of attacks against face recognition systems based on partially leaked templates.

This paper explores the application of large language models (LLMs), like ChatGPT, for biometric tasks. We specifically examine the capabilities of ChatGPT in performing biometric-related tasks, with an emphasis on face recognition, gender detection, and age estimation. Since biometrics are considered as sensitive information, ChatGPT avoids answering direct prompts, and thus we crafted a prompting strategy to bypass its safeguard and evaluate the capabilities for biometrics tasks. Our study reveals that ChatGPT recognizes facial identities and differentiates between two facial images with considerable accuracy. Additionally, experimental results demonstrate remarkable performance in gender detection and reasonable accuracy for the age estimation tasks. Our findings shed light on the promising potentials in the application of LLMs and foundation models for biometrics.

Face recognition systems use the templates (extracted from users’ face images) stored in the system’s database for recognition. In a template inversion attack, the adversary gains access to the stored templates and tries to enter the system using images reconstructed from those templates. In this paper, we propose a framework to evaluate the vulnerability of face recognition systems to template inversion attacks. We build our framework upon a real-world scenario and measure the vulnerability of the system in terms of the adversary’s success attack rate in entering the system using the reconstructed face images. We propose a face reconstruction network based on a new block called “enhanced deconvolution using cascaded convolution and skip connections" (shortly,DSCasConv), and train it with a multi-term loss function. We use our framework to evaluate the vulnerability of state-of-the-art face recognition models, with different network structures and loss functions (in total 31 models), on the MOBIO, LFW, and AgeDB face datasets. Our experiments show that the reconstructed face images can be used to enter the system, which threatens the system’s security. Additionally, the reconstructed face images may reveal important information about each user’s identity, such as race, gender, and age, and hence jeopardize the users’ privacy.

Recent works have demonstrated the feasibility of inverting face recognition systems, enabling to recover convincing face images using only their embeddings. We leverage such template inversion models to develop a novel type of deep morphing attack based on inverting a theoretical optimal morph embedding, which is obtained as an average of the face embeddings of source images. We experiment with two variants of this approach: the first one exploits a fully self-contained embedding-to-image inversion model, while the second leverages the synthesis network of a pre-trained StyleGAN network for increased morph realism. We generate morphing attacks from several source datasets and study the effectiveness of those attacks against several face recognition networks. We showcase that our method can compete with and regularly beat the previous state of the art for deep-learning based morph generation in terms of effectiveness, both in white-box and black-box attack scenarios, and is additionally much faster to run. We hope this might facilitate the development of large scale deep morph datasets for training detection models.

This paper presents the summary of the Efficient Face Recognition Competition (EFaR) held at the 2023 International Joint Conference on Biometrics (IJCB 2023). The competition received 17 submissions from 6 different teams. To drive further development of efficient face recognition models, the submitted solutions are ranked based on a weighted score of the achieved verification accuracies on a diverse set of benchmarks, as well as the deployability given by the number of floating-point operations and model size. The evaluation of submissions is extended to bias, cross-quality, and large-scale recognition benchmarks. Overall, the paper gives an overview of the achieved performance values of the submitted solutions as well as a diverse set of baselines. The submitted solutions use small, efficient network architectures to reduce the computational cost, some solutions apply model quantization. An outlook on possible techniques that are underrepresented in current solutions is given as well.

State-of-the-art face recognition networks are often computationally expensive and cannot be used for mobile applications. Training lightweight face recognition models also requires large identity-labeled datasets. Meanwhile, there are privacy and ethical concerns with collecting and using large face recognition datasets. While generating synthetic datasets for training face recognition models is an alternative option, it is challenging to generate synthetic data with sufficient intra-class variations. In addition, there is still a considerable gap between the performance of models trained on real and synthetic data. In this paper, we propose a new framework (named SynthDistill) to train lightweight face recognition models by distilling the knowledge of a pretrained teacher face recognition model using synthetic data. We use a pretrained face generator network to generate synthetic face images and use the synthesized images to learn a lightweight student network. We use synthetic face images without identity labels, mitigating the problems in the intra-class variation generation of synthetic datasets. Instead, we propose a novel dynamic sampling strategy from the intermediate latent space of the face generator network to include new variations of the challenging images while further exploring new face images in the training batch. The results on five different face recognition datasets demonstrate the superiority of our lightweight model compared to models trained on previous synthetic datasets, achieving a verification accuracy of 99.52% on the LFW dataset with a lightweight network. The results also show that our proposed framework significantly reduces the gap between training with real and synthetic data. The source code for replicating the experiments is publicly released.

In this paper, we use synthetic data and propose a new method to reconstruct high-resolution face images from facial templates in a template inversion attack against face recognition systems. We use a pre-trained face generator network to generate synthetic face images, and then learn a mapping from the facial templates to the intermediate latent space of the face generator network. We train our mapping network with a multi-term loss function. During the inference stage, we use our mapping network to map facial templates to the intermediate latent code and then generate high-quality face images using the face generator network. We propose our method for whitebox and blackbox template inversion attacks against face recognition systems. We use our model (trained on synthetic data) to evaluate the vulnerability of state-of-the-art face recognition systems on real face datasets, including Labeled Faces in the Wild (LFW) and MOBIO datasets. Experimental results show the vulnerability of the state-of-the-art face recognition system to our template inversion attack. Our experiments also show that our template inversion method outperforms previous methods in the literature. The source code of our experiments is publicly available to facilitate reproducibility of our work.

Face recognition systems are increasingly being used in different applications. In such systems, some features (also known as embeddings or templates) are extracted from each face image. Then, the extracted templates are stored in the system’s database during the enrollment stage and are later used for recognition. In this paper, we focus on template inversion attacks against face recognition systems and introduce a novel method (dubbed GaFaR) to reconstruct 3D face from facial templates. To this end, we use a geometry-aware generator network based on generative neural radiance fields (GNeRF), and learn a mapping from facial templates to the intermediate latent space of the generator network. We train our network with a semi-supervised learning approach using real and synthetic images simul taneously. For the real training data, we use a Generative Adversarial Network (GAN) based framework to learn the distribution of the latent space. For the synthetic training data, where we have the true latent code, we directly train in the latent space of the generator network. In addition, during the inference stage, we also propose optimization on the camera parameters to generate face images to improve the success attack rate (up to 17.14% in our experiments). We evaluate the performance of our method in the whitebox and blackbox attacks against state-of-the-art face recognition models on the LFW and MOBIO datasets. To our knowledge, this paper is the first work on 3D face reconstruction from facial templates. The project page is available at: https://www.idiap.ch/paper/gafar

Face recognition systems generally store features (called embeddings) extracted from each face image during the enrollment stage, and then compare the extracted embeddings with the stored embeddings during the recognition stage. In this paper, we focus on the blackbox face reconstruction from facial embeddings stored in the face recognition database. We use a convolutional neural network (CNN) to reconstruct face images and train our network with a multi-term loss function. In particular, we use a different feature extractor trained for face recognition (which the adversary has the whitebox knowledge of it) to minimize the distance of embeddings extracted from the original and reconstructed face images. We evaluate our method in blackbox attacks against five state-of-the-art face recognition models on the MOBIO and LFW datasets. Our experimental results show that our proposed method outperforms previous face reconstruction methods in the literature. The source code of our experiments is publicly available to facilitate the reproducibility of our work.

In this paper, we focus on the template inversion attack against face recognition systems and propose a new method to reconstruct face images from facial templates. Within a generative adversarial network (GAN)-based framework, we learn a mapping from facial templates to the intermediate latent space of a pre-trained face generation network, from which we can generate high-resolution realistic reconstructed face images. We show that our proposed method can be applied in whitebox and blackbox attacks against face recognition systems. Furthermore, we evaluate the transferability of our attack when the adversary uses the reconstructed face image to impersonate the underlying subject in an attack against another face recognition system. Considering the adversary's knowledge and the target face recognition system, we define five different attacks and evaluate the vulnerability of state-of-the-art face recognition systems. Our experiments show that our proposed method achieves high success attack rates in whitebox and blackbox scenarios. Furthermore, the reconstructed face images are transferable and can be used to enter target face recognition systems with a different feature extractor model. We also explore important areas in the reconstructed face images that can fool the target face recognition system.

In this article, we comprehensively evaluate the vulnerability of state-of-the-art face recognition systems to template inversion attacks using 3D face reconstruction. We propose a new method (called GaFaR) to reconstruct 3D faces from facial templates using a pretrained geometry-aware face generation network, and train a mapping from facial templates to the intermediate latent space of the face generator network. We train our mapping with a semi-supervised approach using real and synthetic face images. For real face images, we use a generative adversarial network (GAN)-based framework to learn the distribution of generator intermediate latent space. For synthetic face images, we directly learn the mapping from facial templates to the generator intermediate latent code. Furthermore, to improve the success attack rate, we use two optimization methods on the camera parameters of the GNeRF model. We propose our method in the whitebox and blackbox attacks against face recognition systems and compare the transferability of our attack with state-of-the-art methods across other face recognition systems on the MOBIO and LFW datasets. We also perform practical presentation attacks on face recognition systems using the digital screen replay and printed photographs, and evaluate the vulnerability of face recognition systems to different template inversion attacks.

As the applications of biometric recognition systems are increasing rapidly, there is a growing need to secure the sensitive data used within these systems. Considering privacy challenges in such systems, different biometric template protection (BTP) schemes were proposed in the literature, and the ISO/IEC 24745 standard defined a number of requirements for protecting biometric templates. While there are several studies on evaluating different requirements of the ISO/IEC 24745 standard, there have been few studies on how to measure the linkability of biometric templates. In this paper, we propose a new method for measuring linkability of protected biometric templates. The proposed method is based on maximal leakage, which is a well-studied measure in information-theoretic literature. We show that the resulting linkability measure has a number of important theoretical properties and an operational interpretation in terms of statistical hypothesis testing. We compare the proposed measure to two other linkability measures: one previously introduced in the literature, and a similar measure based on differential privacy. In our experiments, we use the proposed measure to evaluate the linkability of biometric templates from different biometric characteristics (face, voice, and finger vein), which are protected with different BTP schemes. The source codes of our proposed measure and all experiments are publicly available.

Homomorphic Encryption (HE) has become a well-known tool for privacy-preserving recognition in biometric systems. Despite some important advantages of HE (such as preservation of recognition accuracy), there are two main drawbacks in the application of HE to biometric recognition systems: first, the security of the system solely depends on the secrecy of the private (decryption) key; second, the computational costs of the operations on the ciphertexts are expensive. To address these challenges, in this paper we propose a hybrid scheme for the protection of biometric templates, which combines cancelable biometrics (CB) methods and HE. Applying CB prior to HE enhances both the security and privacy of the overall system, since the protected templates remain irreversible even if the secret keys are leaked (commonly referred to as the full disclosure scenario). In addition, we can reduce the dimensionality of templates using CB before applying HE, which speeds up the computation over the ciphertexts. We use BioHashing, Multi-Layer Perceptron (MLP) hashing, and Index-of-Maximum (IoM) hashing as different CB methods, and for each of these schemes, we propose a method for computing scores between hybrid-protected templates in the encrypted domain. We evaluate our proposed hybrid scheme using different state-of-the-art face recognition models (Ar-cFace, ElasticFace, and FaceNet) on the MOBIO and LFW datasets. The source code of our experiments is publicly available, so our work can be fully reproduced.

In this work, we present a simple biometric indexing scheme which is binning and retrieving cancelable deep face templates based on frequent binary patterns. The simplicity of the proposed approach makes it applicable to unprotected as well as protected, i.e. cancelable, deep face templates. As such, this approach represents to the best of the authors' knowledge the first generic indexing scheme that can be applied to arbitrary cancelable face templates (o binary representation). In experiments, deep face templates are obtained from the Labelled Faces in the Wild (LFW) dataset using the ArcFace face recognition system for feature extraction. Protected templates are then generated by employing different cancelable biometric schemes, i.e. BioHashing and two variants of Index-of-Maximum Hashing. The proposed indexing scheme is evaluated on closed- and open-set identification scenarios. It is shown to maintain the recognition accuracy of the baseline system while reducing the penetration rate and hence the workload of identifications to approximately 40%.

State-of-the-art (SOTA) face recognition systems generally use deep convolutional neural networks (CNNs) to extract deep features, called embeddings, from face images. The face embeddings are stored in the system’s database and are used for recognition of the enrolled system users. Hence, these features convey important information about the user’s identity, and therefore any attack using the face embeddings jeopardizes the user’s security and privacy. In this paper, we propose a CNN-based structure to reconstruct face images from face embeddings and we train our network with a multi-term loss function. In our experiments, our network is trained to reconstruct face images from SOTA face recognition models (ArcFace and ElasticFace) and we evaluate our face reconstruction network on the MOBIO and LFW datasets. The source code of all the experiments presented in this paper is publicly available so our work can be fully reproduced.

Abstract With the emergence of social networks and improvements in the internet speed, the video data has become an ever‐increasing portion of the global internet traffic. Besides the content, the quality of a video sequence is an important issue at the user end which is often affected by various factors such as compression. Therefore, monitoring the quality is crucial for the video content and service providers. A simple monitoring approach is to compare the raw video content (uncompressed) with the received data at the receiver. In most practical scenarios, however, the reference video sequence is not available. Consequently, it is desirable to have a general reference‐less method for assessing the perceived quality of any given video sequence. In this paper, a no‐reference video quality assessment technique based on video features is proposed. In particular, a long list of video features (21 sets of features, each consisting of 1 to 216 features) is considered and all possible combinations () for training an Extra Trees regressor is examined. This choice of the regressor is wisely selected and is observed to perform better than other common regressors. The results reveal that the top 20 performing feature subsets all outperform the existing feature‐based assessment methods in terms of the Pearson linear correlation coefficient (PLCC) or the Spearman rank order correlation coefficient (SROCC). Specially, the best performing regressor achieves on the test data over the KonVid‐1k dataset. It is believed that the results of the comprehensive comparison could be potentially useful for other feature‐based video‐related problems. The source codes of the implementations are publicly available.

In digital imaging, especially in the process of data acquisition and transmission, images are often affected by impulsive noise. Therefore, it is essential to remove impulsive noise from images before any further processing. Due to the remarkable performance of deep neural networks in different applications of image processing and computer vision, we present an end-to-end fully convolutional neural network to remove impulsive noise from images. To train our network, we generate a customized dataset with various noise densities in which the highly corrupted images are more frequent. Hence, our convolutional neural network is blind since the percentage of impulsive noise is not required as prior knowledge. Moreover, we define a multi-term loss function to train our network. In particular, we define a novel term to impose the sparsity nature of impulsive noise. Experimental results indicate that our deep learning approach significantly outperforms other state-of-the-art methods in terms of reconstruction quality and speed on a system equipped with GPU. Meanwhile, we introduce a fast iterative method, as a post-processing stage, to further improve the reconstruction quality of our neural network. The proposed post-processing algorithm improves the reconstruction quality in only a fraction of a second.

Biometric recognition systems relying on finger vein have gained a lot of attention in recent years. Besides security, the privacy of finger vein recognition systems is always a crucial concern. To address the privacy concerns, several biometric template protection (BTP) schemes are introduced in the literature. However, despite providing privacy, BTP algorithms often affect the recognition performance. In this paper, we propose a deep-learning-based approach for secure finger vein recognition. We use a convolutional auto-encoder neural network with a multi-term loss function. In addition to the auto-encoder loss function, we deploy triplet loss for the embedding features. Next, we apply Biohashing to our deep features to generate protected templates. The experimental results indicate that the proposed method achieves superior performance to previous finger vein recognition methods protected with Biohashing. Besides, our proposed method has less execution time and requires less memory.1

In this letter, we consider the problem of blind source separation under certain nonlinear mixing conditions using a deep learning approach. Conventionally, the separation of sources within linear mixtures is achieved by applying the independence property of the sources. In the nonlinear regime, however, this property is no longer sufficient. In this letter, we consider nonlinear mixing operators where the non-linearity could be fairly approximated using a Taylor series. Next, for solving the nonlinear BSS problem, we design an end-to-end recurrent neural network (RNN) that learns the inverse of the system, and ultimately separates the sources. For training the RNN, we employ a set of multi-variate polynomial functions to simulate the Taylor expansion of the nonlinear mixture. Numerical experiments show that the proposed method successfully separates the sources with a performance superior to the state of the art approaches.