Feature selection is a critical aspect of eXplainable Artificial Intelligence (XAI), and it has implications for model interpretability and predictive performance. CoFE (Consistency-driven Feature Elimination) framework was introduced recently using a frequentist approach. CoFE eliminates features with inconsistent coefficient signs in Linear Regression models by estimating the Sign Entropy (variability of the sign) of the coefficients using Bootstrapping. However, the uncertainty associated with estimating Sign Entropy using bootstrapping leads to slower convergence and inconsistency in feature subset selection in CoFE. In this paper, we present Bay-CoFE, 1 a Bayesian reformulation of CoFE, to solve the slow convergence and inconsistency issues of CoFE while retaining the benefits of selecting features with lower Sign Entropy in a Bayesian framework. We provide theoretical justifications and empirical evidence to prove Bay-CoFE’s superior convergence properties. Across all datasets, Bay-CoFE achieves significantly superior sign stability compared to traditional feature selection methods (Mann-Whitney U test, p-value < = 1.63e-03 and mean Cliff’s delta ≈ 0.83), with minimal predictive performance differences (Mann-Whitney U test, p-value > 0.1 and mean Cliff’s delta ≈ 0.32), demonstrating a highly favorable trade-off for interpretable modeling.

Medical image segmentation plays a pivotal role in reducing radiologists’ workload by enabling accurate severity assessment and treatment planning. With coronary heart disease becoming an increasing concern globally, there is a growing need for efficient and reliable segmentation methods. In this context, we present TRACS, a self-supervised Triple Attention Network designed to enhance multi-scale feature representation for coronary vessel segmentation. The encoder generates dynamic channel-wise attention weights, effectively capturing fine structural details across multiple scales. This design is particularly well-suited for segmenting thin, elongated structures, such as vessels in angiograms, which are often challenging to delineate due to low contrast and occluded vessels. Unlike earlier self-supervised methods that rely on diffusion processes or multi-generator adversarial learning, which often introduce training complexities and optimization instabilities. TRACS adopts a streamlined architecture that not only ensures stable, consistent performance but also operates with a significantly reduced parameter count, making it a lightweight, efficient self-supervised architecture. The training process combines a fusion of losses to optimize both for regional accuracy and boundary precision. We evaluate TRACS on diverse unseen coronary angiograms (134XCA and 30XCA) and retinal vessel images (DRIVE and STARE). TRACS achieves significant inference gains in segmentation performance and inference speed, operating up to 3x faster than the nearest baseline method and with a significantly lower memory footprint, making it highly efficient for real-time applications and deployment on resource-constrained devices. We supplement the results with a thorough statistical analysis and explainability results from various Gradient and perturbation-based techniques. Results indicate that TRACS outperforms several recent self-supervised methods and matches the performance of supervised baselines in segmentation accuracy and robustness.TRACS offers a practical, domain-independent solution for vessel-structure segmentation in medical imaging. The code is publicly available on github .

Deep learning models often struggle with hard-to-learn classes, which arise not only from class imbalance or long-tailed distributions but also from intrinsic complexity, particularly in medical applications. Existing loss functions typically target long-tailed and imbalanced datasets, yet they often fail to address the challenges posed by classes that remain difficult to learn beyond frequency effects. Moreover, their expected impact is difficult to assess prior to costly model training. We propose a Loss Function Assessment (LFA) framework that evaluates loss functions, quantifying how their gradients contribute to learning between positive and negative labels when one is predicted more poorly, providing a diagnostic of learning dynamics. As a use case, we derive the Focal balanced Exponential Cross Entropy (F-ECE) loss from LFA analysis, combining exponential weighting with focal balancing to illustrate how the framework can inform new loss designs. We validate LFA on widely used losses, including Binary Cross Entropy, Focal Loss, Asymmetric Loss, and F-ECE, across CIFAR10-LT and ImageNet-LT (long-tailed distributions), CelebA (attribute-specific hard classes) and CAD-CAP (medical dataset). F-ECE achieves up to 1.95% recall improvement on CelebA, with consistent gains in F1-score and balanced accuracy across all datasets. Code and protocols are available at: https://github.com/Bozhao-Liu/LFA-demo .

Saliency maps are widely used as a post-hoc approach to explain the decision-making process of Deep Learning (DL) based image classification models, but evaluating their fidelity remains a complex problem. While saliency metrics have been introduced to evaluate the fidelity of saliency maps, existing saliency metrics, such as perturbation-based saliency metrics, have been previously reported to demonstrate statistical inconsistency. Although inconsistencies have been noted in different works, there exists no mechanism for estimating the same, i.e., Inconsistency Estimation (IE). Our primary objective is to address this limitation, and therefore, we propose a framework to estimate the inconsistency of saliency metrics for any given DL model. The framework enables building IE models for estimating the inconsistency by employing a set of perturbation types and schemes. The framework’s modular architecture provides flexibility across (i) perturbation types (Inpainting, Uniform, and Gaussian blur), (ii) perturbation schemes (pixel-wise and patch-wise), (iii) learning mechanisms (Convolutional Neural Networks and Vision Transformers) and (iv) IE modeling techniques (bagging and boosting). Extensive experimental results are shown on three well-known DL architectures (Inception-V3, Xception, and ResNet-50) on three different public datasets, including the Imagenette, Oxford-IIIT Pets Dataset, and PASCAL VOC 2007, along with results on ViTs for Oxford-IIIT Pets Dataset, and PASCAL VOC 2007. With a comprehensive evaluation of seven different perturbation types that include two inpainting, two Gaussian blur (with kernel widths of 0.9 and 1.5), and three uniform perturbations, our work shows the effectiveness of the proposed approach in estimating inconsistency. Statistically founded tests such as repeated cross-validation and the Permutation Test further validate the idea of the proposed framework for estimating the inconsistency of saliency metrics across unseen perturbations, making it useful in real-world scenarios.

Adversarial attacks pose a critical threat to the reliability of medical image classification in Industry 4.0, where smart, interconnected devices drive high-stakes diagnostic decisions. This article introduces Stealth, a simple yet highly optimized black-box adversarial attack specifically designed for medical imaging applications. Stealth is the first framework to utilize low-rank perturbation for adversarial generation. An optimization framework iteratively refines the complete singular value spectrum, reconstructs the perturbed image, and queries the target classifier to verify attack success—without relying on surrogate models or prediction probabilities. Improved imperceptibility is achieved by preserving visual fidelity throughout the perturbation process, ensuring the adversarial outputs remain indistinguishable from clean images. Operating as a true black-box method, Stealth requires only hard-label outputs, making it ideal for real-world deployments in Industry 4.0-enabled medical environments. Unlike existing approaches, it is entirely model-agnostic and image-modality independent, enabling broad applicability across healthcare systems. Extensive experiments across publicly available datasets—Chest X-Ray, Colonoscopy, MRI, and Mammogram—demonstrate that Stealth consistently achieves high attack success rates while surpassing state-of-the-art white-box attacks in perceptual quality, with explainability analysis further confirming its impact on model decision-making. These results expose critical vulnerabilities in current healthcare artificial intelligence (AI) systems and highlight the need for more robust adversarial defenses.

The ACM and IEEE released CS2013 with recommendations for undergraduate computer science programs, updated in 2023 by ACM, IEEE, and AAAI as CS2023. CS2023 emphasizes emerging fields while maintaining fundamental concepts. A paradigm shift towards a broader perspective on education has led higher educational institutions to consider factors such as sustainability, inter-personal competencies, digital competencies and inter-disciplinary skills necessitating a revision of the computer science curriculum across all universities. Accounting for such factors, the Norwegian University of Science and Technology (NTNU), has identified that it is not only the curriculum that needs to be revised, but it is also crucial to enhance the surrounding infrastructure that includes improving lecturer's skills, promoting graduate active learning, fostering life-long learning and ensuring alignment with industry and societal needs. While these goals seem welcoming and ambitious, it is essential to understand and bridge the gap between the program management's perception and the graduates' actual experiences. This paper analysis and maps the expected outcomes of ten principles and twelve competencies defined by the university, aiming to prepare the computer science curriculum for next-generation graduates. The detailed analysis verifies if the degree program meets future educational, societal and industrial standards. To conduct such an analysis, we first design a dedicated survey on the experiences of previous students for each of the ten outlined principles and twelve competencies. The responses gathered from 59 graduates and 7 program managers from three different campuses, are analyzed using quantitative methods to understand their experiences and perceptions. By mapping the responses from program management with the graduated students, the paper determines the gap in curriculum and then presents recommendations to computer science educational programs offered at other universities with similar objectives to bridge the gap. By considering graduate experiences across three campuses with the same curriculum, this study offers insights into how various aspects contribute to achieving the principles and competencies. The findings aim to guide similar CS2023-based programs at universities beyond Scandinavia.

The emergence of Deepfake technology poses significant threats, particularly regarding misinformation and privacy. To mitigate these threats, Deepfake benchmarks play an important role in developing and testing reliable Deepfake detection algorithms. Consequently, it is crucial that these benchmarks do not possess serious biases that hinder the robustness and generalizability of Deepfake detectors during training and subsequently distort their true reliability during operation. This work investigates inherent biases in various Deepfake detection benchmark datasets by training simple classification models based on soft-biometric facial properties that do not contain Deepfake-related clues, i.e., decoy features. These mirage models reach up to 87.42% (balanced) accuracy on benchmark datasets using irrelevant decoy features alone for this task. As large parts of the performance of state-of-the-art models could also be achieved through exploiting benchmark biases, this raises the question of the unbiased performance of Deepfake detectors and their general reliability. Our analysis includes various Deepfake detection benchmarks and analyzes soft-biometric properties in determining their contribution to “solving” these benchmarks. Our findings underscore the need for more unbiased benchmarks beyond simply balancing demographic groups to enable future work on developing reliable solutions.

Face morphing attacks threaten the integrity of biometric identity systems by enabling multiple individuals to share a single identity. To develop and evaluate effective morphing attack detection (MAD) systems, we need access to high-quality, realistic morphed images that reflect the challenges posed in real-world scenarios. However, existing morph generation methods often produce images that are blurry, riddled with artifacts, or poorly constructed—making them easy to detect and not representative of the most dangerous attacks. In this work, we introduce StableMorph, a novel approach that generates highly realistic, artifact-free morphed face images using modern diffusion-based image synthesis. Unlike prior methods, StableMorph produces full-head images with sharp details, avoids common visual flaws, and offers unmatched control over visual attributes. Through extensive evaluation, we show that StableMorph images not only rival or exceed the quality of genuine face images, but also maintain a strong ability to fool face recognition systems—posing a greater challenge to existing MAD solutions and setting a new standard for morph quality in research and operational testing. StableMorph improves the evaluation of biometric security by creating more realistic and effective attacks and supports the development of more robust detection systems.

This paper presents a summary of the 2nd Latent in the Wild Fingerprint Recognition Competition held at the 2025 International Joint Conference on Biometrics. The competition has two tracks: latent fingerprint 1) recognition, and 2) quality assessment. It attracted a total of 12 participating teams from academia and industry for both tracks, representing 10 countries. In total, 8 valid submissions were evaluated by the organizers. The competition aimed to advance the state-of-the-art in latent fingerprint recognition and quality assessment by providing a challenging dataset of latent fingerprints collected in natural, non-ideal conditions. This paper summarizes the dataset, evaluation protocols, submitted methods, and the competition results.

This work summarises and reports the results of the second Presentation Attack Detection competition on ID cards. This new version includes new elements compared to the previous one. (1) An automatic evaluation platform was enabled for automatic benchmarking; (2) Two tracks were proposed in order to evaluate algorithms and datasets respectively; and (3) A new ID card dataset was shared with Track 1 teams to serve as the baseline dataset for the training and optimisation. The Hochschule Darmstadt, Fraunhofer-IGD, and Facephi company jointly organised this challenge. 20 teams were registered, and 74 submitted models were evaluated. For Track 1, the "Dragons" team reached first place with an Average Ranking and Equal Error rate (EER) of (AVRank) of 40.48% and 11.44% EER, respectively. For the more challenging approach in Track 2, the "Incode" team reached the best results with an AVRank of 14.76% and 6.36% EER, improving on the results of the first edition of 74.30% and 21.87% EER, respectively. These results suggest that PAD on ID cards is improving, but it is still a challenging problem related to the number of images, especially of bona fide images.

The unavailability of expert annotations for learning deep models for pathology detection in Wireless Capsule Endoscopy (WCE) has led to an increase in explorations of semi-supervised learning. Semi-supervised models reduce the dependency on large-scale annotations. However, the resulting quality of representations relies on the quality of pseudo-labels generated from unlabeled data. In this work, we develop domain-specific augmentations in conjunction with weighted box fusion and active sampling to better select the unlabeled samples and reduce annotation cost in WCE. We demonstrate the improvement in performance using the proposed idea on two different datasets including SEE-AI and Kvasir-Capsule, with multiple standard metrics like average precision and its variants. The experimental results reveal that domain-specific augmentations with active sampling can help in selecting the most informative unlabeled samples, making it possible to improve semi-supervised models. With an annotation cost of only 35% of the actively selected unlabeled data, our method performs better than the state-of-the-art models on the same datasets for both fully supervised Faster-RCNN and Semi-supervised Unbiased Teacher and Active Teacher. We achieved a gain of +3.09% in AP50 on SEE-AI and +3.37% in AP75 on Kvasir-Capsule over the Active Teacher model. Our code is available at https://github.com/agossouema2011/SSOD_With_DTA_WCE.

Smartphones balance security and convenience by offering both knowledge-based (PINs, patterns) and biometric (facial, fingerprint) verification methods. However, studies have reported that PINs and patterns can be readily circumvented, while synthetically manipulated face data can easily deceive smartphone facial verification mechanisms. In this paper, we design a bimodal user verification mechanism that combines behavioral (pickup gesture) and biological (face) biometrics for user verification on smartphones. This work establishes a baseline for single-user verification scenarios on smartphones using a one-class verification model. The evaluation is performed in two stages: first, performance is assessed in both unimodal and bimodal settings using publicly available datasets; second, the robustness of the employed biological and behavioral traits is examined against four diverse attacks. Our findings emphasize the necessity of investigating diverse attack vectors, particularly fully synthetic data, to design robust user verification mechanisms.

Explanations of Local Interpretable Model-agnostic Explanations (LIME) are often inconsistent across different runs making them unreliable for eXplainable AI (XAI). The inconsistency stems from sign flips and variability in ranks of the segments for each different run. We propose a Bayesian Regularization approach to reduce sign flips, which in turn stabilizes feature rankings and ensures significantly higher consistency in explanations. The proposed approach enforces sparsity by incorporating a Sign Entropy prior on the coefficient distribution and dynamically eliminates features during optimization. Our results demonstrate that the explanations from the proposed method exhibit significantly better consistency and fidelity than LIME (and its earlier variants). Further, our approach exhibits comparable consistency and fidelity with a significantly lower execution time than the latest LIME variant, i.e., SLICE (CVPR 2024).

The importance of quantifying uncertainty in deep networks has become paramount for reliable real-world applications. In this paper, we propose a method to improve uncertainty estimation in medical Image-to-Image (I2I) translation. Our model integrates aleatoric uncertainty and employs Uncertainty-Aware Regularization (UAR) inspired by simple priors to refine uncertainty estimates and enhance reconstruction quality. We show that by leveraging simple priors on parameters, our approach captures more robust uncertainty maps, effectively refining them to indicate precisely where the network encounters difficulties, while being less affected by noise. Our experiments demonstrate that UAR not only improves translation performance, but also provides better uncertainty estimations, particularly in the presence of noise and artifacts. We validate our approach using two medical imaging datasets, showcasing its effectiveness in maintaining high confidence in familiar regions while accurately identifying areas of uncertainty in novel/ambiguous scenarios.

Person re-identification (ReID) has gained significant interest due to growing public safety concerns that require advanced surveillance and identification mechanisms. While most existing ReID research relies on static surveillance cameras, the use of Unmanned Aerial Vehicles (UAVs) for surveillance has recently gained popularity. Noting the promising application of UAVs in ReID, this paper presents a comprehensive overview of UAV-based ReID, highlighting publicly available datasets, key challenges, and methodologies. We summarize and consolidate evaluations conducted across multiple studies, providing a unified perspective on the state of UAV-based ReID research. Despite their limited size and diversity, We underscore current datasets’ importance in advancing UAV-based ReID research. The survey also presents a list of all available approaches for UAV-based ReID. The survey presents challenges associated with UAV-based ReID, including environmental conditions, image quality issues, and privacy concerns. We discuss dynamic adaptation techniques, multi-model fusion, and lightweight algorithms to leverage ground-based person ReID datasets for UAV applications. Finally, we explore potential research directions, highlighting the need for diverse datasets, lightweight algorithms, and innovative approaches to tackle the unique challenges of UAV-based person ReID. • Comprehensive survey of all publicly available datasets for UAV-based person ReID. • Detailed discussion on challenges in UAV-based person ReID. • Detailed analysis of UAV ReID methodologies and current state-of-the-art. • Potential future directions for UAV-based person ReID.

Despite their exceptional performance in the Internet of Medical Things (IoMT), deep learning models are susceptible to adversarial attacks. Existing defense approaches often suffer from limitations, including required model alterations, attack-type awareness, and high-computational complexity. This article introduces MedDefend: a lightweight, three-way detection technique, combining adaptive noise injection with a tailored robust principal component analysis (t-RPCA)-based noise mitigation. Initially, the method employs strategic image- dependent Gaussian noise injection, guided by class activation maps, to mask adversarial perturbations. Subsequently, t-RPCA is employed to eliminate the introduced noise along with the adversarial perturbations. Finally, the model evaluates classification consistency between original and denoised samples to detect potential adversarial examples. MedDefend effectively detects adversarial attacks across various medical imaging modalities, with a lightweight, training-free, and model-agnostic design suitable for IoMT integration. To encourage community engagement and reimplementation, our code is available athttps://github.com/nirmalpadichira/MedDefend/tree/main.

Class Activation Maps (CAMs) are one of the important methods for visualizing regions used by deep learning models. Yet their robustness to different noise remains underexplored. In this work, we evaluate and report the resilience of various CAM methods for different noise perturbations across multiple architectures and datasets. By analyzing the influence of different noise types on CAM explanations, we assess the susceptibility to noise and the extent to which dataset characteristics may impact explanation stability. The findings highlight considerable variability in noise sensitivity for various CAMs. We propose a robustness metric for CAMs that captures two key properties: consistency and responsiveness. Consistency reflects the ability of CAMs to remain stable under input perturbations that do not alter the predicted class, while responsiveness measures the sensitivity of CAMs to changes in the prediction caused by such perturbations. The metric is evaluated empirically across models, different perturbations, and datasets along with complementary statistical tests to exemplify the applicability of our proposed approach.

Local Interpretable Model-agnostic Explanations (LIME) - a widely used post-ad-hoc model agnostic ex-plainable AI (XAI) technique. It works by training a simple transparent (surrogate) model using random samples drawn around the neighborhood of the instance (image) to be explained (IE). Explanations are then extracted for a black-box model and a given IE, using the surrogate model. However, the explanations of LIME suffer from inconsistency across different runs for the same model and the same IE. We identify two main types of inconsistencies: variance in the sign and importance ranks of the segments (superpixels). These factors hinder LIME from obtaining consistent explanations. We analyze these inconsistencies and propose a new method, Stabilized LIME for Consistent Explanations (SLICE). The proposed method handles the stabilization problem in two aspects: using a novel feature selection technique to eliminate spurious superpixels and an adaptive perturbation technique to generate perturbed images in the neighborhood of IE. Our results demonstrate that the explanations from SLICE exhibit significantly better consistency and fidelity than LIME (and its variant BayLime).

Numerous studies have shown that existing Face Recognition Systems (FRS), including commercial ones, often exhibit biases toward certain ethnicities due to under-represented data. In this work, we explore ethnicity alteration and skin tone modification using synthetic face image generation methods to increase the diversity of datasets. We conduct a detailed analysis by first constructing a balanced face image dataset representing three ethnicities: Asian, Black, and Indian. We then make use of existing Generative Adversarial Network-based (GAN) image-to-image translation and manifold learning models to alter the ethnicity from one to another. A systematic analysis is further conducted to assess the suitability of such datasets for FRS by studying the realistic skin-tone representation using Individual Typology Angle (ITA). Further, we also analyze the quality characteristics using existing Face image quality assessment (FIQA) approaches. We then provide a holistic FRS performance analysis using four different systems. Our findings pave the way for future research works in (i) developing both specific ethnicity and general (any to any) ethnicity alteration models, (ii) expanding such approaches to create databases with diverse skin tones, (iii) creating datasets representing various ethnicities which further can help in mitigating bias while addressing privacy concerns.

This paper presents a summary of the Latent in the Wild Fingerprint Recognition Competition held at the 2024 International Joint Conference on Biometrics (IJCB 2024). The competition attracted a total of 6 participating teams from academia and industry, representing 6 countries. In total, 3 valid submissions were evaluated by the organizers. The competition aimed to advance the state-of-the-art in latent fingerprint recognition by providing a challenging dataset of latent fingerprints collected in natural, non-ideal conditions. This paper summarizes the dataset, evaluation criteria, participant methods, and the competition results.

Morphing attacks blend face images of multiple distinct identities into a single photo combining their facial characteristics. Since the morphed image can be verified to multiple identities, these attacks greatly threaten face recognition systems. Morphing Attack Detection (MAD), addresses this problem by determining if an image is a bona fide or morphed image. Previous works on MAD focused on the models’ decision to improve their performance and generalizability. However, it has not been investigated whether the estimated probabilities of such decisions accurately reflect the true confidence with which predictions are made. Since these models struggle with unseen attacks, it is important to determine the models’ decision confidence to decide to trust or distrust the decisions. In this work, we (a) demonstrate that state-of-the-art MAD models struggle with providing reliable confidence estimations, (b) propose two new metrics to categorize their confidence prediction behavior, and (c) demonstrate that a simple calibration method can make the model more trustworthy without changing general model performance. The experiments were conducted in cross-dataset evaluation settings across two different MAD models using three different preprocessing and five morphing generation techniques. We found that both models are not calibrated and therefore not trustworthy. Moreover, it is shown that a simple and effective adjustment of the prediction confidence is possible. This will make future presentation attack detection, and especially MAD, models more trustworthy.

Acquiring face images of sufficiently high quality is important for online ID and travel document issuance applications using face recognition systems (FRS). Low-quality, manipulated (intentionally or unintentionally), or distorted images degrade the FRS performance and facilitate documents’ misuse. Securing quality for enrolment images, especially in the unsupervised self-enrolment scenario via a smartphone, becomes important to assure FRS performance. In this work, we focus on the less studied area of radial distortion (a.k.a., the fish-eye effect) in face images and its impact on FRS performance. We introduce an effective radial distortion detection model that can detect and flag radial distortion in the enrolment scenario. We formalize the detection model as a face image quality assessment (FIQA) algorithm and provide a careful inspection of the effect of radial distortion on FRS performance. Evaluation results show excellent detection results for the proposed models, and the study on the impact on FRS uncovers valuable insights into how to best use these models in operational systems.

Through the Face Morphing attack is possible to use the same legal document by two different people, destroying the unique biometric link between the document and its owner. In other words, a morphed face image has the potential to bypass face verification-based security controls, then representing a severe security threat. Unfortunately, the lack of public, extensive and varied training datasets severely hampers the development of effective and robust Morphing Attack Detection (MAD) models, key tools in contrasting the Face Morphing attack since able to automatically detect the presence of morphing images. Indeed, privacy regulations limit the possibility of acquiring, storing, and transferring MAD-related data that contain personal information, such as faces. Therefore, in this paper, we investigate the use of Federated Learning to train a MAD model on local training samples across multiple sites, eliminating the need for a single centralized training dataset, as common in Machine Learning, and then overcoming privacy limitations. Experimental results suggest that FL is a viable solution that will need to be considered in future research works in MAD.

This paper summarises the Competition on Presentation Attack Detection on ID Cards (PAD-IDCard) held at the 2024 International Joint Conference on Biometrics (IJCB 2024). The competition attracted a total of ten registered teams, both from academia and industry. In the end, the participating teams submitted five valid submissions, with eight models to be evaluated by the organisers. The competition presented an independent assessment of current state-of-the-art algorithms. Today, no independent evaluation on cross-dataset is available; therefore, this work determined the state-of-the-art on ID cards. To reach this goal, a sequestered test set and baseline algorithms were used to evaluate and compare all the proposals. The sequestered test dataset contains ID cards from four different countries. In summary, a team that chose to be "Anonymous" reached the best average ranking results of 74.80%, followed very closely by the "IDVC" team with 77.65%.

It has been shown that digitally manipulated face images can pose a security threat to automated authentication systems (e.g., when such systems are used for border control). In such scenarios, a malicious actor can, in many countries, apply for an identity document using a manipulated face image, which can then be used to gain fraudulent access to a system. Research has shown that humans and algorithms struggle to detect digitally manipulated face images, especially when the type of manipulation is unknown or when evaluated across multiple types of manipulations. In this work, we consider the detection performance of algorithms and humans on datasets consisting of retouched, face swapped and morphed images. Specifically, we investigate the joint performance of algorithms and humans in a differential detection scenario where both a trusted and suspected image are presented simultaneously. To this end, we propose a conditional face image manipulation detection approach where the human decision is only considered when the algorithm is unsure about the decision outcome. The results show that the automated algorithm performs better than the human detectors and that combining the decisions of algorithms and humans, in general, leads to an increased detection performance. To our knowledge, this is the first study to explore the joint detection performance of algorithms and humans in a differential face manipulation detection scenario and when using a variety of face image manipulations.

Face inpainting, the technique of restoring missing or damaged regions in facial images, is pivotal for applications like face recognition in occluded scenarios and image analysis with poor-quality captures. This process not only needs to produce realistic visuals but also preserve individual identity characteristics. The aim of this paper is to inpaint a face given periocular region (eyes-to-face) through a proposed new Generative Adversarial Network (GAN)-based model called Eyes-to-Face Network (E2F-Net). The proposed approach extracts identity and non-identity features from the periocular region using two dedicated encoders have been used. The extracted features are then mapped to the latent space of a pre-trained StyleGAN generator to benefit from its state-of-the-art performance and its rich, diverse and expressive latent space without any additional training. We further improve the StyleGAN's output to find the optimal code in the latent space using a new optimization for GAN inversion technique. Our E2F-Net requires a minimum training process reducing the computational complexity as a secondary benefit. Through extensive experiments, we show that our method successfully reconstructs the whole face with high quality, surpassing current techniques, despite significantly less training and supervision efforts. We have generated seven eyes-to-face datasets based on well-known public face datasets for training and verifying our proposed methods. The code and datasets are publicly available1.

Estimating building footprint maps from geospatial data is vital in urban planning, development, disaster management, and various other applications. Deep learning methodologies have gained prominence in building segmentation maps, offering the promise of precise footprint extraction without extensive post-processing. However, these methods face challenges in generalization and label efficiency, particularly in remote sensing, where obtaining accurate labels can be both expensive and time-consuming. To address these challenges, we propose terrain-aware self-supervised learning, tailored to remote sensing, using digital elevation models from LIght Detection and Ranging (LiDAR) data. We propose to learn a model to differentiate between bare Earth and superimposed structures enabling the network to implicitly learn domain-relevant features without the need for extensive pixel-level annotations. We test the effectiveness of our approach by evaluating building segmentation performance on test datasets with varying label fractions. Remarkably, with only 1% of the labels (equivalent to 25 labeled examples), our method improves over ImageNet pretraining, showing the advantage of leveraging unlabeled data for feature extraction in the domain of remote sensing. The performance improvement is more pronounced in few-shot scenarios and gradually closes the gap with ImageNet pretraining as the label fraction increases. We test on a dataset characterized by substantial distribution shifts (including resolution variation and labeling errors) to demonstrate the generalizability of our approach. When compared to other baselines, including ImageNet pretraining and more complex architectures, our approach consistently performs better, demonstrating the efficiency and effectiveness of self-supervised terrain-aware feature learning.

Latent fingerprints are among the most important and widely used evidence in crime scenes, digital forensics and law enforcement worldwide. Despite the number of advancements reported in recent works, we note that significant open issues such as independent benchmarking and lack of large-scale evaluation databases for improving the algorithms are inadequately addressed. The available databases are mostly of semi-public nature, lack of acquisition in the wild environment, and post-processing pipelines. Moreover, they do not represent a realistic capture scenario similar to real crime scenes, to benchmark the robustness of the algorithms. Further, existing databases for latent fingerprint recognition do not have a large number of unique subjects/fingerprint instances or do not provide ground truth/reference fingerprint images to conduct a cross-comparison against the latent. In this paper, we introduce a new wild large-scale latent fingerprint database that includes five different acquisition scenarios: reference fingerprints from (1) optical and (2) capacitive sensors, (3) smartphone fingerprints, latent fingerprints captured from (4) wall surface, (5) Ipad surface, and (6) aluminium foil surface. The new database consists of 1,318 unique fingerprint instances captured in all above mentioned settings. A total of 2,636 reference fingerprints from optical and capacitive sensors, 1,318 fingerphotos from smartphones, and 9,224 latent fingerprints from each of the 132 subjects were provided in this work. The dataset is constructed considering various age groups, equal representations of genders and backgrounds. In addition, we provide an extensive set of analysis of various subset evaluations to highlight open challenges for future directions in latent fingerprint recognition research.

In this work, we propose QMagFace, a simple and effective face recognition solution (QMagFace) that combines a quality-aware comparison score with a recognition model based on a magnitude-aware angular margin loss. The proposed approach includes model-specific face image qualities in the comparison process to enhance the recognition performance under unconstrained circumstances. Exploiting the linearity between the qualities and their comparison scores induced by the utilized loss, our quality-aware comparison function is simple and highly generalizable. The experiments conducted on several face recognition databases and benchmarks demonstrate that the introduced quality-awareness leads to consistent improvements in the recognition performance. Moreover, the proposed QMagFace approach performs especially well under challenging circumstances, such as cross-pose, cross-age, or cross-quality. Consequently, it leads to state-of-the-art performances on several face recognition benchmarks, such as 98.50% on AgeDB, 83.95% on XQLFQ, and 98.74% on CFP-FP. The code for QMagFace is publicly available1.

Social media platforms present significant threats against underage users targeted for predatory intents. Many early research works have applied the footprints left by online predators to investigate online grooming. While digital forensics tools provide security to online users, it also encounters some critical challenges, such as privacy issues and the lack of data for research in this field. Our literature review investigates all research papers on grooming detection in online conversations by looking at the psychological definitions and aspects of grooming. We study the psychological theories behind the grooming characteristics used by machine learning models that have led to predatory stage detection. Our survey broadly considers the authorship profiling research works used for grooming detection in online conversations, along with predatory conversation detection and predatory identification approaches. Various approaches for online grooming detection have been evaluated based on the metrics used in the grooming detection problem. We have also categorized the available datasets and used feature vectors to give readers a deep knowledge of the problem considering their constraints and open research gaps. Finally, this survey details the constraints that challenge grooming detection, unaddressed problems, and possible future solutions to improve the state-of-the-art and make the algorithms more reliable.

Bias and fairness of biometric algorithms have been key topics of research in recent years, mainly due to the societal, legal and ethical implications of potentially unfair decisions made by automated decision-making models. A considerable amount of work has been done on this topic across different biometric modalities, aiming at better understanding the main sources of algorithmic bias or devising mitigation measures. In this work, we contribute to these efforts and present the first study investigating bias and fairness of sclera segmentation models. Although sclera segmentation techniques represent a key component of sclera-based biometric systems with a considerable impact on the overall recognition performance, the presence of different types of biases in sclera segmentation methods is still underexplored. To address this limitation, we describe the results of a group evaluation effort (involving seven research groups), organized to explore the performance of recent sclera segmentation models within a common experimental framework and study performance differences (and bias), originating from various demographic as well as environmental factors. Using five diverse datasets, we analyze seven independently developed sclera segmentation models in different experimental configurations. The results of our experiments suggest that there are significant differences in the overall segmentation performance across the seven models and that among the considered factors, ethnicity appears to be the biggest cause of bias. Additionally, we observe that training with representative and balanced data does not necessarily lead to less biased results. Finally, we find that in general there appears to be a negative correlation between the amount of bias observed (due to eye color, ethnicity and acquisition device) and the overall segmentation performance, suggesting that advances in the field of semantic segmentation may also help with mitigating bias.

The transition from traditional to digital teaching has led to several challenges for students and educators under the COVID-19 pandemic. Most universities are experiencing remote online delivery and assessment for the first time, which creates several issues, particularly for delivering courses efficiently and evaluating the outcomes without students compromising academic integrity. In this study, we take a closer look at a cross-campus case in delivering a programming course that switched to digital teaching due to COVID-19. We focus on the transition in assessment forms and the gradual adaptation to portfolio evaluation over two years. This form of assessment is more aligned with the constructive alignment theory, and hence contribute to increased learning outcomes as the students receive feedback along the way. Furthermore, we introduced a task where students had to reflect on the answers and solutions. This reflection note may help the teacher to better understand to which level the student has actually understood the theories and skills applied in the solution, and hence reassure that the solution is produced by the student. Observations from our data provide promising direction that it can increase learning benefits and reduce possibilities or the need for cheating and contributes to increased learning outcomes. However, this form of assessment requires a significant effort from teachers and is both time and resource consuming.

This paper presents a summary of the Competition on Face Morphing Attack Detection Based on Privacy-aware Synthetic Training Data (SYN-MAD) held at the 2022 In-ternational Joint Conference on Biometrics (IJCB 2022). The competition attracted a total of 12 participating teams, both from academia and industry and present in 11 differ-ent countries. In the end, seven valid submissions were submitted by the participating teams and evaluated by the organizers. The competition was held to present and at-tract solutions that deal with detecting face morphing at-tacks while protecting people's privacy for ethical and le-gal reasons. To ensure this, the training data was limited to synthetic data provided by the organizers. The submitted solutions presented innovations that led to out-performing the considered baseline in many experimental settings. The evaluation benchmark is now available at: https://github.com/marcohuber/SYN-MAD-2022.

A MasterFace is a face image that can successfully match against a large portion of the population. Since their generation does not require access to the information of the enrolled subjects, MasterFace attacks represent a potential security risk for widely-used face recognition systems. Previous works proposed methods for generating such images and demonstrated that these attacks can strongly compromise face recognition. However, previous works followed evaluation settings consisting of older recognition models, limited cross-dataset and cross-model evaluations, and the use of low-scale testing data. This makes it hard to state the generalizability of these attacks. In this work, we comprehensively analyse the generalizability of MasterFace attacks in empirical and theoretical investigations. The empirical investigations include the use of six state-of-the-art face recognition models, cross-dataset and cross-model evaluation protocols, and utilizing testing datasets of significantly higher size and variance. The results indicate a low generalizability when MasterFaces are training on a different face recognition model than the one used for testing. In these cases, the attack performance is similar to zero-effort imposter attacks. In the theoretical investigations, we define and estimate the face capacity and the maximum MasterFace coverage under the assumption that identities in the face space are well separated. The current trend of increasing the fairness and generalizability in face recognition indicates that the vulnerability of future systems might further decrease. Future works might analyse the utility of MasterFaces for understanding and enhancing the robustness of face recognition models.

Face recognition systems (FRS) are vulnerable to different kinds of attacks. Morphing attack combines multiple face images to obtain a single face image that can verify equally against all contributing subjects. Various Morphing Attack Detection (MAD) algorithms have been proposed in recent years albeit limited generalizability. We present a new approach for MAD in this work with better generalization than state-of-the-art (SOTA) algorithms. We propose an end-to-end multi-stage encoder-decoder network for learning the residuals of morphing process to detect attacks. Leveraging the residuals, we learn an efficient classifier using cross-entropy loss and asymmetric loss. The use of asymmetric loss in our approach is motivated by imbalanced distribution of morphs and bona fides. An extensive set of experiments are conducted on five different datasets consisting of two landmark based and three Generative Adversarial Network (GAN) based morphs in various settings such as digital, print-scan and print-scan-compression. We first demonstrate a near-ideal performance of the proposed MAD with Detection Equal Error Rate (D-EER) of 0% in the best case and 2.58% in the worst case in the digital domain in closed-set protocol, i.e., known attacks. Further, we demonstrate the applicability of the proposed approach on 60 different combinations where the testing set contains unknown morphing attacks in open-set protocol to illustrate the generalization ability of our proposed approach. Through training the proposed approach on landmark-based morph generation data alone, we obtain an EER of 3.59% in the best case and 12.89% in the worst case for morphed images in the digital domain, reducing the error rates from 45.67% and 30.23% respectively, in open-set protocol. We further present an extensive analysis of the proposed approach through Class Activation Maps (CAM) to explain the decisions using by making use of three complementary CAM analysis.

Recently, deep convolutional neural networks have achieved remarkable performance for the task of Single Image Super-Resolution (SISR); however these models set huge amount of computational complexity to achieve such performance. Hence, computationally efficient and low memory models are needed for SISR if such models are deployed on resources with low-computational devices (for instance, Mobile). We propose a novel approach referred to as Channel Split Convolutional Neural Network for Single Image Super-Resolution (CSISR). The proposed work aims to create a light-weight but efficient network to enhance SR performance. It employs a unique channel splitting alongside channel reduction blocks, leading to more effectiveness with a less computational burden to obtain state-of-the-art accuracy. Further, we suggest a new strategy for Channel Attention (CA) using a combination of global average and standard deviation pooling accompanied by conventional non-linear mapping layers to improve the learning. The performance of the proposed network is validated on various benchmark testing datasets for the SISR task, which shows its superiority over other existing methods.

This paper presents a summary of the Masked Face Recognition Competitions (MFR) held within the 2021 International Joint Conference on Biometrics (IJCB 2021). The competition attracted a total of 10 participating teams with valid submissions. The affiliations of these teams are diverse and associated with academia and industry in nine different countries. These teams successfully submitted 18 valid solutions. The competition is designed to motivate solutions aiming at enhancing the face recognition accuracy of masked faces. Moreover, the competition considered the deployability of the proposed solutions by taking the compactness of the face recognition models into account. A private dataset representing a collaborative, multisession, real masked, capture scenario is used to evaluate the submitted solutions. In comparison to one of the topperforming academic face recognition solutions, 10 out of the 18 submitted solutions did score higher masked face verification accuracy.

Face morphing attacks are evolving as a significant threat to the Face Recognition Systems (FRS) operating in border control and passport issuance. As newborn face has very limited discriminative facial characteristics, it is challenging for both human and machines to verify the newborns based on the facial biometrics accurately. Further, the introduction of face morphing elevates the problem of baby trafficking as it can challenge both human and machine-based facial verification. In this paper, we pose a question if the morphed images of newborns can threaten FRS and present first systematic study on the vulnerability analysis of FRS towards morphed faces of newborns. To effectively benchmark threat of newborns’ facial morphing attacks, we introduce a new face morphing dataset constructed based on 42 unique newborns with 852 bona fide and 2451 morphing images. Extensive experiments are carried out on the newly constructed dataset to benchmark the vulnerability against both Commercial-Off-The-Shelf (COTS) FRS (Cognitec FaceVACS-SDK Version 9.4.2) and deep learning based FRS (Arcface) for three different morphing factors. Further, we also evaluate the performance of Morphing Attack Detection (MAD) in detecting such morphing attacks of newborn faces. We conduct experiments on four different Off-The-Shelf MAD techniques to benchmark the detection performance on newborn morph attacks.

For iris recognition in non-cooperative environments, iris segmentation has been regarded as the first most important challenge still open to the biometric community, affecting all downstream tasks from normalization to recognition. In recent years, deep learning technologies have gained significant popularity among various computer vision tasks and also been introduced in iris biometrics, especially iris segmentation. To investigate recent developments and attract more interest of researchers in the iris segmentation method, we organized the 2021 NIR Iris Challenge Evaluation in Non-cooperative Environments: Segmentation and Localization (NIR-ISL 2021) at the 2021 International Joint Conference on Biometrics (IJCB 2021). The challenge was used as a public platform to assess the performance of iris segmentation and localization methods on Asian and African NIR iris images captured in non-cooperative environments. The three best-performing entries achieved solid and satisfactory iris segmentation and localization results in most cases, and their code and models have been made publicly available for reproducibility research.

Morphing attacks have posed a severe threat to Face Recognition System (FRS). Despite the number of advancements reported in recent works, we note serious open issues such as independent benchmarking, generalizability challenges and considerations to age, gender, ethnicity that are inadequately addressed. Morphing Attack Detection (MAD) algorithms often are prone to generalization challenges as they are database dependent. The existing databases, mostly of semi-public nature, lack in diversity in terms of ethnicity, various morphing process and post-processing pipelines. Further, they do not reflect a realistic operational scenario for Automated Border Control (ABC) and do not provide a basis to test MAD on unseen data, in order to benchmark the robustness of algorithms. In this work, we present a new sequestered dataset for facilitating the advancements of MAD where the algorithms can be tested on unseen data in an effort to better generalize. The newly constructed dataset consists of facial images from 150 subjects from various ethnicities, age-groups and both genders. In order to challenge the existing MAD algorithms, the morphed images are with careful subject pre-selection created from the contributing images, and further post-processed to remove morphing artifacts. The images are also printed and scanned to remove all digital cues and to simulate a realistic challenge for MAD algorithms. Further, we present a new online evaluation platform to test algorithms on sequestered data. With the platform we can benchmark the morph detection performance and study the generalization ability. This work also presents a detailed analysis on various subsets of sequestered data and outlines open challenges for future directions in MAD research.

The deep learning models for the Single Image Super-Resolution (SISR) task have found success in recent years. However, one of the prime limitations of existing deep learning-based SISR approaches is that they need supervised training. Specifically, the Low-Resolution (LR) images are obtained through known degradation (for instance, bicubic downsampling) from the High-Resolution (HR) images to provide supervised data as an LR-HR pair. Such training results in a domain shift of learnt models when real-world data is provided with multiple degradation factors not present in the training set. To address this challenge, we propose an unsupervised approach for the SISR task using Generative Adversarial Network (GAN), which we refer to hereafter as DUS-GAN. The novel design of the proposed method accomplishes the SR task without degradation estimation of real-world LR data. In addition, a new human perception-based quality assessment loss, i.e., Mean Opinion Score (MOS), has also been introduced to boost the perceptual quality of SR results. The pertinence of the proposed method is validated with numerous experiments on different reference-based (i.e., NTIRE Real-world SR Challenge validation dataset) and no-reference based (i.e., NTIRE Real-world SR Challenge Track-1 and Track-2) testing datasets. The experimental analysis demonstrates committed improvement from the proposed method over the other state-of-the-art unsupervised SR approaches, both in terms of subjective and quantitative evaluations on different reference metrics (i.e., LPIPS, PI-RMSE graph) and no-reference quality measures such as NIQE, BRISQUE and PIQE. We also provide the implementation of the proposed approach (https://github.com/kalpeshjp89/DUSGAN) to support reproducible research.

The growing popularity of Virtual Reality and Augmented Reality (VR/AR) devices in many applications also demands authentication of users. As the devices inherently capture the eye image while capturing the user interaction, the authentication can be devised using the iris and periocular recognition. While both iris and periocular data being non-ideal unlike the data captured from standard biometric sensors, the authentication performance is expected to be lower. In this work, we present and evaluate a fusion framework for improving the biometric authentication performance. Specifically, we employ score-level fusion for two independent biometric systems of iris and periocular region to avoid expensive feature-level fusion. With a detailed evaluation of three different score-level fusion after the score normalization on a dataset of 12579 images, we report the performance gain in authentication using score-level fusion for iris and periocular recognition.

Face morphing attacks have demonstrated a severe threat in the passport issuance protocol that weakens the border control operations. A morphed face images if used after printing and scanning (re-digitizing) to obtain a passport is very challenging to be detected as attack. In this paper, we present a novel method to detect such morphing attacks using an ensemble of features computed on the scale-space representation derived from the color space for a given image. Given the limited availability of datasets representing realistic morphing attacks, we introduce and present a new print-scan image dataset of morphed face images. Experiments are carried out on the two different datasets and compared with sixteen existing state-of-art Morphing Attack Detection (MAD) mechanism based on single image MAD (S-MAD). The proposed approach indicates a superior MAD performance on both datasets suggesting the applicability in operational scenarios.

Augmented and virtual reality is being deployed in different fields of applications. Such applications might involve accessing or processing critical and sensitive information, which requires strict and continuous access control. Given that Head-Mounted Displays (HMD) developed for such applications commonly contains internal cameras for gaze tracking purposes, we evaluate the suitability of such setup for verifying the users through iris recognition. In this work, we first evaluate a set of iris recognition algorithms suitable for HMD devices by investigating three well-established handcrafted feature extraction approaches, and to complement it, we also present the analysis using four deep learning models. While taking into consideration the minimalistic hardware requirements of stand-alone HMD, we employ and adapt a recently developed miniature segmentation model (EyeMMS) for segmenting the iris. Further, to account for non-ideal and non-collaborative capture of iris, we define a new iris quality metric that we termed as Iris Mask Ratio (IMR) to quantify the iris recognition performance. Motivated by the performance of iris recognition, we also propose the continuous authentication of users in a non-collaborative capture setting in HMD. Through the experiments on a publicly available OpenEDS dataset, we show that performance with EER = 5% can be achieved using deep learning methods in a general setting, along with high accuracy for continuous user authentication.

Face morphing attacks have raised critical concerns as they demonstrate a new vulnerability of Face Recognition Systems (FRS), which are widely deployed in border control applications. The face morphing process uses the images from multiple data subjects and performs an image blending operation to generate a morphed image of high quality. The generated morphed image exhibits similar visual characteristics corresponding to the biometric characteristics of the data subjects that contributed to the composite image and thus making it difficult for both humans and FRS, to detect such attacks. In this paper, we report a systematic investigation on the vulnerability of the Commercial-Off- The-Shelf (COTS) FRS when morphed images under the influence of ageing are presented. To this extent, we have introduced a new morphed face dataset with ageing derived from the publicly available MORPH II face dataset, which we refer to as MorphAge dataset. The dataset has two bins based on age intervals, the first bin - MorphAge-I dataset has 1002 unique data subjects with the age variation of 1 year to 2 years while the MorphAge-II dataset consists of 516 data subjects whose age intervals are from 2 years to 5 years. To effectively evaluate the vulnerability for morphing attacks, we also introduce a new evaluation metric, namely the Fully Mated Morphed Presentation Match Rate (FMMPMR), to quantify the vulnerability effectively in a realistic scenario. Extensive experiments are carried out using two different COTS FRS (COTS I Cognitec FaceVACS-SDK Version 9.4.2 and COTS II - Neurotechnology version 10.0) to quantify the vulnerability with ageing. Further, we also evaluate five different Morph Attack Detection (MAD) techniques to benchmark their detection performance with respect to ageing.

The paper presents a summary of the 2020 Sclera Segmentation Benchmarking Competition (SSBC), the 7th in the series of group benchmarking efforts centred around the problem of sclera segmentation. Different from previous editions, the goal of SSBC 2020 was to evaluate the performance of sclera-segmentation models on images captured with mobile devices. The competition was used as a platform to assess the sensitivity of existing models to i) differences in mobile devices used for image capture and ii) changes in the ambient acquisition conditions. 26 research groups registered for SSBC 2020, out of which 13 took part in the final round and submitted a total of 16 segmentation models for scoring. These included a wide variety of deep-learning solutions as well as one approach based on standard image processing techniques. Experiments were conducted with three recent datasets. Most of the segmentation models achieved relatively consistent performance across images captured with different mobile devices (with slight differences across devices), but struggled most with low-quality images captured in challenging ambient conditions, i.e., in an indoor environment and with poor lighting.

Wrist-wearable devices such as smartwatch hardware have gained popularity as they provide quick access to various information and easy access to multiple applications. Among the numerous smartwatch applications, user verification based on the handwriting is gaining momentum by considering its reliability and user-friendliness. In this paper, we present a novel technique for user verification using a smartwatch based writing pattern or style. The proposed approach leverages accelerometer data captured from the smartwatch that is further represented using 2D Continuous Wavelet Transform (CWT) and deep features extracted using the pre-trained ResNet50. These features are classified using an ensemble of classifiers to make the final decision on user verification. Extensive experiments are carried out on a newly captured dataset using two different smartwatches with three different writing scenarios (or activities). Experimental results provide critical insights and analysis of the results in such a verification scenario.

Along with the deployment of the Face Recognition Systems (FRS), concerns were raised related to the vulnerability of those systems towards various attacks including morphed attacks. The morphed face attack involves two different face images in order to obtain via a morphing process a resulting attack image, which is sufficiently similar to both contributing data subjects. The obtained morphed image can successfully be verified against both subjects visually (by a human expert) and by a commercial FRS. The face morphing attack poses a severe security risk to the e-passport issuance process and to applications like border control, unless such attacks are detected and mitigated. In this work, we propose a new method to reliably detect a morphed face attack using a newly designed demising framework. To this end, we design and introduce a new deep Multi-scale Context Aggregation Network (MS-CAN) to obtain denoised images, which is subsequently used to determine if an image is morphed or not. Extensive experiments are carried out on three different morphed face image datasets. The Morphing Attack Detection (MAD) performance of the proposed method is also benchmarked against 14 different state-of-the-art techniques using the ISO-IEC 30107-3 evaluation metrics. Based on the obtained quantitative results, the proposed method has indicated the best performance on all three datasets and also on cross-dataset experiments.

The periocular region is used for authentication in the recent days under unconstrained acquisition in biometrics. This work presents two new feature extraction techniques to achieve robust and blur invariant biometric verification using periocular images captured using smartphones - (1) Deep Sparse Features (DSF) and (2) Deep Sparse Time Frequency Features (DeSTiFF). Both the approaches are based on extracting features via convolution of periocular images with a set of filters also referred as Deep Sparse Filters. The filters are learnt using natural image patches and sparse filtering approach. The DSF is obtained through convolution via Deep Sparse Filters. Further, convoluted responses are analyzed using Short Term Fourier Transform (STFT) to obtain time and frequency features of the images referred as DeSTIFF. The features obtained from the newly proposed feature extraction techniques are further represented in a collaborative subspace to achieve better verification performance. Both of the proposed feature extraction schemes are evaluated on two publicly available smartphone periocular databases and a new database (Visible Spectrum Periocular Image (VISPI) database) released with this article. The robustness of the proposed feature extraction is exemplified by comparing it with state-of-art approaches along with multiple deep networks where the improvement is evidently seen on large scale database with an average verification accuracy of Genuine Match Rate ≈ 98% at False Match Rate = 0.01%. We further support reproducible research by making the code and the database available for the academic research.

Face recognition systems are widely known for their vulnerability against presentation attacks or spoofing attacks. The exponential deployment of face recognition systems has been further challenged even by the simple and low-cost face artefacts generated using conventional printers. In this paper, we present a novel scheme to detect face presentation attacks posed by high-quality print attacks which are relatively difficult to detect. The proposed scheme leverages the phase information extracted from the spatial-frequency representation of the given image. We also present a new face presentation attack database collected using the iPhone 6S. The new database is comprised of 100 subjects collected in two different sessions that have resulted in a total of 31228 samples (or images). Extensive experiments are carried out on the newly constructed database and the obtained results show the improved performance of the proposed scheme when compared aaainst six different state-of-the-art methods.

Multi-spectral imaging has been explored to handle a set of deficiencies found in traditional imaging that capture the images only in visible spectrum (VIS) or Near-Infra Red (NIR) spectrum. The promising performance obtained in the experimental works indicates the use-case in real-life biometric systems. As biometric systems should also consider protecting biometric templates, it is required to have an efficient template protection scheme for multi-spectral biometric systems to avoid the leakage of biometric data and subsequent linkability issues. In this work, we propose a new template protection scheme for multi-spectral biometric systems through the use of biometric information across different spectra to provide protected templates. Through the proposed approach of kernalized hashing, we can reach a fully unlinkable template protection scheme that works across all spectra in a multi-spectral system with a comparable performance to unprotected system. Further, we propose a template level fusion across all the spectral bands to improve the performance of the multi-spectral biometric system with integrated template protection. Through the use of a relatively large sized multispectral face biometric database of 168 subjects captured in 9 narrow spectral bands in VIS and NIR bands (530nm to 1000nm), we illustrate the effectiveness of the proposed approach in achieving a robust and secure template protection while accounting for irreversibility, unlinkability and renewability. Through the experiments we establish the performance of the proposed template protection approach and demonstrate a high Genuine Match Rate (GMR ≈ 100% at False Accept Rate of FMR=0.01%) and low Equal Error Rate EER= ≈ 0%, while satisfying other requirements of biometric template protection. Further, we present a security analysis of the proposed approach to demonstrate the unlinkability of the biometric templates.

Biometric systems are prone to random and systematic errors which are typically attributed to the variations in terms of inter-session data capture and intra-session variability. Furthermore, these errors cannot be defined and modeled mathematically in many cases, but we can associate them with uncertainty based on certain conditions. In such cases, one of the possible approach to improve biometric system performance is to employ multi-biometric fusion by incorporating the uncertainties. In the literature, researchers have proposed many fusion techniques, but most of these techniques do not take uncertainty into account while performing fusion. Since the decision made by uni-modal biometric comparators do not consider the uncertainty involved in such decisions, it is essential first to model the uncertainty before combining the decision from multiple uni-modal biometric systems efficiently. To this end, we propose a score level multi-biometric fusion scheme using Subjective Logic which incorporates the uncertainty of the system's information channels while fusing the scores. Extensive experiments are carried out on the multi-biometric NIST BSSR1, and the proposed scheme has indicated a superior performance with a genuine match rate of 99.02 % at a false match rate fixed to 0.01 %.

Presentation attacks against Face Recognition System (FRS) have incrementally posed challenges to create new detection methods. Among the various presentation attacks, disguise attacks allow concealing the identity of the attacker thereby increasing the vulnerability of the FRS. In this paper, we present a new approach for attack detection in multi-spectral systems, where face disguise attacks are carried out. The approach is based on using spectral signatures obtained from a spectral camera operating in eight narrow spectral bands across the Visible (VIS) and Near Infra-Red (NIR) (530nm to 1000nm) spectrum and learning deeply coupled auto-encoders. The robustness of the proposed approach is validated using a newly collected spectral face database of subjects conducting both bona fide (i.e. real) presentations and disguise attack presentations. The database is designed to capture 2 different kinds of attacks from 54 subjects, amounting to a total number of 6480 samples. Extensive experiments carried on the multi-spectral face database indicate the robust performance of proposed scheme when benchmarked with three different state-of-the-art methods.

Multi-spectral imaging has recently acquired significant attention in biometrics based authentication due to it's potential ability to capture spatio-spectral images across the electromagnetic spectrum. Especially, in the case of facial biometrics, multi-spectral imaging has shown significant promising results under unknown/varying illumination environment. However, the challenge arises when surveillance cameras provide the visible images while the enrollment are spectral band images. In order to address the backward/cross compatibility of probing visible images from regular surveillance cameras against the high quality spectral band images in enrollment, development of robust algorithms are required. In this paper, we present a new approach of selecting optimal band based on highest correlation coefficients of individual feature vectors from bands in comparison with feature vectors from visible images of respective individual classes for robust recognition performance. The proposed approach of band selection is validated on a newly collected face database of 168 subjects whose face images are collected in 9 different spectral bands and correspondingly their visible images from a regular camera operating in visible spectrum. The extensive set of experiments conducted on the new database with selected single band and multiple spectral bands in enrollment data versus the visible probe image has indicated the significance of the band selection. The new approach of spectral to visible matching with the proposed band selection method shows significant Rank- 1 recognition rate of 94.04% supporting the applicability of proposed method

Multispectral face recognition systems are widely used in various access control applications. The vulnerability of multispectral face recognition sensors towards low-cost Presentation Attack Instrument (PAI) such as printed photos used in attacks has emerged as a serious security threat. In this paper, we present a novel framework to detect presentation attacks against an extended multispectral face sensor. The proposed framework stems from the idea of exploring the complementary information available from different bands of an extended multispectral face sensor. To this extent, two different frameworks are proposed where the first framework is based on image fusion and the second builds on the Presentation Attack Detection (PAD) score level fusion. Extensive experiments are carried out on the extended multispectral face sensor database comprising of 50 subjects with two different presentation attacks generated using the printed photo artefacts. The obtained results indicate the superior performance of the PAD score level fusion on detecting both known and unknown attacks.

Periocular characteristics has gained substantial importance in recent times to supplement the performance of facial biometrics or as a stand-alone characteristics. While most of the current biometric systems for authentication or surveillance operate either in NIR spectrum or visible spectrum, the ocular information can be well utilized if a comparison of images from different spectra has to be conducted. In this work, we present a novel approach employing the features obtained from steerable pyramids to compare the ocular images captured from NIR versus the images captured from visible spectrum. The set of features obtained using the proposed cross-spectral approach are then used to learn a multi-class SVM classifier such that the probe image originating from another spectrum can be classified. Further, a fusion frame-work for combining the scores from different orientations of the steerable pyramid is proposed for a particular scale to strengthen the biometric performance of the algorithm. An extensive set of experiments conducted on a large database consisting of ocular images captured from 120 subjects (240 unique ocular instances) indicates the robustness of the proposed approach with a GMR of 100% at the FMR of 0.01% in a benchmark against other state-of-the-art techniques suggesting the applicability of proposed approach to greater extent.

With the availability of sensor technology across the broad electromagnetic spectrum, multi-spectral imaging is increasingly used in biometric systems. Especially for face recognition, multi-spectral imaging has gained a lot of attention due to it's invariant property against variation caused by unknown illumination. However, obtaining best performance using multi-spectral imaging is still a challenge due to presence of a modality gap between the spectral imaging data and redundant band information. In this paper, we propose a fused band representation with a set of selected bands represented in Quaternion space for spectral band images to efficiently maintain the inter band relationship in spatial domain. The selection is based on measuring the information content in bands using entropy and fusion is carried out in Quaternion space for three best bands. The features from newly obtained image is collaboratively represented to achieve robust performance. The proposed approach is experimentally validated on the extended multi-spectral face database of 168 subjects, whose spectral band images are captured in 9 narrow spectral bands in visible and near infrared range (530nm to 1000nm). The quantitative performance analysis, obtained using the proposed method indicates 96.13% recognition rate at Rank-1, outperforming other state-of-the-art methods.

The Face Recognition System (FRS) is known to be vulnerable to the attacks using the morphed face. As the use of face characteristics are mandatory in the electronic passport (ePass), morphing attacks have raised the potential concerns in the border security. In this paper, we analyze the vulnerability of the FRS to the new attack performed using the averaged face. The averaged face is generated by simple pixel level averaging of two face images corresponding to two different subjects. We benchmark the vulnerability of the commercial FRS to both conventional morphing and averaging based face attacks. We further propose a novel algorithm based on the collaborative representation of the micro-texture features that are extracted from the colour space to reliably detect both morphed and averaged face attacks on the FRS. Extensive experiments are carried out on the newly constructed morphed and averaged face image database with 163 subjects. The database is built by considering the real-life scenario of the passport issuance that typically accepts the printed passport photo from the applicant that is further scanned and stored in the ePass. Thus, the newly constructed database is built to have the print-scanned bonafide, morphed and averaged face samples. The obtained results have demonstrated the improved performance of the proposed scheme on print-scanned morphed and averaged face database.

Smartphone based facial biometric systems have been well used in many of the security applications starting from simple phone unlocking to secure banking applications. This work presents a new approach of exploring the intrinsic characteristics of the smartphone camera to capture a number of stack images in the depth-of-field. With the set of stack images obtained, we present a new feature-free and classifier-free approach to provide the presentation attack resistant face biometric system. With the entire system implemented on the smartphone, we demonstrate the applicability of the proposed scheme in obtaining a stack of images with varying focus to effectively determine the presentation attacks. We create a new database of 13250 images at different focal length to present a detailed analysis of vulnerability together with the evaluation of proposed scheme. An extensive evaluation of the newly created database comprising of 5 different Presentation Attack Instruments (PAI) has demonstrated an outstanding performance on all 5 PAI through proposed approach. With the set ofcomplementary benefits of proposed approach illustrated in this work, we deduce the robustness towards unseen 2D attacks.

This work presents the 2ndCross-Spectrum Iris/Periocular Recognition Competition (Cross-Eyed2017). The main goal of the competition is to promote and evaluate advances in cross-spectrum iris and periocular recognition. This second edition registered an increase in the participation numbers ranging from academia to industry: five teams submitted twelve methods for the periocular task and five for the iris task. The benchmark dataset is an enlarged version of the dual-spectrum database containing both iris and periocular images synchronously captured from a distance and within a realistic indoor environment. The evaluation was performed on an undisclosed test-set. Methodology, tested algorithms, and obtained results are reported in this paper identifying the remaining challenges in path forward.

Spectral face recognition is gaining importance as the information from different bands can lead to robust face representation that are presentation (a.k.a, spoofing) attack resistant. However, the key challenge here is to process the high dimensional spatio-spectral data to extract and represent the reliable data while discarding redundant data information for processing. In this work, we present a new approach to represent the spectral images in a holistic manner by employing the well-known Projection Metric Learning (PML) in Grassmann manifold such that the redundant information from the spectral data is discarded while retaining significant information. The approach is adopted to learn discriminative information from an high dimensional spectral dataset. Further, we propose an extension using collaborative representation of learnt projection metrics for improving the classification accuracy of spectral data. With the extensive set of experiments conducted on a relatively large scale extended-spectral face image database (6048 images) of 168 subjects, we demonstrate the applicability of the proposed framework. The obtained results indicates highest accuracy by achieving a Rank-1 recognition rate of 98.21% in classifying the spectral face images.

Gender prediction based on facial features has received significant attention in computer vision and biometric community. Most of the gender classification studies mainly focused there attention on approaches that operate in the visible spectrum. In this paper we present gender classification using extended multi-spectral face data captured in nine narrow spectral bands across the visible near infrared spectrum (530nm to 1000nm). Further, we present the proposed method, that learns for this image set the discriminative spectral band features in the affine space and then classifies the features with a Support Vector Machine (SVM) in a robust manner. The extensive experimental results are presented on the reasonable sample size of 78300 spectral band images using our proposed method. The obtained results shows 90.49±3.56% average classification accuracy, indicating the applicability of our proposed method for gender classification.

Contact lens detection in the eye is a significant task to improve the reliability of iris recognition systems. A contact lens overlays the iris region and prevents the iris sensor from capturing the normal iris region. In this paper, we present a novel scheme for detection to detecting a contact lens using Deep Convolutional Neural Network (CNN). The proposed CNN architecture ContlensNet is structured to have fifteen layers and configured for the three-class detection problem with the following classes: images with textured (or colored) contact lens, soft (or transparent) contact lens, and no contact lens. The proposed ContlensNet is trained using numerous iris image patches and the problem of overfitting the network is addressed by using the dropout regularization method. Extensive experiments are carried out on two publicly available large-scale databases, namely: IIIT-Delhi Contact lens iris database (IIITD) and Notre Dame cosmetic contact lens database 2013 (ND) that are comprised of contact lens iris samples captured using four different sensors. The obtained results have demonstrated the improved performance of the proposed scheme with an average performance improvement of more than 10% in Correct Classification Rate (CCR%) when compared with eight different state-of-the-art contact lens detection systems.

Ocular recognition on smartphone authentication applications are gaining popularity in academic research and in the commercial sector where operators are requesting reliable and robust biometric authentication. The wide acceptance of such ocular based authentication systems also depends on the verification performance on large scale testing with different data subject ethnic groups and platforms. In this work, we evaluate such a large database of ocular images collected using three different phones. Further, we benchmark the verification performance and propose a new framework to improve it. The proposed framework is based on collaboratively represented features from deep sparse filtering. We obtain a verification performance of Genuine Match Rate (GMR) of 97.56% at a False Match Rate (FMR) of 0.001% for periocular images obtained from a Samsung device. The overall performance of around 95% GMR at a FMR of 0.001% not only indicates the robust nature of the proposed framework, but also illustrates the efficacy in applying them for real-life authentication scenarios.

Drug abuse leads to high degree of the change in the facial structure of a person due to multiple factors which include loss of fat in face, change of facial structure due to changes in facial muscles and change of skin texture due to appearance of acnes. The combination of such effects lead to completely deformed face which presents a complex challenge to face based authentication systems. To study the impact of the drug-abuse on face recognition systems, in this work, we create a new face database collected before and after the drug-abuse. The newly collected database which is referred as Drug Abuse Database (DAD) consists of images obtained from 101 subjects. Further, the collected database is analysed using various state-of-art face recognition algorithms along with a widely used commercial-off-the-shelf (COTS) system. The obtained performance of GMR = 12.90% at FMR = 0.01% indicates the challenge presented by such face images and underlines the importance of newer algorithms to handle such challenge.

The challenges of presentation attacks (spoofing attacks) at sensor level is increasing for biometric systems due to the evolving method of artefact presentation. The sophisticated attacks now employ high quality printed artefacts and electronic screens to present the biometric samples which make it difficult to separate the real presentations and artefact presentations. In this work, we propose a new scheme to detect the artefacts in both NIR and visible spectrum biometric sensors for ocular biometric systems using a new set of feature descriptor. The scheme employs adaptive and quantized texture patters obtained from local microfeatures and global spatial features for different color channels in an image. Further, the texture descriptors are used to learn a spectrally regressed discriminant classifier to classify the normal ocular images against the artefact ocular images. The proposed scheme is used to perform extensive experiments on 5 publicly available ocular datasets including 2 datasets acquired in NIR domain and two datasets acquired using smartphones along with a dataset acquired using high quality camera. The experiments conducted on all the datasets have consistently indicated the robust performance against attacks by showing a classification error of 0%.

A light field sensor can provide useful information in terms of multiple depth (or focus) images, holding additional information that is quite useful for biometric applications. In this paper, we examine the applicability of a light field camera for biometric applications by considering two prominently used biometric characteristics: 1) face and 2) iris. To this extent, we employed a Lytro light field camera to construct two new and relatively large scale databases, for both face and iris biometrics. We then explore the additional information available from different depth images, which are rendered by light field camera, in two different manners: 1) by selecting the best focus image from the set of depth images and 2) combining all the depth images using super-resolution schemes to exploit the supplementary information available within the set elements. Extensive evaluations are carried out on our newly constructed database, demonstrating the significance of using additional information rendered by a light field camera to improve the overall performance of the biometric system.

The gaining popularity of the visible spectrum iris recognition has sparked the interest in adopting it for various access control applications. Along with the popularity of visible spectrum iris recognition comes the threat of identity spoofing, presentation, or direct attack. This paper presents a novel scheme for detecting video presentation attacks in visible spectrum iris recognition system by magnifying the phase information in the eye region of the subject. The proposed scheme employs modified Eulerian video magnification (EVM) to enhance the subtle phase information in eye region and novel decision module to classify it as artefact(spoof attack) or normal presentation. The proposed decision module is based on estimating the change of phase information obtained from EVM, specially tailored to detect presentation attacks on video-based iris recognition systems in visible spectrum. The proposed scheme is extensively evaluated on the newly constructed database consisting of 62 unique iris video acquired using two smartphones-iPhone 5S and Nokia Lumia 1020. We also construct the artefact database with 62 iris acquired by replaying normal presentation iris video on iPad with retina display. Extensive evaluation of proposed presentation attack detection (PAD) scheme on the newly constructed database has shown an outstanding performance of average classification error rate = 0% supporting the robustness of the proposed PAD scheme.

The vulnerability of face recognition systems isa growing concern that has drawn the interest from both academic and research communities. Despite the availability of a broad range of face presentation attack detection (PAD)(or countermeasure or antispoofing) schemes, there exists no superior PAD technique due to evolution of sophisticated presentation attacks (or spoof attacks). In this paper, we present a new perspective for face presentation attack detection by introducing light field camera (LFC). Since the use of a LFC can record the direction of each incoming ray in addition to the intensity, it exhibits an unique characteristic of rendering multiple depth(or focus) images in a single capture. Thus, we present a novel approach that involves exploring the variation of the focus between multiple depth (or focus) images rendered by the LFC that in turn can be used to reveal the presentation attacks. To this extent, we first collect a new face artefact database using LFC that comprises of 80 subjects. Face artefacts are generated by simulating two widely used attacks, such as photo print and electronic screen attack. Extensive experiments carried out on the light field face artefact database have revealed the outstanding performance of the proposed PAD scheme when benchmarked with various well established state-of-the-art schemes.

Multimodal biometric systems based on fingerprint and finger vein modality provide promising features useful for robust and reliable identity verification. In this paper, we present a robust imaging device that can capture both fingerprint and finger vein simultaneously. The presented low-cost sensor employs a single camera followed by both near infrared and visible light sources organized along with the physical structures to capture good quality finger vein and fingerprint samples. We further present a novel finger vein recognition algorithm that explores both the maximum curvature method and Spectral Minutiae Representation (SMR). Extensive experiments are carried out on our newly collected database that comprises of 1500 samples of fingerprint and finger vein from 150 unique fingers corresponding to 41 subjects. Our results demonstrate the efficacy of the proposed sensor with a lowest Equal Error Rate of 0.78%.

The face quality assessment from video must quantitatively measure the applicability of the face images that are typically captured over multiple frames with various degradations. In this work, we address the face quality assessment from the video captured using Automatic Border Control (ABC) system. To this extent, we employed MorphoWayTM ABC system as a data capture device to construct a new database by simulating real-life scenario. We then propose a new scheme for face quality estimation that can be viewed in three steps: (1) Pose estimation by detecting face parts (eyes and nose) to separate frontal from non-frontal faces. (2) We then consider the frontal face and evaluate its corresponding image quality by analyzing its texture components using Grey Level Co-occurrence Matrix (GLCM). (3) Finally, we quantify the quality of the given face image using likelihood values obtained using Gaussian Mixture Model (GMM). Extensive experiments are carried out on our new database that exhibits various quality degradations due to head pose variations, change in illumination, expression, motion blur, etc. The experimental results have indicated that the proposed face quality assessment algorithm can effectively classify the input image into relevant quality bins that in turn can be employed for the improved face verification.

The advanced technologies and sensors in smartphones has led to showcase their potential as a biometric sensor. In this work, we present the feasibility study and challenges in the path forward for using smartphone as a biometric sensor for iris recognition in visible spectrum. Especially, with a limited shelf-life of smartphones, it is anticipated to have enrolment and verification using different camera. In this work, we propose an improvement to segmentation scheme for contactless iris acquisition by approximating the radius range. The proposed method has resulted in a segmentation accuracy of 81%. We also propose various protocols for real-life verification scenarios using smartphones for visible spectrum iris recognition. Finally, results from an extensive set of experiments are presented to validate the anticipated challenges in using smartphone based iris recognition. Being the first of its kind, this work provides the benchmarking results for the smartphone iris database. The best EER is obtained for iPhone in indoor scenario with an impressive EER of 0.48%.

In this paper, we present an empirical study on exploring the Light Field Camera (LFC) for identifying multiple faces present at different distance. Since LFC can render multiple focus images in single exposure, one can combine these multiple images to obtain single all-in-focus image. Thus the constructed all-in-focus image will have all regions in focus and hence allows one to capture more information about the subject present even at a far distance. At the same time one can also construct the super resolution image to further improve the face recognition at a distance. Thus, in this work, we explore both all-in-focus and super resolution schemes to evaluate the multiple face recognition at a distance using LFC. We carry out extensive experiments on light field face dataset and present both qualitative and quantitative results.