Custom Diffusion Models (CDMs) offer impressive capabilities for personalization in generative modeling, yet they remain vulnerable to catastrophic forgetting when learning new concepts sequentially. Existing approaches primarily focus on minimizing interference between concepts, often neglecting the potential for positive inter-concept interactions. In this work, we present Forget Less by Learning from Parents (FLLP), a novel framework that introduces a parent-child inter-concept learning mechanism in hyperbolic space to mitigate forgetting. By embedding concept representations within a Lorentzian manifold, naturally suited to modeling tree-like hierarchies, we define parent-child relationships in which previously learned concepts serve as guidance for adapting to new ones. Our method not only preserves prior knowledge but also supports continual integration of new concepts. We validate FLLP on three public datasets and one synthetic benchmark, showing consistent improvements in both robustness and generalization.

Custom Diffusion Models (CDMs) have gained significant attention due to their remarkable ability to personalize generative processes. However, existing CDMs suffer from catastrophic forgetting when continuously learning new concepts. Most prior works attempt to mitigate this issue under the sequential learning setting with a fixed order of concept inflow and neglect inter-concept interactions. In this paper, we propose a novel framework -Forget Less by Learning Together (FL2T) - that enables concurrent and order-agnostic concept learning while addressing catastrophic forgetting. Specifically, we introduce a set-invariant inter-concept learning module where proxies guide feature selection across concepts, facilitating improved knowledge retention and transfer. By leveraging inter-concept guidance, our approach preserves old concepts while efficiently incorporating new ones. Extensive experiments, across three datasets, demonstrates that our method significantly improves concept retention and mitigates catastrophic forgetting, highlighting the effectiveness of inter-concept catalytic behavior in incremental concept learning of ten tasks with at least 2% gain on average CLIP Image Alignment scores.

Temporal action segmentation is a critical task in video understanding, where the goal is to assign action labels to each frame in a video. While recent advances leverage iterative refinement-based strategies, they fail to explicitly utilize the hierarchical nature of human actions. In this work, we propose HybridTAS - a novel framework that incorporates a hybrid of Euclidean and hyperbolic geometries into the denoising process of diffusion models to exploit the hierarchical structure of actions. Hyperbolic geometry naturally provides tree-like relationships between embeddings, enabling us to guide the action label denoising process in a coarse-to-fine manner: higher diffusion timesteps are influenced by abstract, high-level action categories (root nodes), while lower timesteps are refined using fine-grained action classes (leaf nodes). Extensive experiments on three benchmark datasets, GTEA, 50Salads, and Breakfast, demonstrate that our method achieves state-of-the-art performance, validating the effectiveness of hyperbolic-guided denoising for the temporal action segmentation task.

In today’s data-driven analytics landscape, deep learning has become a powerful tool, with latent representations, known as embeddings, playing a central role in several applications. In the face analytics domain, such embeddings are commonly used for biometric recognition (e.g., face identification). However, these embeddings, or templates, can inadvertently expose sensitive attributes such as age, gender, and ethnicity. Leaking such information can compromise personal privacy and affect civil liberty and human rights. To address these concerns, we introduce a multi-layer protection framework for embeddings. It consists of a sequence of operations: (a) encrypting embeddings using Fully Homomorphic Encryption (FHE), and (b) hashing them using irreversible feature manifold hashing. Unlike conventional encryption methods, FHE enables computations directly on encrypted data, allowing downstream analytics while maintaining strong privacy guarantees. To reduce the overhead of encrypted processing, we employ embedding compression. Our proposed method shields latent representations of sensitive data from leaking private attributes (such as age and gender) while retaining essential functional capabilities (such as face identification). Extensive experiments on two datasets using two face encoders demonstrate that our approach outperforms several state-of-the-art privacy protection methods.

The high performance of current deep face recognition systems and their unauthorized usage have raised a severe concern for privacy in the physical, adversarial, and digital domains. To protect privacy, users are exploring several ways, and one such method that recently gained attention is individuals deliberately obscuring their faces with their hands, presumably to avoid facial recognition technology. Since deep face recognition algorithms can handle partial tampering of faces, this raises a critical question of whether these deliberate attempts can protect privacy. In the literature, no evaluation exists that showcases that this type of hiding can bypass the face recognition algorithms. Therefore, in this first-ever study, we have performed extensive research by first developing multiple nose and mouth occlusion datasets using synthetic patches and real-life objects. Our extensive experimentation reveals several interesting observations reflecting the fact that even when a patch is a face patch extracted from an unseen subject, it can fool the face recognition networks. Further, not only face recognition networks, but also it is observed that the proposed patches are effective in deceiving the soft biometric classifier, i.e., the classifier detecting the gender and ethnicity of individuals.

Deepfake detection research has seen tremendous success and has achieved remarkably high performance on a few existing datasets. However, the significant drawback of the existing works is the generalizability of the detection algorithms under cross-datasets and cross-attack/manipulation settings. On top of that, another critical bottleneck of deepfake detection literature is the understanding of the fairness quotient of these algorithms. One big reason for such a less explored domain is the unavailability of deep fake datasets covering multiple ethnicities and genders with proper annotations. For example, the popular deepfake detection datasets such as FaceForensics++ and Celeb-DF are highly biased toward Caucasian ethnicity. Recently, a multi-ethnicity multi-modal dataset namely FakeAVCeleb has been released which can fulfill this gap. Henceforth by utilizing the potential of this dataset, we have performed the fairness study of deepfake detection algorithms. For that, several image classifiers are selected which range from deep convolutional neural networks to handcrafted image feature extraction to vision transformers. The experiments performed using such a wide variety of classifiers reveal that the deepfake detectors are not fair and can detect one ethnicity with high accuracy but fail miserably on others. For instance, the performance of one of the popular deepfake detection networks namely XceptionNet shows a reduction of more than 30% when dealing with different ethnicities and genders. Not only ethnicity or gender but also the type of classifiers have a huge impact on the performance. We assert that the proposed study can help in building a fair, robust, and accurate deepfake classifier utilizing insightful findings that can help in the selection of an effective and robust backbone architecture.

Modern face recognition systems utilize deep neural networks to extract salient features from a face. These features denote embeddings in latent space and are often stored as templates in a face recognition system. These embeddings are susceptible to data leakage and, in some cases, can even be used to reconstruct the original face image. To prevent compromising identities, template protection schemes are commonly employed. However, these schemes may still not prevent the leakage of soft biometric information such as age, gender and race. To alleviate this issue, we propose a novel technique that combines Fully Homomorphic Encryption (FHE) with an existing template protection scheme known as PolyProtect. We show that the embeddings can be compressed and encrypted using FHE and transformed into a secure PolyProtect template using polynomial transformation, for additional protection. We demonstrate the efficacy of the proposed approach through extensive experiments on multiple datasets. Our proposed approach ensures irreversibility and unlinkability, effectively preventing the leakage of soft biometric attributes from face embeddings without compromising recognition accuracy.

Face being an active medium of communication is a significant part of our social media life; however, faces are vulnerable to manipulations. Among various manipulations, face morphing is a well-known tampering technique that aims to generate images containing information from more than one identity. Morphed images are heavily used for various malicious purposes including sarcasm, money laundering, and pornography. For many of the above harmful purposes, these manipulated images are uploaded on social media platforms where they can further go through tampering using social-media filters. Interestingly, the existing morph attack detection works have not addressed social media’s impact on deceiving face morph detectors. In this research, for the first time, we have generated authentic (or real) and face-morphed images impacted by one of the premium features of social media platforms known as filtering. We have used 13 Instagram filters and performed an extensive study on the proposed social-media morphed dataset. It is demonstrated that these filters can radically reduce the morph detection performances of several popular deep-learning classifiers. Therefore, to effectively address the concerns of face morphing and social media filtering, we propose a robust ViT-CNN architecture to advance the morph image detection performance.

One of the critical steps in biometrics pipeline is detection of presentation attacks, a physical adversary. Several presentation (adversary) attack detection (PAD) algorithms, including iris PAD, have been proposed and have shown superlative performance. However, a recent study, on a small-scale database, has highlighted that iris PAD may have gender biases. In this research, we present a rigorous study on gender bias in iris presentation attack detection algorithms using a large-scale and gender-balanced database. The paper provides several interesting observations which can help in building future presentation attack detection algorithms with aim of fair treatment of each demography. In addition, we also present a robust iris presentation attack detection algorithm by combining gender-covariate based classifiers. The proposed robust classifier not only reduces the difference in accuracy between different genders but also improves the overall performance of the PAD system.

Contactless fingerprint matching using smartphone cameras can alleviate major challenges of traditional fingerprint systems including hygienic acquisition, portability and presentation attacks. However, development of practical and robust contactless fingerprint matching techniques is constrained by the limited availablity of large scale real-world datasets. To motivate further advances in contactless fingerprint matching across sensors, we introduce the RidgeBase benchmark dataset. RidgeBase consists of more than 15,000 contactless and contact-based fingerprint image pairs acquired from 88 individuals under different background and lighting conditions using two smartphone cameras and one flatbed contact sensor. Unlike existing datasets, RidgeBase is designed to promote research under different matching scenarios that include Single Finger Matching and Multi-Finger Matching for both contactless-to-contactless (CL2CL) and contact-to-contactless (C2CL) verification and identification. Furthermore, due to the high intra-sample variance in contactless fingerprints belonging to the same finger, we propose a set-based matching protocol inspired by the advances in facial recognition datasets. This protocol is specifically designed for pragmatic contactless fingerprint matching that can account for variances in focus, polarity and finger-angles. We report qualitative and quantitative baseline results for different protocols using a COTS fingerprint matcher (Verifinger) and a Deep CNN based approach on the RidgeBase dataset. The dataset can be downloaded here: https://www.buffalo.edu/cubs/research/datasets/ridgebase-benchmark-dataset.html

This paper proposes a non-interactive end-to-end solution for secure fusion and matching of biometric templates using fully homomorphic encryption (FHE). Given a pair of encrypted feature vectors, we perform the following ciphertext operations, i) feature concatenation, ii) fusion and dimensionality reduction through a learned linear projection, iii) scale normalization to unit ℓ2-norm, and iv) match score computation. Our method, dubbed HEFT (Homomorphi-cally Encrypted Fusion of biometric Templates), is custom-designed to overcome the unique constraint imposed by FHE, namely the lack of support for non-arithmetic operations. From an inference perspective, we systematically explore different data packing schemes for computationally efficient linear projection and introduce a polynomial approximation for scale normalization. From a training perspective, we introduce an FHE-aware algorithm for learning the linear projection matrix to mitigate errors induced by approximate normalization. Experimental evaluation for template fusion and matching of face and voice biometrics shows that HEFT (i) improves biometric verification performance by 11.07% and 9.58% AUROC compared to the respective unibiometric representations while compressing the feature vectors by a factor of 16 (512D to 32D), and (ii) fuses a pair of encrypted feature vectors and computes its match score against a gallery of size 1024 in 884 ms. Code and data are available at https://github.com/humananalysis/encrypted-biometric-fusion

The vulnerability of iris recognition algorithms against presentation attacks demands a robust defense mechanism. Much research has been done in the literature to create a robust attack detection algorithm; however, most of the algorithms suffer from generalizability, such as inter database testing or unseen attack type. The problem of attack detection can further be exacerbated if the images contain noise such as Gaussian or Salt-Pepper noise. In this research, we propose a multi-task deep learning model with a denoising convolutional skip autoencoder and a classifier to inbuilt robustness against noisy images. The Gaussian noise layer is introduced as a dropout between the encoder network’s hidden layers, which helps the model learn generalized features that are robust to data noise. The proposed algorithm is evaluated on multiple presentation attack databases and extensive experiments across different noise types and a comparison with other deep learning models show the generalizability and efficacy of the proposed model.

Adversarial attacks have been demonstrated to fool the deep classification networks. There are two key characteristics of these attacks: firstly, these perturbations are mostly additive noises carefully crafted from the deep neural network itself. Secondly, the noises are added to the whole image, not considering them as the combination of multiple components from which they are made. Motivated by these observations, in this research, we first study the role of various image components and the impact of these components on the classification of the images. These manipulations do not require the knowledge of the networks and external noise to function effectively and hence have the potential to be one of the most practical options for real-world attacks. Based on the significance of the particular image components, we also propose a transferable adversarial attack against unseen deep networks. The proposed attack utilizes the projected gradient descent strategy to add the adversarial perturbation to the manipulated component image. The experiments are conducted on a wide range of networks and four databases including ImageNet and CIFAR-100. The experiments show that the proposed attack achieved better transferability and hence gives an upper hand to an attacker. On the ImageNet database, the success rate of the proposed attack is up to 88.5%, while the current state-of-the-art attack success rate on the database is 53.8%. We have further tested the resiliency of the attack against one of the most successful defenses namely adversarial training to measure its strength. The comparison with several challenging attacks shows that: (i) the proposed attack has a higher transferability rate against multiple unseen networks and (ii) it is hard to mitigate its impact. We claim that based on the understanding of the image components, the proposed research has been able to identify a newer adversarial attack unseen so far and unsolvable using the current defense mechanisms.

Adversarial perturbations have demonstrated the vulnerabilities of deep learning algorithms to adversarial attacks. Existing adversary detection algorithms attempt to detect the singularities; however, they are in general, loss-function, database, or model dependent. To mitigate this limitation, we propose DAMAD-a generalized perturbation detection algorithm which is agnostic to model architecture, training data set, and loss function used during training. The proposed adversarial perturbation detection algorithm is based on the fusion of autoencoder embedding and statistical texture features extracted from convolutional neural networks. The performance of DAMAD is evaluated on the challenging scenarios of cross-database, cross-attack, and cross-architecture training and testing along with traditional evaluation of testing on the same database with known attack and model. Comparison with state-of-the-art perturbation detection algorithms showcase the effectiveness of the proposed algorithm on six databases: ImageNet, CIFAR-10, Multi-PIE, MEDS, point and shoot challenge (PaSC), and MNIST. Performance evaluation with nearly a quarter of a million adversarial and original images and comparison with recent algorithms show the effectiveness of the proposed algorithm.

During this unprecedented time of the COVID19 pandemic, wearing face masks has become a necessity. While these masks aim to secure an individual from getting infected by any kind of viruses including COVID-19; they significantly obfuscate the identity. The situation becomes even worse when an attacker performs a crime and the place does not have any surveillance cameras. The identification of criminals in such conditions highly depends on the witnesses and generation of sketches based on their description. To the best of our knowledge, in the literature, no work has been performed for matching sketch images with masks. In this research, we have first created the mask sketch face database using more than 50 identities. The sketch images are generated using a different variant of pencils, which can be seen as different sketch artists. The recognition experiments are performed using state-of-the-art face embedding networks including ArcFace and DeepID which show that the recognition performance degrades significantly when the sketch mask images are used for identification. In another set of experiments, it is observed that the recognition algorithm is robust in handling the digital face mask images. However, the ineffectiveness in handling the variations that occurred due to sketches is a serious concern and needs attention.

Deep neural networks are generally trained using large amounts of data to achieve state-of-the-art accuracy in many possible computer vision and image analysis applications ranging from object recognition to natural language processing. It is also claimed that these networks can memorize the data which can be extracted from the network parameters such as weights and gradient information. The adversarial vulnerability of the deep networks is usually evaluated on the unseen test set of the databases. If the network is memorizing the data, then the small perturbation in the training image data should not drastically change its performance. Based on this assumption, we first evaluate the robustness of deep neural networks on small perturbations added in the training images used for learning the parameters of the network. It is observed that, even if the network has seen the images it is still vulnerable to these small perturbations. Further, we propose a novel data augmentation technique to increase the robustness of deep neural networks to such perturbations.

Data privacy is a growing concern as more cloud-based solutions for extracting insights from images are available. Fully Homomorphic Encryption (FHE) is one of the state-of-the-art techniques to enable privacy preserving machine learning. However, encrypted deep neural network-based inference methods face the fundamental challenge of optimizing computational depth and complexity to achieve an acceptable trade-off between accuracy and practical feasibility. Existing works only report high level implementations without providing rigorous analysis of the intricacies involved in the FHE implementation of generic primitive operators of a convolutional neural network (CNN). In this paper, we use the CKKS encryption scheme available in the open-source HElib library to run encrypted inference experiments on the MNIST dataset. The experiments indicate that efficient ciphertext packing schemes, model optimization and multi-threading strategies play a critical role in determining the throughput and latency of the inference process. We also show that operational parameters of the chosen FHE scheme such as the degree of the cyclotomic polynomial, depth limitations of the underlying leveled HE scheme, and the computational precision parameters result in significant trade-offs between accuracy, security level and computational time of the machine learning model. The key contribution of the paper is the analysis and recommendation of optimization techniques for efficient encrypted CNN inference.

Deep learning algorithms provide state-of-the-art results on a multitude of applications. However, it is also well established that they are highly vulnerable to adversarial perturbations. It is often believed that the solution to this vulnerability of deep learning systems must come from deep networks only. Contrary to this common understanding, in this article, we propose a non-deep learning approach that searches over a set of well-known image transforms such as Discrete Wavelet Transform and Discrete Sine Transform, and classifying the features with a support vector machine-based classifier. Existing deep networks-based defense have been proven ineffective against sophisticated adversaries, whereas image transformation-based solution makes a strong defense because of the non-differential nature, multiscale, and orientation filtering. The proposed approach, which combines the outputs of two transforms, efficiently generalizes across databases as well as different unseen attacks and combinations of both (i.e., cross-database and unseen noise generation CNN model). The proposed algorithm is evaluated on large scale databases, including object database (validation set of ImageNet) and face recognition (MBGC) database. The proposed detection algorithm yields at-least 84.2% and 80.1% detection accuracy under seen and unseen database test settings, respectively. Besides, we also show how the impact of the adversarial perturbation can be neutralized using a wavelet decomposition-based filtering method of denoising. The mitigation results with different perturbation methods on several image databases demonstrate the effectiveness of the proposed method.

While significant advancements have been made in the field of multimedia forensics to detect altered content, existing techniques mostly focus on enabling the content recipient to verify the content integrity without any inputs from the content creator. In many application scenarios, the creator has a strong incentive to establish the provenance and integrity of the multimedia data created and released by him. Hence, there is a strong need for mechanisms that allow the content creator to prove the authenticity of the released content. Since blockchain technology provides an immutable distributed database, it is an ideal solution for reliably time-stamping content with its creation time and storing an irrefutable signature of the content at the time of its creation. However, a simple digital signature scheme does not allow modification of the content after the initial commitment. Authorized multimedia content alteration by its creator is often necessary (e.g., redaction of faces to protect the privacy of individuals in a video, redaction of sensitive fields in a text document) before the content is distributed. The main contributions of this paper are: (i) a novel sanitizable signature scheme that enables the content creator to prove the integrity of the redacted content, while preventing the recipients from reconstructing the redacted segments based on the published commitment, and (ii) a blockchain-based solution for securely managing the sanitizable signature. The proposed solution employs a robust hashing scheme using chameleon hash function and Merkle tree to generate the initial signature, which is stored on the blockchain. The auxiliary data required for the integrity verification step is retained by the content creator and only a signature of this auxiliary data is stored on the blockchain. Any modifications to the multimedia content requires only updating the signature of the auxiliary data, which is securely recorded on the blockchain. We demonstrate that the proposed approach enables verification of integrity of redacted multimedia content without compromising the content privacy requirements.

Deep neural network (DNN) architecture based models have high expressive power and learning capacity. However, they are essentially a black box method since it is not easy to mathematically formulate the functions that are learned within its many layers of representation. Realizing this, many researchers have started to design methods to exploit the drawbacks of deep learning based algorithms questioning their robustness and exposing their singularities. In this paper, we attempt to unravel three aspects related to the robustness of DNNs for face recognition: (i) assessing the impact of deep architectures for face recognition in terms of vulnerabilities to attacks inspired by commonly observed distortions in the real world that are well handled by shallow learning methods along with learning based adversaries; (ii) detecting the singularities by characterizing abnormal filter response behavior in the hidden layers of deep networks; and (iii) making corrections to the processing pipeline to alleviate the problem. Our experimental evaluation using multiple open-source DNN-based face recognition networks, including OpenFace and VGG-Face, and two publicly available databases (MEDS and PaSC) demonstrates that the performance of deep learning based face recognition algorithms can suffer greatly in the presence of such distortions. The proposed method is also compared with existing detection algorithms and the results show that it is able to detect the attacks with very high accuracy by suitably designing a classifier using the response of the hidden layers in the network. Finally, we present several effective countermeasures to mitigate the impact of adversarial attacks and improve the overall robustness of DNN-based face recognition.

A personal or enterprise collection of a large set of face images may contain many types of tags used for querying the collection. Often the tags have many irrelevant content that may not reflect the image content in terms of the facial characteristics. In this paper, we propose a data curation method to filter out the irrelevant face images using a face recognition based subgraph identification. Results on retrievals from the Internet using popular celebrities show the efficacy of our approach after we cleanse the images collection retrieved and applying our algorithm to the collection.

Classifier fusion is a well-studied problem in which decisions from multiple classifiers are combined at the score, rank, or decision level to obtain better results than a single classifier. Subsequently, various techniques for combining classifiers at each of these levels have been proposed in the literature. Many popular methods entail scaling and normalizing the scores obtained by each classifier to a common numerical range before combining the normalized scores using the sum rule or another classifier. In this research, we explore an alternative method to combine classifiers at the score level. The Pool Adjacent Violators (PAV) algorithm has traditionally been utilized to convert classifier match scores to confidence values that model posterior probabilities for test data. The PAV algorithm and other score normalization techniques have studied the same problem without being aware of each other. In this first ever study to combine the two, we propose the PAV algorithm for classifier fusion on publicly available NIST multi-modal biometrics score dataset. We observe that it provides several advantages over existing techniques and find that the interpretation learned by the PAV algorithm is more robust than the scaling learned by other popular normalization algorithms such as min-max. Moreover, the PAV algorithm enables the combined score to be interpreted as confidence and is able to further improve the results obtained by other approaches. We also observe that utilizing traditional normalization techniques first for individual classifiers and then normalizing the fused score using PAV offers a performance boost compared to only using the PAV algorithm.

Often deep learning methods are associated with huge amounts of training data. The deeper the network gets, the larger is the need for training data. A large amount of labeled data helps the network learn about the variations it needs to handle in the prediction stage. It is not easy for everyone to get access to huge amounts of labeled data leaving a few to have the luxury to design very deep networks. In this paper, we propose to flatten the disparity by using the modeling methods to minimize the need for huge amounts of data for training a deep network. Using face recognition as an example, we demonstrate how limited labeled data can be leveraged to obtain near state of the art performance with generalization capability across multiple databases. In addition, we show that the normalization in the overall network can improve the speed and resource requirement for the prediction/inferencing stage.

It has been shown that significant age difference between a probe and gallery face image can decrease the matching accuracy. If the face images can be normalized in age, there can be a huge impact on the face verification accuracy and thus many novel applications such as matching driver's license, passport and visa images with the real person's images can be effectively implemented. Face progression can address this issue by generating a face image for a specific age. Many researchers have attempted to address this problem focusing on predicting older faces from a younger face. In this paper, we propose a novel method for robust and automatic face progression in totally unconstrained conditions. Our method takes into account that faces belonging to the same age-groups share age patterns such as wrinkles while faces across different age-groups share some common patterns such as expressions and skin colors. Given training images of K different age-groups the proposed method learns to recover K low-rank age and one low-rank common components. These extracted components from the learning phase are used to progress an input face to younger as well as older ages in bidirectional fashion. Using standard datasets, we demonstrate that the proposed progression method outperforms state-of-the-art age progression methods and also improves matching accuracy in a face verification protocol that includes age progression.

Biometric systems can be attacked in several ways and the most common being spoofing the input sensor. Therefore, anti-spoofing is one of the most essential prerequisite against attacks on biometric systems. For face recognition it is even more vulnerable as the image capture is non-contact based. Several anti-spoofing methods have been proposed in the literature for both contact and non-contact based biometric modalities often using video to study the temporal characteristics of a real vs. spoofed biometric signal. This paper presents a novel multi-feature evidence aggregation method for face spoofing detection. The proposed method fuses evidence from features encoding of both texture and motion (liveness) properties in the face and also the surrounding scene regions. The feature extraction algorithms are based on a configuration of local binary pattern and motion estimation using histogram of oriented optical flow. Furthermore, the multi-feature windowed videolet aggregation of these orthogonal features coupled with support vector machine-based classification provides robustness to different attacks. We demonstrate the efficacy of the proposed approach by evaluating on three standard public databases: CASIA-FASD, 3DMAD and MSU-MFSD with equal error rate of 3.14%, 0%, and 0%, respectively.

User authentication in the context of a secure transaction needs to be continuously evaluated for the risks associated with the transaction authorization. The situation becomes even more critical when there are regulatory compliance requirements. Need for such systems have grown dramatically with the introduction of smart mobile devices which make it far easier for the user to complete such transaction quickly but with a huge exposure to risk. Biometrics can play a very significant role in addressing such problems as a key indicator of the user identity and thus reducing the risk of fraud. While unimodal biometrics authentication systems are being increasingly experimented by mainstream mobile system manufacturers (e.g., fingerprint in iOS), we explore various opportunities of reducing risk in a multimodal biometrics system. The multimodal system is based on fusion of several biometrics combined with a policy manager. A new biometric modality: chirography which is based on user writing on multi-touch screens using their finger is introduced. Coupling with chirography, we also use two other biometrics: face and voice. Our fusion strategy is based on inter-modality score level fusion that takes into account a voice quality measure. The proposed system has been evaluated on an in-house database that reflects the latest smart mobile devices. On this database, we demonstrate a very high accuracy multi-modal authentication system reaching an EER of 0.1% in an office environment and an EER of 0.5% in challenging noisy environments.

Face recognition algorithms are generally trained for matching high-resolution images and they perform well for similar resolution test data. However, the performance of such systems degrades when a low-resolution face image captured in unconstrained settings, such as videos from cameras in a surveillance scenario, are matched with high-resolution gallery images. The primary challenge, here, is to extract discriminating features from limited biometric content in low-resolution images and match it to information rich high-resolution face images. The problem of cross-resolution face matching is further alleviated when there is limited labeled positive data for training face recognition algorithms. In this paper, the problem of cross-resolution face matching is addressed where low-resolution images are matched with high-resolution gallery. A co-transfer learning framework is proposed, which is a cross-pollination of transfer learning and co-training paradigms and is applied for cross-resolution face matching. The transfer learning component transfers the knowledge that is learnt while matching high-resolution face images during training to match low-resolution probe images with high-resolution gallery during testing. On the other hand, co-training component facilitates this transfer of knowledge by assigning pseudolabels to unlabeled probe instances in the target domain. Amalgamation of these two paradigms in the proposed ensemble framework enhances the performance of cross-resolution face recognition. Experiments on multiple face databases show the efficacy of the proposed algorithm and compare with some existing algorithms and a commercial system. In addition, several high profile real-world cases have been used to demonstrate the usefulness of the proposed approach in addressing the tough challenges.

Iris recognition has gained popularity due to factors such as its perceived high accuracy, significant usability advantages attributed to its non-contact acquisition method, and the availability of low cost sensors due to improvements in technology. However, non-contact biometrics authentication systems are vulnerable to different types of attacks than contact-type biometrics, such as fingerprints, for which there are a number of simple techniques to guard against attacks. In particular, the fashion industry has developed designer contact lenses with patterns that range from a simple change in eye color to the imposition of stars or other festive decorations. As these lenses are readily available and can be personalized at a very affordable price, their use in thwarting or spoofing iris-based authentication systems becomes plausible. Given the high security nature of many of these systems, there is a urgent need for a some countermeasure to this type of attack. In this paper, we describe a novel method to detect the presence of fake iris patterns, such as designer contact lenses, during the image acquisition stage to further enhance the basic security value of iris biometrics. Exploiting the anatomy and geometry of the human eye, we present a structured light projection method to detect the presence of artificial items obscuring the real iris. The detection principle has been verified using an inexpensive experimental setup consisting of a miniature projector and an offset camera. We also describe a novel algorithm to process the acquired images to find patterned contact lenses, and measure its performance using data collected with our apparatus. We argue that the addition of the proposed system and algorithm to existing iris biometrics based authentication systems will significantly improve their security.

Face recognition systems, trained in controlled environment, often fail to efficiently match low resolution images with high resolution images. In this research, a co-transfer learning framework is proposed in which knowledge learnt in controlled high resolution environment is transferred for matching low resolution probe images with high resolution gallery. The proposed framework seamlessly combines transfer learning and co-training to perform knowledge transfer by updating classifier's decision boundary with low resolution probe instances. Experiments are performed on the CMU-Multi-PIE and SCface database with gallery images of size 72 × 72 and size of probe images varying from 48 × 48 to 16 × 16. The results show that, in terms of rank-1 identification accuracy, the proposed algorithm outperforms existing approaches by at least 5%.

In this study, we propose a novel biometric signature for human identification based on anatomically unique structures of the left ventricle of the heart. An algorithm is developed that analyzes the 3 primary anatomical structures of the left ventricle: the endocardium, myocardium, and papillary muscles. Comparisons of these analyses between probe and gallery images produces a similarity score that is used as the basis of the biometric. The performance of the algorithm is tested on a cohort of 10 de-identified subjects imaged by Cardiac MRI. Perfect matching between individuals is obtained with good separation between the genuine and impostor classes. In summary, this study demonstrates using anatomy of the left ventricle of the human heart for the purposes of a biometric signature.

Noncontact biometrics such as face and iris have additional benefits over contact-based biometrics such as fingerprint and hand geometry. However, three important challenges need to be addressed in a noncontact biometrics-based authentication system: ability to handle unconstrained acquisition, robust and accurate matching, and privacy enhancement without compromising security. In this paper, we propose a unified framework based on random projections and sparse representations, that can simultaneously address all three issues mentioned above in relation to iris biometrics. Our proposed quality measure can handle segmentation errors and a wide variety of possible artifacts during iris acquisition. We demonstrate how the proposed approach can be easily extended to handle alignment variations and recognition from iris videos, resulting in a robust and accurate system. The proposed approach includes enhancements to privacy and security by providing ways to create cancelable iris templates. Results on public data sets show significant benefits of the proposed approach.

Privacy and security are essential requirements in practical biometric systems. In order to prevent the theft of biometric patterns, it is desired to modify them through revocable and non invertible transformations called Cancelable Biometrics. In this paper, we propose an efficient algorithm for generating a Cancelable Iris Biometric based on Sectored Random Projections. Our algorithm can generate a new pattern if the existing one is stolen, retain the original recognition performance and prevent extraction of useful information from the transformed patterns. Our method also addresses some of the drawbacks of existing techniques and is robust to degradations due to eyelids and eyelashes.

While biometrics-based identification is a key technology in many critical applications such as searching for an identity in a watch list or checking for duplicates in a citizen ID card system, there are many technical challenges in building a solution because the size of the database can be very large (often in 100s of millions) and the intrinsic errors with the underlying biometrics engines. Often multi-modal biometrics is proposed as a way to improve the underlying biometrics accuracy performance. In this paper, we propose a score based fusion scheme tailored for identification applications. The proposed algorithm uses a gradient descent method to learn weights for each modality such that weighted sum of genuine scores is larger than the weighted sum of all the impostor scores. During the identification phase, top K candidates from each modality are retrieved and a super-set of identities is constructed. Using the learnt weights, we compute the weighted score for all the candidates in the superset. The highest scoring candidate is declared as the top candidate for identification. The proposed algorithm has been tested using NIST BSSR-1 dataset and results in terms of accuracy as well as the speed (execution time) are shown to be far superior than the published results on this dataset.

Biometric matching decisions have traditionally been made based solely on a score that represents the similarity of the query biometric to the enrolled biometric(s) of the claimed identity. Fusion schemes have been proposed to benefit from the availability of multiple biometric samples (e.g., multiple samples of the same fingerprint) or multiple different biometrics (e.g., face and fingerprint). These commonly adopted fusion approaches rarely make use of the large number of non-matching biometric samples available in the database in the form of other enrolled identities or training data. In this paper, we study the impact of combining this information with the existing fusion methodologies in a cohort analysis framework. Experimental results are provided to show the usefulness of such a cohort-based fusion of face and fingerprint biometrics.

We present a face reconstruction approach for revocable face matching. The proposed approach generates photometrically valid cancelable face images by following the image formation process. Given a face image, the approach estimates facial albedo followed by a subject-specific key based photometric deformation to generate a cancelable face image. The proposed approach allows for using any available face matcher to perform verification or recognition in the transformed domain, a capability missing from most existing works on cancelable face matching. Experiments are performed to evaluate the performance, privacy and cancelable aspects of the face images reconstructed using the approach. Results obtained are very promising and make a strong case for such backward compatible cancelable face representations that can seamlessly make use of advancements in automatic face recognition research.

Cancelable biometric systems are gaining in popularity for use in person authentication for applications where the privacy and security of biometric templates are important considerations. A variety of approaches have been proposed in the literature. In this work, we have chosen two (a registration based and a registration free) techniques and performed a comparative study focusing on template representation size, useful dataset coverage, system accuracy and transform strength. Results show that both systems have their own advantages that are suited for use in specific applications.

A person only has two irises - if his pattern is stolen he quickly runs out of alternatives. Thus methods that protect the true iris pattern need to be adopted in practical biometric applications. In particular, it is desirable to have a system that can generate a new unique pattern if the one being used is lost, or generate different unique patterns for different applications to prevent cross-matching. For backwards compatibility, these patterns should look like plausible irises so they can be handled with the same processing tools. However, they should also non-invertibly hide the true biometric so it is never exposed, or even stored. In this paper four such ldquocancelablerdquo biometric methods are proposed that work with conventional iris recognition systems, either at the unwrapped image level or at the binary iris code level.

Biometric identification has numerous advantages over conventional ID and password systems; however, the lack of anonymity and revocability of biometric templates is of concern. Several methods have been proposed to address these problems. Many of the approaches require a precise registration before matching in the anonymous domain. We introduce binary string representations of fingerprints that obviates the need for registration and can be directly matched. We describe several techniques for creating anonymous and revocable representations using these binary string representations. The match performance of these representations is evaluated using a large database of fingerprint images. We prove that given an anonymous representation, it is computationally infeasible to invert it to the original fingerprint, thereby preserving privacy. To the best of our knowledge, this is the first linear, anonymous and revocable fingerprint representation that is implicitly registered.

Biometrics-based authentication systems offer obvious usability advantages over traditional password and token-based authentication schemes. However, biometrics raises several privacy concerns. A biometric is permanently associated with a user and cannot be changed. Hence, if a biometric identifier is compromised, it is lost forever and possibly for every application where the biometric is used. Moreover, if the same biometric is used in multiple applications, a user can potentially be tracked from one application to the next by cross-matching biometric databases. In this paper, we demonstrate several methods to generate multiple cancelable identifiers from fingerprint images to overcome these problems. In essence, a user can be given as many biometric identifiers as needed by issuing a new transformation "key." The identifiers can be cancelled and replaced when compromised. We empirically compare the performance of several algorithms such as Cartesian, polar, and surface folding transformations of the minutiae positions. It is demonstrated through multiple experiments that we can achieve revocability and prevent cross-matching of biometric databases. It is also shown that the transforms are noninvertible by demonstrating that it is computationally as hard to recover the original biometric identifier from a transformed version as by randomly guessing. Based on these empirical results and a theoretical analysis we conclude that feature-level cancelable biometric construction is practicable in large biometric deployments.

The 12 regular papers and three correspondences in this special issue focus on human detection and recognition. The papers represent gait, face (3-D, 2-D, video), iris, palmprint, cardiac sounds, and vulnerability of biometrics and protection against the spoof attacks.

The fourteen papers in this special section are devoted to recent advancements in biometric systems and application devices.

Traditional fingerprint acquisition is limited to single-image capture and processing. With the advent of faster capture hardware, faster processors, and advances in video compression standards, newer systems can capture and exploit video signals for tasks that are difficult using a single image. We propose the use of fingerprint video sequences to investigate detecting two aspects of the dynamic behavior of fingerprints. Specifically, we are interested in the detection of distortion of fingerprint impressions due to excessive force and the detection of the positioning of fingers during image capture. These issues often lead to difficulties in establishing a precise match between acquired images. The proposed techniques investigate dynamic characteristics of fingerprints across video sequence frames. A significant advantage of our approach for distortion analysis is that it works directly on MPEG-1,-2 encoded fingerprint video bitstreams. The proposed methods have been tested on the NIST-24 live-scan fingerprint video database and the results are promising. We also describe a new concept called the "resultant biometrics", a new type of biometrics which has both a physiological, physical (e.g., force, torque, linear motion, rotation) component and/or a temporal characteristic, added by a subject to an existing biometric. This resultant biometric is both desirable and efficient in terms of easy modification of compromised biometrics and is harder to produce with spoof body parts.

Biometrics-based authentication and identification systems have to handle images acquired in noisy and hostile environments. The signal quality is assessed to decide if there is sufficient signal strength to process further. Poor quality signals require "enhancement" before further processing of the input signal. Often enhancement implies creating a more visibly pleasing image. However, biometrics signals need to improve the image quality for machine processability. This means that the enhancement algorithm should have some weak model about the sample (image) formation process. Enhancement is then some type of "normalization" or "beautification". We present a weak model-based image enhancement algorithm for fingerprint images. The results of the proposed algorithm are presented in terms of the improvements in the overall system performance measured in terms of a receiver operating characteristics curve.

Distortions in fingerprint images arising from the elasticity of finger skin and the pressure and movement of fingers during image capture lead to great difficulties in establishing a match between multiple images acquired from a single finger. In a single fingerprint image depicting a finger at some given instant of time, it is difficult to get any distortion information. Further, static two-dimensional or three-dimensional (electronic) copies of fingerprints can be fabricated and used to spoof remote biometric security systems since the input required by the systems is not a function of time. This paper addresses these issues, by proposing the novel use of fingerprint video sequences to investigate and exploit dynamic behaviors manifested by fingers over time during image acquisition. In particular, we present a novel approach to detect and estimate distortion occurring in fingerprint video streams. Our approach directly works on MPEG-{1, 2} encoded fingerprint video bitstreams to estimate interfield flow without decompression, and uses flow characteristics to investigate temporal behaviour of the fingerprints. The joint temporal and motion analysis leads to a novel technique to detect and characterize distortion reliably. The proposed method has been tested on the NIST 24 database and the results are very promising.

Biometrics-based authentication is becoming popular because of increasing ease-of-use and reliability. Performance evaluation of such systems is an important issue. We attempt to address two aspects of performance evaluation that have been conventionally neglected. First, the "difficulty" of the data that is used in a study influences the evaluation results. We propose some measures to characterize the data set so that the performance of a given system on different data sets can be compared. Second, conventional studies often have reported the false reject and false accept rates (FRR, FAR) in the form of match score distributions. However, no confidence intervals are computed for these distributions, hence no indication of the significance of the estimates is given. In this paper, we systematically study and compare parametric and nonparametric (bootstrap) methods for measuring confidence intervals. We give special attention to false reject rate estimates.

It is desirable to enhance the fingerprint image for achieving good fingerprint matching performance. In our earlier work, we have proposed a method for learning a set of partitioned least-squares filters from a given set of images and ground truth pairs. In this paper, we propose a refined approach to learn a class of hierarchical filter banks to extend our enhancement technique. The new technique is shown to obtain much better results particularly when the input image is not easily restored using our earlier technique. We evaluate the performance of our new approach to fingerprint enhancement filter design by assessing the effect of enhancement on performance of (i) fingerprint feature extraction and (ii) fingerprint matching.

Fingerprint images contain varying amount of noise because of the limitations of the fingerprint acquisition process. It is often necessary to enhance such noisy fingerprint images so that the features extracted from them are reliable. We propose a novel approach to fingerprint enhancement where a set of filters are learned using the "learn-from-example" paradigm. An expert provides the ground truth information for ridges in a small set of representative fingerprint images. The space of local fingerprint patterns in a small neighborhood is partitioned into a set of expressive yet computationally simple classes. A filter is learnt for each partition by finding the optimal linear mapping (in least-square sense) from the input to the enhanced space. The proposed approach offers distinct performance and speed advantages for a wide variety of fingerprint images.

Fingerprint matching is challenging as the matcher has to minimize two competing error rates: the False Accept Rate and the False Reject Rate. We propose a novel, efficient, accurate and distortion-tolerant fingerprint authentication technique based on graph representation. Using the fingerprint minutiae features, a labeled, and weighted graph of minutiae is constructed for both the query fingerprint and the reference fingerprint. In the first phase, we obtain a minimum set of matched node pairs by matching their neighborhood structures. In the second phase, we include more pairs in the match by comparing distances with respect to matched pairs obtained in first phase. An optional third phase, extending the neighborhood around each feature, is entered if we cannot arrive at a decision based on the analysis in first two phases. The proposed algorithm has been tested with excellent results on a large private livescan database obtained with optical scanners.

Computer vision algorithms are natural candidates for high performance computing systems. Algorithms in computer vision are characterized by complex and repetitive operations on large amounts of data involving a variety of data interactions (e.g., point operations, neighborhood operations, global operations). In this paper, we describe the use of the custom computing approach to meet the computation and communication needs of computer vision algorithms. By customizing hardware architecture at the instruction level for every application, the optimal grain size needed for the problem at hand and the instruction granularity can be matched. A custom computing approach can also reuse the same hardware by reconfiguring at the software level for different levels of the computer vision application. We demonstrate the advantages of our approach using Splash 2-a Xilinx 4010-based custom computer.

In an automatic fingerprint identification or authentication system, the matcher subsystem handles the most complex task of compensating for scaling, translation, rotation and structural distortions of the fingerprint minutiae features due to skin elasticity. We analyze the effect of controlled image acquisition on matcher performance. We show that simple steps in image acquisition can enhance the system performance vastly.

With the use of inkless scanners as input devices for acquiring fingerprints of a person, the digital image of the finger is restricted to the area in contact with the sensor The conventional method of fingerprint image acquisition involves obtaining a nail-to-nail image of the finger known as the rolled fingerprint impression. We present a method of constructing a rolled fingerprint from an image sequence of partial fingerprints using a live-scan fingerprint imager.

A page layout segmentation algorithm for locating text, background and halftone areas is presented. The algorithm has been implemented on Splash 2-an FPGA-based array processor. The speed as determined by the Xilinx synthesis tools projects an application speed of 5 MHz. For documents of size 1,024/spl times/1,024 pixels, a significant speedup of two orders of magnitude compared to a SparcStation 20 has been achieved.

With the current rapid growth in multimedia technology, there is an imminent need for efficient techniques to search and query large image databases. Because of their unique and peculiar needs, image databases cannot be treated in a similar fashion to other types of digital libraries. The contextual dependencies present in images, and the complex nature of two-dimensional image data make the representation issues more difficult for image databases. An invariant representation of an image is still an open research issue. For these reasons, it is difficult to find a universal content-based retrieval technique. Current approaches based on shape, texture, and color for indexing image databases have met with limited success. Further, these techniques have not been adequately tested in the presence of noise and distortions. A given application domain offers stronger constraints for improving the retrieval performance. Fingerprint databases are characterized by their large size as well as noisy and distorted query images. Distortions are very common in fingerprint images due to elasticity of the skin. In this paper, a method of indexing large fingerprint image databases is presented. The approach integrates a number of domain-specific high-level features such as pattern class and ridge density at higher levels of the search. At the lowest level, it incorporates elastic structural feature-based matching for indexing the database. With a multilevel indexing approach, we have been able to reduce the search space. The search engine has also been implemented on Splash 2-a field programmable gate array (FPGA)-based array processor to obtain near-ASIC level speed of matching. Our approach has been tested on a locally collected test data and on NIST-9, a large fingerprint database available in the public domain.

Convolution is a fundamental operation in many signal and image processing applications. Since the computation and communication pattern in a convolution operation is regular, a number of special architectures have been designed and implemented for this operator. The Von Neumann architectures cannot meet the real-time requirements of applications that use convolution as an intermediate step. We combine the advantages of systolic algorithms with the low cost of developing application specific designs using field programmable gate arrays (FPGAs) to build a scalable convolver for use in computer vision systems. The performance of the systolic algorithm of (Kung et al., 1981) is compared theoretically and experimentally with many other convolution algorithms reported in the literature. The implementation of a convolution operation on Splash 2, an attached processor based on Xilinx 4010 FPGAs, is reported with impressive performance gains.

Parallel implementations of two computer vision algorithms on distributed cluster platforms are described. The first algorithm is a square-error data clustering method whose parallel implementation is based on the well-known sequential CLUSTER program. The second algorithm is a motion parameter estimation algorithm used to determine correspondence between two images taken of the same scene. Both algorithms have been implemented and tested on cluster platforms using the PVM package. Performance measurements demonstrate that it is possible to attain good performance in terms of execution time and speedup for large-scale problems, provided that adequate memory; swap space, and I/O capacity are available at each node.