For the task of evaluating face recognition algorithms, the research community has adopted a set of de facto standard datasets. These datasets tend to emphasize “difficult pairs” – paired images chosen for differences in factors like age (as in AgeDB-30 and CALFW) and pose (as in CPLFW and CFP-FP). Difficult pairs allow for a more robust evaluation of algorithms, offering granular insight into the factors that are problematic for specific algorithms. However, the existing datasets ignore one factor that has historically proven highly challenging for many face recognition algorithms: race. The faces in these datasets are overwhelmingly White.In this work, we address this demographic gap with the curation of an all-Black dataset for evaluation: AgeDB-30M, where “M” indicates “melanated”. It is the first publicly-available dataset of difficult cross-age image pairs solely from the Black demographic. We hope that AgeDB-30M is a valuable tool for the research community, supporting continued efforts toward more robust algorithmic evaluation, particularly with respect to issues of bias and fairness.

One common critique of biometric authentication is that if an individual’s biometric is compromised, then the individual has no recourse. The concept of revocable biometrics was developed to address this concern. A biometric scheme is revocable if an individual can have their current enrollment in the scheme revoked, so that the compromised biometric template becomes worthless, and the individual can re-enroll with a new template that has similar recognition power. We show that modern deep CNN face matchers inherently allow for a robust revocable biometric scheme. For a given state-of-the-art deep CNN backbone and training set, it is possible to generate an unlimited number of distinct face matcher models that have both (1) equivalent recognition power, and (2) strongly incompatible biometric templates. The equivalent recognition power extends to the point of generating impostor and genuine distributions that have the same shape and placement on the similarity dimension, meaning that the models can share a similarity threshold for a 1-in-10,000 false match rate. The biometric templates from different model instances are so strongly incompatible that the cross-instance similarity score for images of the same person is typically lower than the sameinstance similarity score for images of different persons. That is, a stolen biometric template that is revoked is of less value in attempting to match the re-enrolled identity than the average impostor template. We also explore the feasibility of using a Vision Transformer (ViT) backbone-based face matcher in the revocable biometric system proposed in this work and demonstrate that it is less suitable compared to typical ResNet-based deep CNN backbones.

The first layer of a deep CNN backbone applies filters to an image to extract the basic features available to later layers. During training, some filters may go inactive, meaning all weights in the filter are near zero. An inactive filter in the final model represents a missed opportunity to extract a useful feature. This phenomenon is especially prevalent in specialized CNNs such as for face recognition (as opposed to, e.g., ImageNet). For example, in one of the most widely-used face recognition models (ArcFace), about half of the filters in the first layer are inactive. We propose a novel approach designed and tested specifically for face recognition networks, known as “CRAFT: Contextual Re-Activation of Filters for Face Recognition Training”. Additionally, CRAFT achieves statistically significant improvements in accuracy over standard training on face recognition benchmarks such as AgeDB-30, CPLFW, LFW, CALFW, CFP-FP, IJBB, and IJBC where accuracy has largely saturated. Notable improvements are observed, with significant gains on the highly challenging Hadrian and Eclipse datasets.

Automated one-to-many ($1: \mathrm{N}$) face recognition is a powerful investigative tool commonly used by law enforcement agencies. In this context, potential matches resulting from automated 1:N recognition are reviewed by human examiners prior to possible use as investigative leads. While automated 1:N recognition can achieve near-perfect accuracy under ideal imaging conditions, operational scenarios may necessitate the use of surveillance imagery, which is often degraded in various quality dimensions. One important quality dimension is image resolution, typically quantified by the number of pixels on the face. The common metric for this is inter-pupillary distance (IPD), which measures the number of pixels between the pupils. Low IPD is known to degrade the accuracy of automated face recognition. However, the threshold IPD for reliability in human face recognition remains undefined. This study aims to explore the boundaries of human recognition accuracy by systematically testing accuracy across a range of IPD values. We find that at low IPDs ($10 \mathrm{px}, 5 \mathrm{px}$), human accuracy is at or below chance levels ($50.7 \%, 35.9 \%$), even as confidence in decision-making remains relatively high ($77 \%, 70.7 \%$). Our findings indicate that, for low IPD images, human recognition ability could be a limiting factor to overall system accuracy.

How well can humans recognize faces at extremely low resolution? We conducted a controlled study with 100 participants to evaluate this question—and now FG2025 attendees can try it for themselves. Our interactive demo challenges attendees to match heavily degraded probe images to high-quality reference images, simulating conditions common in operational face recognition contexts. In doing so, it highlights the perceptual limits of human recognition and the risk of misidentification in high-stakes settings. The demo runs offline on standard laptops, collects no personal data, and takes about three minutes to complete.

A person’s facial hairstyle, such as presence and size of beard, can significantly impact face recognition accuracy. While previous research has examined the facial hair effect using binary attributes, no work utilizes a segmentation model to capture the full extent of the facial hair. To investigate the effect of facial hair size in a rigorous manner, we first created a set of fine-grained facial hair annotations to train a segmentation model. Cross-dataset evaluation is performed and accuracy across African-American and Caucasian face images is reported. We then use our facial hair predictions to categorize image pairs according to the degree of difference or similarity in the facial hairstyle. We find that the False Match Rates for image pairs with different categories of facial hairstyle varies by a factor of over 10 for African-American males and over 25 for Caucasian males on MORPH dataset. Also, False Non-Match Rates of 4 race categories on BA-Test dataset are analyzed to measure the accuracy bias in unconstrained settings. Our findings suggest that, while facial hair can cause a shift in similarity score distributions, this effect can be mitigated by employing an adaptive threshold based on facial hair predictions. Facial hair annotations: https://github.com/kaganozturk/Effects-of-Facial-Hair-on-Face-Recognition.

One-to-many facial identification is documented to achieve high accuracy in the case where both the probe and the gallery are ‘mugshot quality’ images. However, an increasing number of documented instances of wrongful arrest following one-to-many facial identification have raised questions about its accuracy. Probe images used in one-to-many facial identification are often cropped from frames of surveillance video and deviate from ‘mugshot quality’ in various ways. This paper systematically explores how the accuracy of one-to-many facial identification is degraded by the person in the probe image choosing to wear dark sunglasses. We show that sunglasses degrade accuracy for mugshot-quality images by an amount similar to strong blur or noticeably lower resolution. Further, we demonstrate that the combination of sunglasses with blur or lower resolution results in even more pronounced loss in accuracy. These results have important implications for developing objective criteria to qualify a probe image for the level of accuracy to be expected if it used for one-to-many identification. To ameliorate the accuracy degradation caused by dark sunglasses, we show that it is possible to recover about 38% of the lost accuracy by synthetically adding sunglasses to all the gallery images, without model re-training. We also show that the frequency of wearing-sunglasses images is very low in existing training sets, and that increasing the representation of wearing-sunglasses images can greatly reduce the error rate. The image set assembled for this research is available at https://cvrl.nd.edu/projects/data/ to support replication and further research.

LivDet-Iris 2025 is the sixth edition of the iris liveness detection competition. Held every two to three years, the competition aims to foster the development of robust algorithms capable of detecting a wide range of physically-and digitally-presented attacks in iris biometrics. The 2025 edition obtained the largest number of submissions in the history of the competition: ten algorithms from five institutions, and one commercial iris recognition system. LivDet-Iris 2025 also introduced new tasks compared to previous editions: (Task 1) a benchmark offered by an industry partner, (Task 2) morphed iris images, in which two different-identity samples were blended into one image, and (Task 3) evaluation of presentation attack detection robustness against advanced manufacturing techniques for textured contact lenses. This edition, for the first time in the series, offers a systematic testing of a commercial iris recognition system (software and hardware) using physical artifacts presented to the sensor. Dermalog-Iris team submitted algorithms that won all tasks, achieving the area under the ROC curve of 90.57%, 68.23% and 99.99% in tasks 1, 2, and 3, respectively. Additionally, we include results for baseline algorithms, based on modern deep convolutional neural networks and trained with all available public datasets of iris images representing bona fide samples and anomalies (physical attacks, eye diseases, post-mortem cases, and synthetically-generated iris images). Test samples created for tasks 2 and 3, and baseline models are made available to offer the state-of-the-art benchmark for iris liveness detection.

Facial filters are now commonplace for social media users around the world. Previous work has demonstrated that facial filters can negatively impact automated face recognition performance. However, these studies focus on small numbers of hand-picked filters in particular styles. In order to more effectively incorporate the wide ranges of filters present on various social media applications, we introduce a framework that allows for larger-scale study of the impact of facial filters on automated recognition. This framework includes a controlled dataset of face images, a principled filter selection process that selects a representative range of filters for experimentation, and a set of experiments to evaluate the filters’ impact on recognition. We demonstrate our framework with a case study of filters from the American applications Instagram and Snapchat and the Chinese applications Meitu and Pitu to uncover cross-cultural differences. Finally, we show how the filtering effect in a face embedding space can easily be detected and restored to improve face recognition performance.

This paper studies how to synthesize face images of non-existent persons, to create a dataset that allows effective training of face recognition (FR) models. Besides generating realistic face images, two other important goals are: 1) the ability to generate a large number of distinct identities (inter-class separation), and 2) a proper variation in appearance of the images for each identity (intra-class variation). However, existing works 1) are typically limited in how many well-separated identities can be generated and 2) either neglect or use an external model for attribute augmentation. We propose Vec2Face, a holistic model that uses only a sampled vector as input and can flexibly generate and control the identity of face images and their attributes. Composed of a feature masked autoencoder and an image decoder, Vec2Face is supervised by face image reconstruction and can be conveniently used in inference. Using vectors with low similarity among themselves as inputs, Vec2Face generates well-separated identities. Randomly perturbing an input identity vector within a small range allows Vec2Face to generate faces of the same identity with proper variation in face attributes. It is also possible to generate images with designated attributes by adjusting vector values with a gradient descent method. Vec2Face has efficiently synthesized as many as 300K identities, whereas 60K is the largest number of identities created in the previous works. As for performance, FR models trained with the generated HSFace datasets, from 10k to 300k identities, achieve state-of-the-art accuracy, from 92\% to 93.52\%, on five real-world test sets (\emph{i.e.}, LFW, CFP-FP, AgeDB-30, CALFW, and CPLFW). For the first time, the FR model trained using our synthetic training set achieves higher accuracy than that trained using a same-scale training set of real face images on the CALFW, IJBB, and IJBC test sets.

Ensuring logical consistency in predictions is a crucial yet overlooked aspect in face attribute classification. We explore the potential reasons for this oversight and introduce two pressing challenges to the field: 1) How can we ensure that a model, when trained with data checked for logical consistency, yields predictions that are logically consistent? 2) How can we achieve the same with training data that hasn't undergone logical consistency checks? Minimizing manual effort is also essential for enhancing automation. To address these challenges, we introduce two datasets, FH41K and CelebA-logic, and propose LogicNet, which combines adversarial learning and label poisoning to learn the logical relationship between attributes without the need for post-processing steps. The accuracy of LogicNet surpasses that of the next-best approach by 13.36%, 9.96%, and 1.01% on FH37K, FH41K, and CelebA-logic, respectively. In real-world case analysis, our approach can achieve a reduction of more than 50% in the average number of failed cases (logically inconsistent attributes) compared to other methods. Code link: https://github.com/HaiyuWu/LogicNet.

While biases such as gender, race, and age have been closely examined in biometric recognition, especially in face and fingerprint traits, their exploration in gait-based recognition is lacking, except for one study. We formulate conditional-matched bootstrap analysis to control for confounding covariates like clothing style, height, and walking speed. The goal is to isolate genuine gender effects on gait recognition. We delve into gender-based disparities in gait recognition by using several state-of-the-art gait recognition methodologies - GaitSet, GaitPart, and GaitGL. For our analysis, the widely-referenced OU-MVLP dataset served as our foundation, which we enhanced with annotations about clothing style, body height, and walking speed. The results were illuminating. We observed a disparity in recognition performance across genders on the original dataset, with recognition for females higher than for males. However, after controlling for covariate distributions using conditional-matched bootstrap analysis, the gap was reduced, with clothing type emerging as the most significant contributor. Code available at https://github.com/azimIbragimov/gait-gender

The prevailing approach in face recognition is to specialize a deep network for general computer vision to the face recognition task. ResNet being specialized for use as the backbone in SoTA face recognition systems is a prime example of this. One significant architectural deviation in the ResNet backbone adapted for face recognition is the linearization of the output spatial map from the last convolution layer to feed the linear layer, rather than utilizing Global Average Pooling (GAP). The utilization of GAP treats all pixel values in the output spatial map as equally significant and averages them naively, thereby compromising the performance of the face recognition model. However, linearization of the spatial map inflates the total parameters in the model by up to 58% (R34) in the lighter version of the ResNet backbone that is typically used for face recognition. Leveraging the prior knowledge that face images during training and testing are pre-aligned, we introduce a novel Gaussian Weighted Pooling (GWP) layer, integrating a pre-computed Gaussian Attention Kernel with the Average Pooling Layer that weighs the importance of the pixel based on the spatial position. Our findings show that utilizing GWP consistently outperforms GAP and achieves results comparable to those of parameter-inflated baseline models.

Face image synthesis has progressed beyond the point at which humans can effectively distinguish authentic faces from synthetically-generated ones. Recently developed synthetic face image detectors boast ``better-than-human'' discriminative ability, especially those guided by human perceptual intelligence during the model's training process. In this paper, we investigate whether these human-guided synthetic face detectors can assist non-expert human operators in the task of synthetic image detection when compared to models trained without human-guidance. We conducted a large-scale experiment with more than 1,560 subjects classifying whether an image shows an authentic or synthetically-generated face, and annotating regions supporting their decisions. In total, 56,015 annotations across 3,780 unique face images were collected. All subjects first examined samples without any AI support, followed by samples given (a) the AI's decision (``synthetic'' or ``authentic''), (b) class activation maps illustrating where the model deems salient for its decision, or (c) both the AI's decision and AI's saliency map. Synthetic faces were generated with six modern Generative Adversarial Networks. Interesting observations from this experiment include: (1) models trained with human-guidance, which are also more accurate in our experiments, offer better support to human examination of face images when compared to models trained traditionally using cross-entropy loss, (2) binary decisions presented to humans results in their better performance than when saliency maps are presented, (3) understanding the AI's accuracy helps humans to increase trust in a given model and thus increase their overall accuracy. This work demonstrates that although humans supported by machines achieve better-than-random accuracy of synthetic face detection, the approaches of supplying humans with AI support and of building trust are key factors determining high effectiveness of the human-AI tandem.

Face attribute research has so far used only simple binary attributes for facial hair; e.g., beard / no beard. We have created a new, more descriptive facial hair annotation scheme and applied it to create a new facial hair attribute dataset, FH37K. Face attribute research also so far has not dealt with logical consistency and completeness. For example, in prior research, an image might be classified as both having no beard and also having a goatee (a type of beard). We show that the test accuracy of previous classification methods on facial hair attribute classification drops significantly if logical consistency of classifications is enforced. We propose a logically consistent prediction loss, LCPLoss, to aid learning of logical consistency across attributes, and also a label compensation training strategy to eliminate the problem of no positive prediction across a set of related attributes. Using an attribute classifier trained on FH37K, we investigate how facial hair affects face recognition accuracy, including variation across demographics. Results show that similarity and difference in facial hairstyle have important effects on the impostor and genuine score distributions in face recognition. The code is at https://github.com/HaiyuWu/LogicalConsistency.

In this article, we analyze how changing the underlying 3D shape of the base identity in face images can distort their overall appearance, especially from the perspective of deep face recognition. As done in popular training data augmentation schemes, we graphically render real and synthetic face images with randomly chosen or best-fitting 3D face models to generate novel views of the base identity. We compare deep features generated from these images to assess the perturbation these renderings introduce into the original identity. We perform this analysis at various degrees of facial yaw with the base identities varying in gender and ethnicity. Additionally, we investigate if adding some form of context and background pixels in these rendered images, when used as training data, further improves the downstream performance of a face recognition model. Our experiments demonstrate the significance of facial shape in accurate face matching and underpin the importance of contextual data for network training.

This paper describes the results of the 2023 edition of the “LivDet” series of iris presentation attack detection (PAD) competitions. New elements in this fifth competition include (1) GAN-generated iris images as a category of presentation attack instruments (PAI), and (2) an evaluation of human accuracy at detecting PAI as a reference benchmark. Clarkson University and the University of Notre Dame contributed image datasets for the competition, composed of samples representing seven different PAI categories, as well as baseline PAD algorithms. Fraunhofer IGD, Beijing University of Civil Engineering and Architecture, and Hochschule Darmstadt contributed results for a total of eight PAD algorithms to the competition. Accuracy results are analyzed by different PAI types, and compared to human accuracy. Overall, the Fraunhofer IGD algorithm, using an attention-based pixel-wise binary supervision network, showed the best-weighted accuracy results (average classification error rate of 37.31%), while the Beijing University of Civil Engineering and Architecture’s algorithm won when equal weights for each PAI were given (average classification rate of 22.15%). These results suggest that iris PAD is still a challenging problem.

Can deep learning models achieve greater generalization if their training is guided by reference to human perceptual abilities? And how can we implement this in a practical manner? This paper proposes a training strategy to ConveY Brain Oversight to Raise Generalization (CYBORG). This new approach incorporates human-annotated saliency maps into a loss function that guides the model’s learning to focus on image regions that humans deem salient for the task. The Class Activation Mapping (CAM) mechanism is used to probe the model’s current saliency in each training batch, juxtapose this model saliency with human saliency, and penalize large differences. Results on the task of synthetic face detection, selected to illustrate the effectiveness of the approach, show that CYBORG leads to significant improvement in accuracy on unseen samples consisting of face images generated from six Generative Adversarial Networks across multiple classification network architectures. We also show that scaling to even seven times the training data, or using non-human-saliency auxiliary information, such as segmentation masks, and standard loss cannot beat the performance of CYBORG-trained models. As a side effect of this work, we observe that the addition of explicit region annotation to the task of synthetic face detection increased human classification accuracy. This work opens a new area of research on how to incorporate human visual saliency into loss functions in practice. All data, code and trained models used in this work are offered with this paper.

Thorough evaluation is critical for developing models that are fair and robust. In this work, we describe the Conditional Attribute Subsampling Toolkit (CAST) for selecting data subsets for fine-grained scientific evaluations. Our toolkit efficiently filters data given an arbitrary number of conditions for metadata attributes. The purpose of the toolkit is to allow researchers to easily to evaluate models on targeted test distributions. The functionality of CAST is demonstrated on the WebFace42M face Recognition dataset. We calculate over 50 attributes for this dataset including race, image quality, facial features, and accessories. Using our toolkit, we create over a hundred test sets conditioned on one or multiple attributes. Results are presented for subsets of various demographics and image quality ranges. Using eleven different subsets, we build a face recognition 1:1 verification benchmark called C11 that exclusively contains pairs that are near the decision threshold. Evaluation on C11 with state-of-the-art methods demonstrates the suitability of the proposed benchmark. The toolkit is publicly available at https://github.com/WesRobbins/CAST.

Research in presentation attack detection (PAD) for iris recognition has largely moved beyond evaluation in “closed-set” scenarios, to emphasize ability to generalize to presentation attack types not present in the training data. This paper offers multiple contributions to understand and extend the state-of-the-art in open-set iris PAD. First, it describes the most authoritative evaluation to date of iris PAD. We have curated the largest publicly-available image dataset for this problem, drawing from 26 benchmarks previously released by various groups, and adding 150,000 images being released with this paper, to create a set of 450,000 images representing authentic iris and seven types of presentation attack instrument (PAI). We formulate a leave-one-PAI-out evaluation protocol, and show that even the best algorithms in the closed-set evaluations exhibit catastrophic failures on multiple attack types in the open-set scenario. This includes algorithms performing well in the most recent LivDet-Iris 2020 competition, which may come from the fact that the LivDet-Iris protocol emphasizes sequestered images rather than unseen attack types. Second, we evaluate the accuracy of five open-source iris presentation attack algorithms available today, one of which is newly-proposed in this paper, and build an ensemble method that beats the winner of the LivDet-Iris 2020 by a substantial margin. This paper demonstrates that closed-set iris PAD, when all PAIs are known during training, is a solved problem, with multiple algorithms showing very high accuracy, while open-set iris PAD, when evaluated correctly, is far from being solved. The newly-created dataset, new open-source algorithms, and evaluation protocol, all made publicly available with this paper, provide experimental artifacts that researchers can use to measure progress on this important problem.

Variation in face recognition accuracy across demographic groups has attracted attention from news media, civil liberties advocates and academic researchers. The problem is challenging, in that both the impostor distribution (matches across different people) and the genuine distribution (matches across same people) may vary across demographic groups. Simple answers such as balancing the number of subjects and images in the training data do not have a substantial impact on demographic accuracy disparities. We present the first investigation into whether parts of the face - such as eyes, nose, mouth - show the same accuracy differences across demographic groups as are seen with matching the whole face. We show that matching focused on different parts of the face may result in opposite accuracy differences across demographics. For example, using the eye region for face matching results in Caucasian males having a better impostor distribution (lower similarity scores) than Caucasian females, but using the nose regionfor face matching results in Caucasian females having a better impostor distribution. We also show that it is possible to select face region(s) that effectively minimize the difference in the impostor or genuine distributions across at least some demographics. Our results suggest that a new pathway to reducing accuracy disparity across demographic groups may be to weight the parts of the face differently in matching.

Accuracy alone is not sufficient to establish the efficacy of an AI algorithm-issues of demographic bias are an important area of concern. Demographic bias in face recognition algorithms has attracted more attention from the re-search community to date, but bias can also be a problem for face image analysis algorithms, such as detection of manipulated face images. In this paper, we investigate performance of humans and algorithms at detecting retouched face images of subjects from different origin (America, India, China) and gender groups. To be representative of the state of retouching detection, we use eight different algorithms from the literature. In addition to overall human accuracy, differences across origin and gender of the human performing the task are analyzed. We observe different bias patterns, such as algorithms show higher in-group accuracy than out-group, while the extent of retouching and familiarity drives differences in detection accuracy for humans. This is the first work to analyze and compare bias exhibited by humans and algorithms in similar tasks of detecting retouched face images.

Deep learning has driven remarkable accuracy increases in many computer vision problems. One ongoing challenge is how to achieve the greatest accuracy in cases where training data is limited. A second ongoing challenge is that trained models oftentimes do not generalize well even to new data that is subjectively similar to the training set. We address these challenges in a novel way, with the first-ever (to our knowledge) exploration of encoding human judgement about salient regions of images into the training data. We compare the accuracy and generalization of a state-of-the-art deep learning algorithm for a difficult problem in biometric presentation attack detection when trained on (a) original images with typical data augmentations, and (b) the same original images transformed to encode human judgement about salient image regions. The latter approach results in models that achieve higher accuracy and better generalization, decreasing the error of the LivDet-Iris 2020 winner from 29.78% to 16.37%, and achieving impressive generalization in a leave-one-attack-type-out evaluation scenario. This work opens a new area of study for how to embed human intelligence into training strategies for deep learning to achieve high accuracy and generalization in cases of limited training data.

Remote photoplethysmography (rPPG) is a technique for estimating blood volume changes from reflected light without the need for a contact sensor. We present the first examples of presentation attacks in the digital and physical domains on rPPG from face video. Digital attacks are easily performed by adding imperceptible periodic noise to the input videos. Physical attacks are performed with illumination from visible spectrum LEDs placed in close proximity to the face, while still being difficult to perceive with the human eye. We also show that our attacks extend beyond medical applications, since the method can effectively generate a strong periodic pulse on 3D-printed face masks, which presents difficulties for pulse-based face presentation attack detection (PAD). The paper concludes with ideas for using this work to improve robustness of rPPG methods and pulse-based face PAD.

Media reports have accused face recognition of being “biased”, “sexist” and “racist”. There is consensus in the research literature that face recognition accuracy is lower for females, who often have both a higher false match rate and a higher false non-match rate. However, there is little published research aimed at identifying the cause of lower accuracy for females. For instance, the 2019 Face Recognition Vendor Test that documents lower female accuracy across a broad range of algorithms and datasets also lists “Analyze cause and effect” under the heading “What we did not do”. We present the first experimental analysis to identify major causes of lower face recognition accuracy for females on datasets where previous research has observed this result. Controlling for equal amount of visible face in the test images mitigates the apparent higher false non-match rate for females. Additional analysis shows that makeup-balanced datasets further improves females to achieve lower false non-match rates. Finally, a clustering experiment suggests that images of two different females are inherently more similar than of two different males, potentially accounting for a difference in false match rates.

Advances in face synthesis have raised alarms about the deceptive use of synthetic faces. Can synthetic identities be effectively used to fool human observers? In this paper, we introduce a study of the human perception of synthetic faces generated using different strategies including a state-of-the-art deep learning-based GAN model. This is the first rigorous study of the effectiveness of synthetic face generation techniques grounded in experimental techniques from psychology. We answer important questions such as how often do GAN-based and more traditional image processing-based techniques confuse human observers, and are there subtle cues within a synthetic face image that cause humans to perceive it as a fake without having to search for obvious clues? To answer these questions, we conducted a series of large-scale crowdsourced behavioral experiments with different sources of face imagery. Results show that humans are unable to distinguish synthetic faces from real faces under several different circumstances. This finding has serious implications for many different applications where face images are presented to human users.

This paper is the first to explore the question of whether images that are classified incorrectly by a face analytics algorithm (e.g., gender classification) are any more or less likely to participate in an image pair that results in a face recognition error. We analyze results from three different gender classification algorithms (one open-source and two commercial), and two face recognition algorithms (one open-source and one commercial), on image sets representing four demographic groups (African-American female and male, Caucasian female and male). For impostor image pairs, our results show that pairs in which one image has a gender classification error have a better impostor distribution than pairs in which both images have correct gender classification, and so are less likely to generate a false match error. For genuine image pairs, our results show that individuals whose images have a mix of correct and incorrect gender classification have a worse genuine distribution (increased false non-match rate) compared to individuals whose images consistently have correct gender classification. Thus, compared to images that generate correct gender classification, images with gender classification error have a lower false match rate and a higher false non-match rate.

We present the Deception Detection and Physiological Monitoring (DDPM) dataset and initial baseline results on this dataset. Our application context is an interview scenario in which the interviewee attempts to deceive the interviewer on selected responses. The interviewee is recorded in RGB, near-infrared, and long-wave infrared, along with cardiac pulse, blood oxygenation, and audio. After collection, data were annotated for interviewer/interviewee, curated, ground-truthed, and organized into train / test parts for a set of canonical deception detection experiments. Baseline experiments found random accuracy for micro-expressions as an indicator of deception, but that saccades can give a statistically significant response. We also estimated subject heart rates from face videos (remotely) with a mean absolute error as low as 3.16 bpm. The database contains almost 13 hours of recordings of 70 subjects, and over 8 million visible-light, near-infrared, and thermal video frames, along with appropriate meta, audio and pulse oximeter data. To our knowledge, this is the only collection offering recordings of five modalities in an interview scenario that can be used in both deception detection and remote photoplethysmography research.

A dramatic rise in the flow of manipulated image content on the Internet has led to a prompt response from the media forensics research community. New mitigation efforts leverage cutting-edge data-driven strategies and increasingly incorporate usage of techniques from computer vision and machine learning to detect and profile the space of image manipulations. This paper addresses Image Provenance Analysis, which aims at discovering relationships among different manipulated image versions that share content. One important task in provenance analysis, like most visual understanding problems, is establishing a visual description and dissimilarity computation method that connects images that share full or partial content. But the existing handcrafted or learned descriptors - generally appropriate for tasks such as object recognition - may not sufficiently encode the subtle differences between near-duplicate image variants, which significantly characterize the provenance of any image. This paper introduces a novel data-driven learning-based approach that provides the context for ordering images that have been generated from a single image source through various transformations. Our approach learns transformation-aware embeddings using weak supervision via composited transformations and a rank-based Edit Sequence Loss. To establish the effectiveness of the proposed approach, comparisons are made with state-of-the-art handcrafted and deep-learning-based descriptors, as well as image matching approaches. Further experimentation validates the proposed approach in the context of image provenance analysis and improves upon existing approaches.

Diversity and unpredictability of artifacts potentially presented to an iris sensor calls for presentation attack detection methods that are agnostic to specificity of presentation attack instruments. This article proposes a method that combines two-dimensional and three-dimensional properties of the observed iris to address the problem of spoof detection in case when some properties of artifacts are unknown. The 2D (textural) iris features are extracted by a state-of-the-art method employing Binary Statistical Image Features (BSIF) and an ensemble of classifiers is used to deliver 2D modality-related decision. The 3D (shape) iris features are reconstructed by a photometric stereo method from only two images captured under near-infrared illumination placed at two different angles, as in many current commercial iris recognition sensors. The map of normal vectors is used to assess the convexity of the observed iris surface. The combination of these two approaches has been applied to detect whether a subject is wearing a textured contact lens to disguise their identity. Extensive experiments with NDCLD'15 dataset, and a newly collected NDIris3D dataset show that the proposed method is highly robust under various open-set testing scenarios, and that it outperforms all available open-source iris PAD methods tested in identical scenarios. The source code and the newly prepared benchmark are made available along with this article.

Images from social media can reflect diverse viewpoints, heated arguments, and expressions of creativity, adding new complexity to retrieval tasks. Researchers working on Content-Based Image Retrieval (CBIR) have traditionally tuned their algorithms to match filtered results with user search intent. However, we are now bombarded with composite images of unknown origin, authenticity, and even meaning. With such uncertainty, users may not have an initial idea of what the search query results should look like. For instance, hidden people, spliced objects, and subtly altered scenes can be difficult for a user to detect initially in a meme image, but may contribute significantly to its composition. It is pertinent to design systems that retrieve images with these nuanced relationships in addition to providing more traditional results, such as duplicates and near-duplicates - and to do so with enough efficiency at large scale. We propose a new approach for spatial verification that aims at modeling object-level regions using image keypoints retrieved from an image index, which is then used to accurately weight small contributing objects within the results, without the need for costly object detection steps. We call this method the Objects in Scene to Objects in Scene (OS2OS) score, and it is optimized for fast matrix operations, which can run quickly on either CPUs or GPUs. It performs comparably to state-of-the-art methods on classic CBIR problems (Oxford 5K, Paris 6K, and Google-Landmarks), and outperforms them in emerging retrieval tasks such as image composite matching in the NIST MFC2018 dataset and meme-style imagery from Reddit.

Matching live images (“selfies”) to images from ID documents is a problem that can arise in various applications. A challenging instance of the problem arises when the face image on the ID document is from early adolescence and the live image is from later adolescence. We explore this problem using a private dataset called Chilean Young Adult (CHIYA) dataset, where we match live face images taken at age 18-19 to face images on scanned ID documents created at ages 9 to 18. State-of-the-art deep learning face matchers (e.g., ArcFace) have relatively poor accuracy for document-to-selfie face matching. To achieve higher accuracy, we fine-tune the best available open-source model with triplet loss for a few-shot learning. Experiments show that our approach achieves higher accuracy than the DocFace+ model recently developed for this problem. Our fine-tuned model was able to improve the true acceptance rate for the most difficult (largest age span) subset from 62.92% to 96.67% at a false acceptance rate of 0.01%. Our fine-tuned model is available for use by other researchers.

Deep learning methods have greatly increased the accuracy of face recognition, but an old problem still persists: accuracy is usually higher for men than women. It is often speculated that lower accuracy for women is caused by under-representation in the training data. This work investigates female under-representation in the training data is truly the cause of lower accuracy for females on test data. Using a state-of-the-art deep CNN, three different loss functions, and two training datasets, we train each on seven subsets with different male/female ratios, totaling forty two trainings, that are tested on three different datasets. Results show that (1) gender balance in the training data does not translate into gender balance in the test accuracy, (2) the “gender gap” in test accuracy is not minimized by a gender-balanced training set, but by a training set with more male images than female images, and (3) training to minimize the accuracy gap does not result in highest female, male or average accuracy.

The high demand for ever more accurate biometric systems has driven the search for methods that reconstruct the iris surface in a 3D model. The intent in adding the depth dimension is to improve accuracy even in large databases. Here, we present a novel approach to iris recognition from 3D models. First, the iris 3D model is reconstructed from a single image using irisDepth, a CNN based method. Then, a 3D descriptor called Spin Image is obtained for keypoints of the 3D model. After that, matches are found between keypoints in the query and the reference 3D models using k-dimensional trees. Finally, those keypoint matches are used to determine the spatial transformation that best aligns the 3D models. A combination of the transformation error and the inlier ratio is used as the metric to assess the similarity of two iris 3D models. We applied this method in a dataset of 100 eyes and 2,000 iris 3D models. Our results indicate that using the proposed method is more effective than alternative methods, such as Dougman's iris code, point-to-point distance between the 3D models, the 3D rubber sheet model, and CNN-based methods.

Gabor kernels are widely accepted as dominant filters for iris recognition. In this work we investigate, given the current interest in neural networks, if Gabor kernels are the only family of functions performing best in iris recognition, or if better filters can be learned directly from iris data. We use (on purpose) a single-layer convolutional neural network as it mimics an iris code-based algorithm. We learn two sets of data-driven kernels; one starting from randomly initialized weights and the other from open-source set of Gabor kernels. Through experimentation, we show that the network does not converge on Gabor kernels, instead converging on a mix of edge detectors, blob detectors and simple waves. In our experiments carried out with three subject-disjoint datasets we found that the performance of these learned kernels is comparable to the open-source Gabor kernels. These lead us to two conclusions: (a) a family of functions offering optimal performance in iris recognition is wider than Gabor kernels, and (b) we probably hit the maximum performance for an iris coding algorithm that uses a single convolutional layer, yet with multiple filters. Released with this work is a framework to learn data-driven kernels that can be easily transplanted into open-source iris recognition software (for instance, OSIRIS - Open Source IRIS).

Launched in 2013, LivDet-Iris is an international competition series open to academia and industry with the aim to assess and report advances in iris Presentation Attack Detection (PAD). This paper presents results from the fourth competition of the series: LivDet-Iris 2020. This year's competition introduced several novel elements: (a) incorporated new types of attacks (samples displayed on a screen, cadaver eyes and prosthetic eyes), (b) initiated LivDet-Iris as an on-going effort, with a testing protocol available now to everyone via the Biometrics Evaluation and Testing (BEAT)* open-source platform to facilitate reproducibility and benchmarking of new algorithms continuously, and (c) performance comparison of the submitted entries with three baseline methods (offered by the University of Notre Dame and Michigan State University), and three open-source iris PAD methods available in the public domain. The best performing entry to the competition reported a weighted average APCER of 59.10% and a BPCER of 0.46% over all five attack types. This paper serves as the latest evaluation of iris PAD on a large spectrum of presentation attack instruments.

Previous studies generally agree that face recognition accuracy is higher for older persons than for younger persons. But most previous studies were before the wave of deep learning matchers, and most considered accuracy only in terms of the verification rate for genuine pairs. This paper investigates accuracy for age groups 16-29, 30-49 and 50-70, using three modern deep CNN matchers, and considers differences in the impostor and genuine distributions as well as verification rates and ROC curves. We find that accuracy is lower for older persons and higher for younger persons. In contrast, a pre deep learning matcher on the same dataset shows the traditional result ofhigher accuracy for older persons, although its overall accuracy is much lower than that of the deep learning matchers. Comparing the impostor and genuine distributions, we conclude that impostor scores have a larger effect than genuine scores in causing lower accuracy for the older age group. We also investigate the effects of training data across the age groups. Our results show that fine-tuning the deep CNN models on additional images ofolder persons actually lowers accuracy for the older age group. Also, we fine-tune and train from scratch two models using age-balanced training datasets, and these results also show lower accuracy for older age group. These results argue that the lower accuracy for the older age group is not due to imbalance in the original training data.

We propose a multi-scale GAN model to hallucinate realistic context (forehead, hair, neck, clothes) and background pixels automatically from a single input face mask, without any user supervision. Instead of swapping a face on to an existing picture, our model directly generates realistic context and background pixels based on the features of the provided face mask. Unlike facial inpainting algorithms, it can generate realistic hallucinations even for a large number of missing pixels. Our model is composed of a cascaded network of GAN blocks, each tasked with hallucination of missing pixels at a particular resolution while guiding the synthesis process of the next GAN block. The hallucinated full face image is made photo-realistic by using a combination of reconstruction, perceptual, adversarial and identity preserving losses at each block of the network. With a set of extensive experiments, we demonstrate the effectiveness of our model in hallucinating context and background pixels from face masks varying in facial pose, expression and lighting, collected from multiple datasets subject disjoint with our training data. We also compare our method with popular face inpainting and face swapping models in terms of visual quality, realism and identity preservation. Additionally, we analyze our cascaded pipeline and compare it with the progressive growing of GANs, and explore its usage as a data augmentation module for training CNNs.

We propose an algorithm to generate realistic face images of both real and synthetic identities (people who do not exist) with different facial yaw, shape and resolution. The synthesized images can be used to augment datasets to train CNNs or as massive distractor sets for biometric verification experiments without any privacy concerns. Additionally, law enforcement can make use of this technique to train forensic experts to recognize faces. Our method samples face components from a pool of multiple face images of real identities to generate the synthetic texture. Then, a real 3D head model compatible to the generated texture is used to render it under different facial yaw transformations. We perform multiple quantitative experiments to assess the effectiveness of our synthesis procedure in CNN training and its potential use to generate distractor face images. Additionally, we compare our method with popular GAN models in terms of visual quality and execution time.

In most iris recognition systems the texture of the iris image is either the result of the interaction between the iris and Near Infrared (NIR) light, or between the iris pigmentation and visible-light. The iris, however, is a three-dimensional organ, and the information contained on its relief is not being exploited completely. In this article, we present an image acquisition method that enhances viewing the structural information of the iris. Our method consists of adding lateral illumination to the visible light frontal illumination to capture the structural information of the muscle fibers of the iris on the resulting image. These resulting images contain highly textured patterns of the iris. To test our method, we collected a database of 1,920 iris images using both a conventional NIR device, and a custom-made device that illuminates the eye in lateral and frontal angles with visible-light (LFVL). Then, we compared the iris recognition performance of both devices by means of a Hamming distance distribution analysis among the corresponding binary iris codes. The ROC curves show that our method produced more separable distributions than those of the NIR device, and much better distribution than using frontal visible-light alone. Eliminating errors produced by images captured with different iris dilation (13 cases), the NIR produced inter-class and intra-class distributions that are completely separable as in the case of LFVL. This acquisition method could also be useful for 3D iris scanning.

Creative works, whether paintings or memes, follow unique journeys that result in their final form. Understanding these journeys, a process known as "provenance analysis," provides rich insights into the use, motivation, and authenticity underlying any given work. The application of this type of study to the expanse of unregulated content on the Internet is what we consider in this paper. Provenance analysis provides a snapshot of the chronology and validity of content as it is uploaded, re-uploaded, and modified over time. Although still in its infancy, automated provenance analysis for online multimedia is already being applied to different types of content. Most current works seek to build provenance graphs based on the shared content between images or videos. This can be a computationally expensive task, especially when considering the vast influx of content that the Internet sees every day. Utilizing non-content-based information, such as timestamps, geotags, and camera IDs can help provide important insights into the path a particular image or video has traveled during its time on the Internet without large computational overhead. This paper tests the scope and applicability of metadata-based inferences for provenance graph construction in two different scenarios: digital image forensics and cultural analytics.

We propose a new iris presentation attack detection method using three-dimensional features of an observed iris region estimated by photometric stereo. Our implementation uses a pair of iris images acquired by a common commercial iris sensor (LG 4000). No hardware modifications of any kind are required. Our approach should be applicable to any iris sensor that can illuminate the eye from two different directions. Each iris image in the pair is captured under near-infrared illumination at a different angle relative to the eye. Photometric stereo is used to estimate surface normal vectors in the non-occluded portions of the iris region. The variability of the normal vectors is used as the presentation attack detection score. This score is larger for a texture that is irregularly opaque and printed on a convex contact lens, and is smaller for an authentic iris texture. Thus the problem is formulated as binary classification into (a) an eye wearing textured contact lens and (b) the texture of an actual iris surface (possibly seen through a clear contact lens). Experiments were carried out on a database of approx. 2,900 iris image pairs acquired from approx. 100 subjects. Our method was able to correctly classify over 95% of samples when tested on contact lens brands unseen in training, and over 98% of samples when the contact lens brand was seen during training. The source codes of the method are made available to other researchers.

Binarized statistical image features (BSIF) have been successfully used for texture analysis in many computer vision tasks, including iris recognition and biometric presentation attack detection. One important point is that all applications of BSIF in iris recognition have used the original BSIF filters, which were trained on image patches extracted from natural images. This paper tests the question of whether domain-specific BSIF can give better performance than the default BSIF. The second important point is in the selection of image patches to use in training for BSIF. Can image patches derived from eye-tracking experiments, in which humans perform an iris recognition task, give better performance than random patches? Our results say that (1) domain-specific BSIF features can out-perform the default BSIF features, and (2) selecting image patches in a task-specific manner guided by human performance can out-perform selecting random patches. These results are important because BSIF is often regarded as a generic texture tool that does not need any domain adaptation, and human-task-guided selection of patches for training has never (to our knowledge) been done. This paper follows the reproducible research requirements, and the new iris-domain-specific BSIF filters, the patches used in filter training, the database used in testing and the source codes of the designed iris recognition method are made available along with this paper to facilitate applications of this concept.

Predicting gender from iris images has been reported by several researchers as an application of machine learning in biometrics. Recent works on this topic have suggested that the preponderance of the gender cues is located in the periocular region rather than in the iris texture itself. This paper focuses on teasing out whether the information for gender prediction is in the texture of the iris stroma, the periocular region, or both. We present a larger dataset for gender from iris, and evaluate gender prediction accuracy using linear SVM and CNN, comparing hand-crafted and deep features. We use probabilistic occlusion masking to gain insight on the problem. Results suggest the discriminative power of the iris texture for gender is weaker than previously thought, and that the gender-related information is primarily in the periocular region.

This paper advances the state of the art in human examination of iris images by (1) assessing the impact of different iris conditions in identity verification, and (2) introducing an annotation step that improves the accuracy of people's decisions. In a first experimental session, 114 subjects were asked to decide if pairs of iris images depict the same eye (genuine pairs) or two distinct eyes (impostor pairs). The image pairs sampled six conditions: (1) easy for algorithms to classify, (2) difficult for algorithms to classify, (3) large difference in pupil dilation, (4) disease-affected eyes, (5) identical twins, and (6) post-mortem samples. In a second session, 85 of the 114 subjects were asked to annotate matching and non-matching regions that supported their decisions. Subjects were allowed to change their initial classification as a result of the annotation process. Results suggest that: (a) people improve their identity verification accuracy when asked to annotate matching and non-matching regions between the pair of images, (b) images depicting the same eye with large difference in pupil dilation were the most challenging to subjects, but benefited well from the annotation-driven classification, (c) humans performed better than iris recognition algorithms when verifying genuine pairs of post-mortem and disease-affected eyes (i.e., samples showing deformations that go beyond the distortions of a healthy iris due to pupil dilation), and (d) annotation does not improve accuracy of analyzing images from identical twins, which remain confusing for people.

The adoption of large-scale iris recognition systems around the world has brought to light the importance of detecting presentation attack images (textured contact lenses and printouts). This paper presents a new approach in iris presentation attack detection (PAD) by exploring combinations of convolutional neural networks (CNNs) and transformed input spaces through binarized statistical image features (BSIFs). Our method combines lightweight CNNs to classify multiple BSIF views of the input image. Following explorations on complementary input spaces leading to more discriminative features to detect presentation attacks, we also propose an algorithm to select the best (and most discriminative) predictors for the task at hand. An ensemble of predictors makes use of their expected individual performances to aggregate their results into a final prediction. Results show that this technique improves on the current state of the art in iris PAD, outperforming the winner of LivDet-Iris 2017 competition both for intra- and cross-dataset scenarios, and illustrating the very difficult nature of the cross-dataset scenario.

Prior art has shown it is possible to estimate, through image processing and computer vision techniques, the types and parameters of transformations that have been applied to the content of individual images to obtain new images. Given a large corpus of images and a query image, an interesting further step is to retrieve the set of original images whose content is present in the query image, as well as the detailed sequences of transformations that yield the query image given the original images. This is a problem that recently has received the name of image provenance analysis. In these times of public media manipulation (e.g., fake news and meme sharing), obtaining the history of image transformations is relevant for fact checking and authorship verification, among many other applications. This article presents an end-to-end processing pipeline for image provenance analysis, which works at real-world scale. It employs a cutting-edge image filtering solution that is custom-tailored for the problem at hand, as well as novel techniques for obtaining the provenance graph that expresses how the images, as nodes, are ancestrally connected. A comprehensive set of experiments for each stage of the pipeline is provided, comparing the proposed solution with state-of-the-art results, employing previously published datasets. In addition, this work introduces a new dataset of real-world provenance cases from the social media site Reddit, along with baseline results.

In this paper, we propose a novel face synthesis approach that can generate an arbitrarily large number of synthetic images of both real and synthetic identities. Thus a face image dataset can be expanded in terms of the number of identities represented and the number of images per identity using this approach, without the identity-labeling and privacy complications that come from downloading images from the web. To measure the visual fidelity and uniqueness of the synthetic face images and identities, we conducted face matching experiments with both human participants and a CNN pre-trained on a dataset of 2.6M real face images. To evaluate the stability of these synthetic faces, we trained a CNN model with an augmented dataset containing close to 200,000 synthetic faces. We used a snapshot of this trained CNN to recognize extremely challenging frontal (real) face images. Experiments showed training with the augmented faces boosted the face recognition performance of the CNN.

The need to verify identity has become an everyday experience for most people. Biometrics is the principal means for reliable identification of people. Although iris recognition is the most reliable current technique for biometric identification, it has limitations because only segments of the iris are available due to occlusions from the eyelids, eyelashes, specular highlights, etc. The goal of this research is to study iris reconstruction from several 2D near infrared iris images, adding depth information to iris recognition. We expect that adding depth information from the iris surface will make it possible to identify people from a smaller segment of the iris. We designed a sensor for 2D near-infrared iris image acquisition. The method follows a pre-processing stage with the goal of performing iris enhancement, eliminating occlusions, reflections and extreme gray-level values, ending in iris texture equalization. The last step is the 3D iris model reconstruction based on several 2D iris images acquired at different angles. Results from each stage are presented.

Digital retouching of face images is becoming more widespread due to the introduction of software packages that automate the task. Several researchers have introduced algorithms to detect whether a face image is original or retouched. However, previous work on this topic has not considered whether or how accuracy of retouching detection varies with the demography of face images. In this paper, we introduce a new Multi-Demographic Retouched Faces (MDRF) dataset, which contains images belonging to two genders, male and female, and three ethnicities, Indian, Chinese, and Caucasian. Further, retouched images are created using two different retouching software packages. The second major contribution of this research is a novel semi-supervised autoencoder incorporating “sub-class” information to improve classification. The proposed approach outperforms existing state-of-the-art detection algorithms for the task of generalized retouching detection. Experiments conducted with multiple combinations of ethnicities show that accuracy of retouching detection can vary greatly based on the demographics of the training and testing images.

Presentation attacks such as using a contact lens with a printed pattern or printouts of an iris can be utilized to bypass a biometric security system. The first international iris liveness competition was launched in 2013 in order to assess the performance of presentation attack detection (PAD) algorithms, with a second competition in 2015. This paper presents results of the third competition, LivDet-Iris 2017. Three software-based approaches to Presentation Attack Detection were submitted. Four datasets of live and spoof images were tested with an additional cross-sensor test. New datasets and novel situations of data have resulted in this competition being of a higher difficulty than previous competitions. Anonymous received the best results with a rate of rejected live samples of 3.36% and rate of accepted spoof samples of 14.71%. The results show that even with advances, printed iris attacks as well as patterned contacts lenses are still difficult for software-based systems to detect. Printed iris images were easier to be differentiated from live images in comparison to patterned contact lenses as was also seen in previous competitions.

Deriving relationships between images and tracing back their history of modifications are at the core of Multimedia Phylogeny solutions, which aim to combat misinformation through doctored visual media. Nonetheless, most recent image phylogeny solutions cannot properly address cases of forged composite images with multiple donors, an area known as multiple parenting phylogeny (MPP). This paper presents a preliminary undirected graph construction solution for MPP, without any strict assumptions. The algorithm is underpinned by robust image representative keypoints and different geometric consistency checks among matching regions in both images to provide regions of interest for direct comparison. The paper introduces a novel technique to geometrically filter the most promising matches as well as to aid in the shared region localization task. The strength of the approach is corroborated by experiments with real-world cases, with and without image distractors (unrelated cases).

As image tampering becomes ever more sophisticated and commonplace, the need for image forensics algorithms that can accurately and quickly detect forgeries grows. In this paper, we revisit the ideas of image querying and retrieval to provide clues to better localize forgeries. We propose a method to perform large-scale image forensics on the order of one million images using the help of an image search algorithm and database to gather contextual clues as to where tampering may have taken place. In this vein, we introduce five new strongly invariant image comparison methods and test their effectiveness under heavy noise, rotation, and color space changes. Lastly, we show the effectiveness of these methods compared to passive image forensics using Nimble [1], a new, state-of-the-art dataset from the National Institute of Standards and Technology (NIST).

Departing from traditional digital forensics modeling, which seeks to analyze single objects in isolation, multimedia phylogeny analyzes the evolutionary processes that influence digital objects and collections over time. One of its integral pieces is provenance filtering, which consists of searching a potentially large pool of objects for the most related ones with respect to a given query, in terms of possible ancestors (donors or contributors) and descendants. In this paper, we propose a two-tiered provenance filtering approach to find all the potential images that might have contributed to the creation process of a given query q. In our solution, the first (coarse) tier aims to find the most likely “host” images - the major donor or background - contributing to a composite/doctored image. The search is then refined in the second tier, in which we search for more specific (potentially small) parts of the query that might have been extracted from other images and spliced into the query image. Experimental results with a dataset containing more than a million images show that the two-tiered solution underpinned by the context of the query is highly useful for solving this difficult task.

Real-world datasets are often imbalanced, with an important class having many fewer examples than other classes. In medical data, normal examples typically greatly outnumber disease examples. A classifier learned from imbalanced data, will tend to be very good at the predicting examples in the larger (normal) class, yet the smaller (disease) class is typically of more interest. Imbalance is dealt with at the feature vector level (create synthetic feature vectors or discard some examples from the larger class) or by assigning differential costs to errors. Here, we introduce a novel method for over-sampling minority class examples at the image level, rather than the feature vector level. Our method was applied to the problem of Glioblastoma patient survival group prediction. Synthetic minority class examples were created by adding Gaussian noise to original medical images from the minority class. Uniform local binary patterns (LBP) histogram features were then extracted from the original and synthetic image examples with a random forests classifier. Experimental results show the new method (Image SMOTE) increased minority class predictive accuracy and also the AUC (area under the receiver operating characteristic curve), compared to using the imbalanced dataset directly or to creating synthetic feature vectors.

Predicting a person's gender based on the iris texture has been explored by several researchers. This paper considers several dimensions of experimental work on this problem, including person-disjoint train and test, and the effect of cosmetics on eyelash occlusion and imperfect segmentation. We also consider the use of multi-layer perceptron and convolutional neural networks as classifiers, comparing the use of data-driven and hand-crafted features. Our results suggest that the gender-from-iris problem is more difficult than has so far been appreciated. Estimating accuracy using a mean of N person-disjoint train and test partitions, and considering the effect of makeup - a combination of experimental conditions not present in any previous work - we find a much weaker ability to predict genderfrom-iris texture than has been suggested in previous work.

This paper presents a solution to automatically recognize the correct left/right and upright/upside-down orientation of iris images. This solution can be used to counter spoofing attacks directed to generate fake identities by rotating an iris image or the iris sensor during the acquisition. Two approaches are compared on the same data, using the same evaluation protocol: 1) feature engineering, using hand-crafted features classified by a support vector machine (SVM) and 2) feature learning, using data-driven features learned and classified by a convolutional neural network (CNN). A data set of 20 750 iris images, acquired for 103 subjects using four sensors, was used for development. An additional subject-disjoint data set of 1,939 images, from 32 additional subjects, was used for testing purposes. Both same-sensor and cross-sensor tests were carried out to investigate how the classification approaches generalize to unknown hardware. The SVM-based approach achieved an average correct classification rate above 95% (89%) for recognition of left/right (upright/upside-down) orientation when tested on subject-disjoint data and camera-disjoint data, and 99% (97%) if the images were acquired by the same sensor. The CNN-based approach performed better for same-sensor experiments, and presented slightly worse generalization capabilities to unknown sensors when compared with the SVM. We are not aware of any other papers on the automatic recognition of upright/upside-down orientation of iris images, or studying both hand-crafted and data-driven features in same-sensor and cross-sensor subject-disjoint experiments. The data sets used in this paper, along with random splits of the data used in cross-validation, are being made available.

Unconstrained face recognition is still an open problem as the state-of-the-art algorithms have not yet reached high recognition performance in real-world environments. This paper addresses this problem by proposing a new approach called sparse fingerprint classification algorithm (SFCA). In the training phase, for each enrolled subject, a grid of patches is extracted from each subject's face images in order to construct representative dictionaries. In the testing phase, a grid is extracted from the query image and every patch is transformed into a binary sparse representation using the dictionary, creating a fingerprint of the face. The binary coefficients vote for their corresponding classes and the maximum-vote class decides the identity of the query image. Experiments were carried out on seven widely-used face databases. The results demonstrate that when the size of the data set is small or medium (e.g., the number of subjects is not greater than one hundred), SFCA is able to deal with a larger degree of variability in ambient lighting, pose, expression, occlusion, face size, and distance from the camera than other current state-of-the-art algorithms.

Iris recognition systems are a mature technology that is widely used throughout the world. In identification (as opposed to verification) mode, an iris to be recognized is typically matched against all N enrolled irises. This is the classic "1-to-N search". In order to improve the speed of large-scale identification, a modified "1-to-First" search has been used in some operational systems. A 1-to-First search terminates with the first below-threshold match that is found, whereas a 1-to-N search always finds the best match across all enrollments. We know of no previous studies that evaluate how the accuracy of 1-to-First search differs from that of 1-to-N search. Using a dataset of over 50,000 iris images from 2,800 different irises, we perform experiments to evaluate the relative accuracy of 1-to-First and 1-to-N search. We evaluate how the accuracy difference changes with larger numbers of enrolled irises, and with larger ranges of rotational difference allowed between iris images. We find that False Match error rate for 1-to-First is higher than for 1-to-N, and the the difference grows with larger number of enrolled irises and with larger range of rotation.

Digitally altering, or retouching, face images is a common practice for images on social media, photo sharing websites, and even identification cards when the standards are not strictly enforced. This research demonstrates the effect of digital alterations on the performance of automatic face recognition, and also introduces an algorithm to classify face images as original or retouched with high accuracy. We first introduce two face image databases with unaltered and retouched images. Face recognition experiments performed on these databases show that when a retouched image is matched with its original image or an unaltered gallery image, the identification performance is considerably degraded, with a drop in matching accuracy of up to 25%. However, when images are retouched with the same style, the matching accuracy can be misleadingly high in comparison with matching original images. To detect retouching in face images, a novel supervised deep Boltzmann machine algorithm is proposed. It uses facial parts to learn discriminative features to classify face images as original or retouched. The proposed approach for classifying images as original or retouched yields an accuracy of over 87% on the data sets introduced in this paper and over 99% on three other makeup data sets used by previous researchers. This is a substantial increase in accuracy over the previous state-of-the-art algorithm, which has shown <;50% accuracy in classifying original and retouched images from the ND-IIITD retouched faces database.

Previous researchers have explored various approaches for predicting the gender of a person based on the features of the iris texture. This paper is the first to predict gender directly from the same binary iris code that could be used for recognition. We found that the information for gender prediction is distributed across the iris, rather than localized in particular concentric bands. We also found that using selected features representing a subset of the iris region achieves better accuracy than using features representing the whole iris region. We used the measures of mutual information to guide the selection of bits from the iris code to use as features in gender prediction. Using this approach, with a person-disjoint training and testing evaluation, we were able to achieve 89% correct gender prediction using the fusion of the best features of iris code from the left and right eyes.

The strong, neutral, or weak (SNoW) face impostor pairs problem is intended to explore the causes and impact of impostor face pairs that are inherently strong (easily recognized as nonmatches) or weak (possible false matches). The SNoW technique develops three partitions within the impostor score distribution of a given data set. Results provide evidence that varying degrees of impostor scores impact the overall performance of a face recognition system. This paper extends our earlier work to incorporate improvements regarding outlier detection for partitioning, explores the SNoW concept for the additional modalities of fingerprint and iris, and presents methods for how to begin to reveal the causes of weak impostor pairs. We also show a clear operational difference between strong and weak comparisons as well as identify partition stability across multiple algorithms.

The progression of research into understanding and mitigating the effects of pupil dilation on iris biometrics is at a point where a formalization of the problem is necessary to tie together several research directions and results. Past research has shown that differences in dilation in a (probe, gallery) pair lead to an increase in false non-match rates. Additionally, analysis continues to show that there is at least an approximate linear relationship between increase in dilation difference and degradation in match scores. Lastly, dilation-aware based enrollment techniques have shown to be a promising approach to addressing matching errors due to pupil dilation difference. This paper establishes a framework based on an assumed linear relationship between match scores and dilation difference and shows that the optimal image to enroll based on pupil dilation is the image which has a dilation value near the mean or median depending on the measure of dilation difference.

The use of an artificial replica of a biometric characteristic in an attempt to circumvent a system is an example of a biometric presentation attack. Liveness detection is one of the proposed countermeasures, and has been widely implemented in fingerprint and iris recognition systems in recent years to reduce the consequences of spoof attacks. The goal for the Liveness Detection (LivDet) competitions is to compare software-based iris liveness detection methodologies using a standardized testing protocol and large quantities of spoof and live images. Three submissions were received for the competition Part 1; Biometric Recognition Group de Universidad Autonoma de Madrid, University of Naples Federico II, and Faculdade de Engenharia de Universidade do Porto. The best results from across all three datasets was from Federico with a rate of falsely rejected live samples of 28.6% and the rate of falsely accepted fake samples of 5.7%.

The 2013 Boston Marathon bombing represents a case where automatic facial biometrics tools could have proven invaluable to law enforcement officials, yet the lack of robustness of current tools in unstructured environments limited their utility. In this work, we focus on complications that confound face detection algorithms. We first present a simple multi-pose generalization of the Viola-Jones algorithm. Our results on the Face Detection Data set and Benchmark (FDDB) show that it makes a significant improvement over the state of the art for published algorithms. Conversely, our experiments demonstrate that the improvements attained by accommodating multiple poses can be negligible compared to the gains yielded by normalizing scores and using the most appropriate classifier for uncontrolled data. We conclude with a qualitative evaluation of the proposed algorithm on publicly available images of the Boston Marathon crowds. Although the results of our evaluations are encouraging, they confirm that there is still room for improvement in terms of robustness to out-of-plane rotation, blur and occlusion.

We introduce a method for extracting the social network structure for the persons appearing in a set of video clips. Individuals are unknown, and are not matched against known enrollments. An identity cluster representing an individual is formed by grouping similar-appearing faces from different videos. Each identity cluster is represented by a node in the social network. Two nodes are linked if the faces from their clusters appeared together in one or more video frames. Our approach incorporates a novel active clustering technique to create more accurate identity clusters based on feedback from the user about ambiguously matched faces. The final output consists of one or more network structures that represent the social group(s), and a list of persons who potentially connect multiple social groups. Our results demonstrate the efficacy of the proposed clustering algorithm and network analysis techniques.

Techniques concerning postdisaster assessment from remotely sensed images have been studied by different research communities in the past decade. Such an assessment benefits a range of stakeholders, e.g., government organizations, insurance industry, local communities, and individual homeowners. This work explores detailed damage assessment on an individual building basis by utilizing supervised classification. In contrast with previous research efforts in the field, this work attempts at predicting the type of damages such as missing tiles, collapsed rooftop, and presence of holes, gaps, or cavities. Various existing and novel intensity-, edge-, and color-based features are evaluated. Additionally, preprocessing steps that automatically correct photometric and geometric differences are proposed. Furthermore, a study on the reliability of high-resolution aerial imagery in damage interpretation is conducted by comparing results with the assessment of expert volunteers. Results show that the proposed damage detection framework is very effective and performs at a level similar to that of the experts. This paper concludes that the type and extent of damage to individual rooftops can be identified with good accuracy from high-resolution aerial images. It is envisaged that the automated tools presented in this paper would play a significant role in rapid posthurricane damage estimation and in helping to better manage rescue and recovery missions.

Clustering approaches can alleviate the burden of tagging face identities in ad hoc video and image collections. We introduce a novel semisupervised framework for clustering face patterns into identity groups using minimal human interaction. This technique combines concepts from ensemble clustering and active learning to improve clustering accuracy. The framework actively queries the user for a soft link constraint between each pair of neighboring faces that are ambiguously matched according to the ensemble. We demonstrate the efficacy of our approach with the broadest evaluation of active face clustering algorithms to date. Our evaluations focus on data that is appropriate for human-in-the-loop face recognition, including blurry point-and-shoot videos, images of women seen before and after the application of makeup, and photographs of twins. The results indicate that ensemble-based constrained clustering algorithms are generally more robust to noise than alternative approaches. Finally, we show that the proposed clustering algorithm is more accurate and parsimonious than the current state-of-the-art.

Facial recognition algorithms should be able to operate even when similar-looking individuals are encountered, or even in the extreme case of identical twins. An experimental data set comprised of 17486 images from 126 pairs of identical twins (252 subjects) collected on the same day and 6864 images from 120 pairs of identical twins (240 subjects) with images taken a year later was used to measure the performance on seven different face recognition algorithms. Performance is reported for variations in illumination, expression, gender, and age for both the same day and cross-year image sets. Regardless of the conditions of image acquisition, distinguishing identical twins are significantly harder than distinguishing subjects who are not identical twins for all algorithms.

The presence of a contact lens, particularly a textured cosmetic lens, poses a challenge to iris recognition as it obfuscates the natural iris patterns. The main contribution of this paper is to present an in-depth analysis of the effect of contact lenses on iris recognition. Two databases, namely, the IIIT-D Iris Contact Lens database and the ND-Contact Lens database, are prepared to analyze the variations caused due to contact lenses. We also present a novel lens detection algorithm that can be used to reduce the effect of contact lenses. The proposed approach outperforms other lens detection algorithms on the two databases and shows improved iris recognition performance.

Face images captured by surveillance cameras usually have poor resolution in addition to uncontrolled poses and illumination conditions, all of which adversely affect the performance of face matching algorithms. In this paper, we develop a completely automatic, novel approach for matching surveillance quality facial images to high-resolution images in frontal pose, which are often available during enrollment. The proposed approach uses multidimensional scaling to simultaneously transform the features from the poor quality probe images and the high-quality gallery images in such a manner that the distances between them approximate the distances had the probe images been captured in the same conditions as the gallery images. Tensor analysis is used for facial landmark localization in the low-resolution uncontrolled probe images for computing the features. Thorough evaluation on the Multi-PIE dataset and comparisons with state-of-the-art super-resolution and classifier-based approaches are performed to illustrate the usefulness of the proposed approach. Experiments on surveillance imagery further signify the applicability of the framework. We also show the usefulness of the proposed approach for the application of tracking and recognition in surveillance videos.

While automatic image registration has been extensively studied in other areas of image processing, it is still a complex problem in the framework of remote sensing for disaster response. This problem is difficult because there can be substantial change in the image content between the two images, and the time of day and lighting typically are different between the two images. In this work, we propose a two-step approach to achieve fast and robust registration of before- after-disaster aerial image pairs. First, the images are coarsely registered using a phase-correlation based algorithm. In the second step, transformed images are finely registered by matching features across grids and estimating the perspective transform. Our proposed algorithm is evaluated for robustness, accuracy and speed. It is found to achieve 100% registration success on 23 image pairs which proved challenging to either of the component approaches.

Plastic surgery procedures can significantly alter facial appearance, thereby posing a serious challenge even to the state-of-the-art face matching algorithms. In this paper, we propose a novel approach to address the challenges involved in automatic matching of faces across plastic surgery variations. In the proposed formulation, part-wise facial characterization is combined with the recently popular sparse representation approach to address these challenges. The sparse representation approach requires several images per subject in the gallery to function effectively which is often not available in several use-cases, as in the problem we address in this work. The proposed formulation utilizes images from sequestered non-gallery subjects with similar local facial characteristics to fulfill this requirement. Extensive experiments conducted on a recently introduced plastic surgery database [17] consisting of 900 subjects highlight the effectiveness of the proposed approach.

Several sources of variation in facial appearance that affect face matching performance have long been investigated. The recently introduced GBU challenge problem [1] indicates that there can be significant variation in performance across different partitions of the data, even when the impact of most known factors is eliminated or significantly reduced by the data collection and experimentation protocol. The GBU challenge problem consists of three partitions which are called the Good (easy to match), the Bad (average matching difficulty) and the Ugly (difficult to match). In this paper, we investigate various image and facial characteristics that can account for the observed significant difference in performance across these partitions. Given a match pair, we aim to predict the partition it belongs to. Partial Least Squares (PLS)-based regression is used to perform the prediction task. Our analysis indicates that the match pairs from the three partitions differ from each other in terms of simple but often ignored factors like image sharpness, hue, saturation and extent of facial expressions.

Automatic color balancing approaches for different applications have been studied by different research communities in the past decade. However, in this paper we address color balancing for the purpose of change detection. Images of a scene taken at different times may have variations in lighting and structural content. For such multitemporal images, an ideal color correction approach should be effective at transferring the color palette of the source image to the target image for the unchanged areas while being able to transfer the global color characteristics for the changed area without creating visual artifacts. Towards this goal, we propose a new local color balancing approach that uses adaptive windowing. We evaluated the proposed method against other state-of-the-art ones using a database consisting of aerial image pairs. The test image pairs were taken at different times, under different lighting conditions, and with different scene geometries and camera positions. On this database, our proposed approach outperformed other state-of-the-art algorithms.

Driven by key law enforcement and commercial applications, research on face recognition from video sources has intensified in recent years. The ensuing results have demonstrated that videos possess unique properties that allow both humans and automated systems to perform recognition accurately in difficult viewing conditions. However, significant research challenges remain as most video-based applications do not allow for controlled recordings. In this survey, we categorize the research in this area and present a broad and deep review of recently proposed methods for overcoming the difficulties encountered in unconstrained settings. We also draw connections between the ways in which humans and current algorithms recognize faces. An overview of the most popular and difficult publicly available face video databases is provided to complement these discussions. Finally, we cover key research challenges and opportunities that lie ahead for the field as a whole.

Face recognition performance degrades considerably when the input images are of Low Resolution (LR), as is often the case for images taken by surveillance cameras or from a large distance. In this paper, we propose a novel approach for matching low-resolution probe images with higher resolution gallery images, which are often available during enrollment, using Multidimensional Scaling (MDS). The ideal scenario is when both the probe and gallery images are of high enough resolution to discriminate across different subjects. The proposed method simultaneously embeds the low-resolution probe images and the high-resolution gallery images in a common space such that the distance between them in the transformed space approximates the distance had both the images been of high resolution. The two mappings are learned simultaneously from high-resolution training images using an iterative majorization algorithm. Extensive evaluation of the proposed approach on the Multi-PIE data set with probe image resolution as low as 8 6 pixels illustrates the usefulness of the method. We show that the proposed approach improves the matching performance significantly as compared to performing matching in the low-resolution domain or using super-resolution techniques to obtain a higher resolution test image prior to recognition. Experiments on low-resolution surveillance images from the Surveillance Cameras Face Database further highlight the effectiveness of the approach.

The issue of interoperability between iris sensors is an important topic in large-scale and long-term applications of iris biometric systems. This work compares three commercially available iris sensors and three iris matching systems and investigates the impact of cross-sensor matching on system performance in comparison to single-sensor performance. Several factors which may impact single-sensor and cross-sensor performance are analyzed, including changes in the acquisition environment and differences in dilation ratio between iris images. The sensors are evaluated using three different iris matching algorithms, and conclusions are drawn regarding the interaction between the sensors and the matching algorithm in both the cross-sensor and single-sensor scenarios. Finally, the relative performances of the three sensors are compared.

Periocular biometrics is the recognition of individuals based on the appearance of the region around the eye. Periocular recognition may be useful in applications where it is difficult to obtain a clear picture of an iris for iris biometrics, or a complete picture of a face for face biometrics. Previous periocular research has used either visible-light (VL) or near-infrared (NIR) light images, but no prior research has directly compared the two illuminations using images with similar resolution. We conducted an experiment in which volunteers were asked to compare pairs of periocular images. Some pairs showed images taken in VL, and some showed images taken in NIR light. Participants labeled each pair as belonging to the same person or to different people. Untrained participants with limited viewing times correctly classified VL image pairs with 88% accuracy, and NIR image pairs with 79% accuracy. For comparison, we presented pairs of iris images from the same subjects. In addition, we investigated differences between performance on light and dark eyes and relative helpfulness of various features in the periocular region under different illuminations. We calculated performance of three computer algorithms on the periocular images. Performance for humans and computers was similar.

The paper measures the ability of face recognition algorithms to distinguish between identical twin siblings. The experimental dataset consists of images taken of 126 pairs of identical twins (252 people) collected on the same day and 24 pairs of identical twins (48 people) with images collected one year apart. In terms of both the number of paris of twins and lapsed time between acquisitions, this is the most extensive investigation of face recognition performance on twins to date. Recognition experiments are conducted using three of the top submissions to the Multiple Biometric Evaluation (MBE) 2010 Still Face Track [1]. Performance results are reported for both same day and cross year matching. Performance results are broken out by lighting conditions (studio and outside); expression (neutral and smiling); gender and age. Confidence intervals were generated by a bootstrap method. This is the most detailed covariate analysis of face recognition of twins to date.

Current iris biometric systems enroll a person based on the best eye image taken at the time of acquisition. However, recent research has shown that simply taking the best eye image and ignoring pupil dilation leads to degradations in system performance. In particular, the probability of a false non-match increases when there is a considerable variation in pupil size between the enrolled eye image and the probe eye image. Therefore, methods of enrollment that take into account pupil dilation are needed to ensure reliability of an iris biometric system. Our research examines a strategy to improve system performance by implementing a dilation-aware enrollment phase that chooses eye images based on their respective empirical dilation ratio distribution. We compare our strategy of enrollment to that of the randomly chosen eye images, which is the current enrollment procedure for most iris biometric systems. Our results show that there is a noticeable improvement over the random scenario when pupil dilation is accounted for during the enrollment phase.

Existing 3D face recognition algorithms have achieved high enough performances against public datasets like FRGC v2, that it is difficult to achieve further significant increases in recognition performance. However, the 3D TEC dataset is a more challenging dataset which consists of 3D scans of 107 pairs of twins that were acquired in a single session, with each subject having a scan of a neutral expression and a smiling expression. The combination of factors related to the facial similarity of identical twins and the variation in facial expression makes this a challenging dataset. We conduct experiments using state of the art face recognition algorithms and present the results. Our results indicate that 3D face recognition of identical twins in the presence of varying facial expressions is far from a solved problem, but that good performance is possible.

We introduce the questionable observer detection problem: Given a collection of videos of crowds, determine which individuals appear unusually often across the set of videos. The algorithm proposed here detects these individuals by clustering sequences of face images. To provide robustness to sensor noise, facial expression and resolution variations, blur, and intermittent occlusions, we merge similar face image sequences from the same video and discard outlying face patterns prior to clustering. We present experiments on a challenging video dataset. The results show that the proposed method can surpass the performance of a clustering algorithm based on the VeriLook face recognition software by Neurotechnology both in terms of the detection rate and the false detection frequency.

It has been widely accepted that iris biometric systems are not subject to a template aging effect. Baker et al. [1] recently presented the first published evidence of a template aging effect, using images acquired from 2004 through 2008 with an LG 2200 iris imaging system, representing a total of 13 subjects (26 irises). We report on a template aging study involving two different iris recognition algorithms, a larger number of subjects (43), a more modern imaging system (LG 4000), and over a shorter time-lapse (2 years). We also investigate the degree to which the template aging effect may be related to pupil dilation and/or contact lenses. We find evidence of a template aging effect, resulting in an increase in match hamming distance and false reject rate.

The most common iris biometric algorithm represents the texture of an iris using a binary iris code. Not all bits in an iris code are equally consistent. A bit is deemed fragile if its value changes across iris codes created from different images of the same iris. Previous research has shown that iris recognition performance can be improved by masking these fragile bits. Rather than ignoring fragile bits completely, we consider what beneficial information can be obtained from the fragile bits. We find that the locations of fragile bits tend to be consistent across different iris codes of the same eye. We present a metric, called the fragile bit distance, which quantitatively measures the coincidence of the fragile bit patterns in two iris codes. We find that score fusion of fragile bit distance and Hamming distance works better for recognition than Hamming distance alone. To our knowledge, this is the first and only work to use the coincidence of fragile bit locations to improve the accuracy of matches.

This paper describes the large-scale experimental results from the Face Recognition Vendor Test (FRVT) 2006 and the Iris Challenge Evaluation (ICE) 2006. The FRVT 2006 looked at recognition from high-resolution still frontal face images and 3D face images, and measured performance for still frontal face images taken under controlled and uncontrolled illumination. The ICE 2006 evaluation reported verification performance for both left and right irises. The images in the ICE 2006 intentionally represent a broader range of quality than the ICE 2006 sensor would normally acquire. This includes images that did not pass the quality control software embedded in the sensor. The FRVT 2006 results from controlled still and 3D images document at least an order-of-magnitude improvement in recognition performance over the FRVT 2002. The FRVT 2006 and the ICE 2006 compared recognition performance from high-resolution still frontal face images, 3D face images, and the single-iris images. On the FRVT 2006 and the ICE 2006 data sets, recognition performance was comparable for high-resolution frontal face, 3D face, and the iris images. In an experiment comparing human and algorithms on matching face identity across changes in illumination on frontal face images, the best performing algorithms were more accurate than humans on unfamiliar faces.

The ten papers in this special issue focus on recent advances in biometrics. Four of the papers are revised and extended versions of papers presented in the closing session of the BTAS 08 conference.

We take advantage of the temporal continuity in an iris video to improve matching performance using signal-level fusion. From multiple frames of a frontal iris video, we create a single average image. For comparison, we reimplement three score-level fusion methods (Ma, Krichen, and Schmid). We find that our signal-level fusion ofNimages performs better than Ma's or Krichen's score-level fusion methods ofNHamming distance scores. Our signal-level fusion performs comparably to Schmid's log-likelihood method of score-level fusion, and our method achieves this performance using less computation time. We compare our signal fusion method with another new method: a multigallery, multiprobe method involving score-level fusion ofN2Hamming distances. The multigallery, multiprobe score fusion has slightly better recognition performance, while the signal fusion has significant advantages in memory and computation requirements. No published prior work has shown any advantage of the use of video over still images in iris biometrics.

The five papers in this special section illustrate the breadth of activities in current biometrics research.

While recent years have seen progress in face recognition from 3D images, nonfrontal head pose is still a challenge to existing techniques. We introduce a new system for 3D face recognition that is robust to facial pose variation. Large degrees of facial pose variation may lead to a significant fraction of the features visible in frontal images being occluded. High accuracy automatic feature and pose detection is performed by a new technique called rotated profile signatures (RPS). Experiments are performed on the largest available database of 3D faces acquired under varying pose. This database contains over 7,300 total images of 406 unique subjects gathered at the University of Notre Dame. Experimental results show that the RPS detection algorithm is capable of performing nose detection with greater than 96.5% accuracy across the pose variation represented in the data set used.

Complex simulations can generate very large amounts of data stored disjointedly across many local disks. Learning from this data can be problematic due to the difficulty of obtaining labels for the data. We present an algorithm for the application of semi-supervised learning on disjoint data generated by complex simulations. Our semi-supervised technique shows a statistically significant accuracy improvement over supervised learning using the same underlying learning algorithm and requires less labeled data for comparable results.

We describe an ensemble approach to learning salient regions from data partitioned according to the distributed processing requirements of large-scale simulations. The volume of the data is such that classifiers can train only on data local to a given partition. Classes will likely be missing from some, or even most, partitions. We combine a fast ensemble learning algorithm with scaled probabilistic majority voting in order to learn an accurate classifier from such data. We order predicted regions to increase the likelihood that most of the initial set of presented regions are salient. Results from a simulated casing being dropped show that regions of interest are successfully identified and ordered. This approach is much faster than manually browsing and visualizing terabyte or larger simulations to find regions of interest.

In this paper, we introduce a new system for 3D face recognition based on the fusion of results from a committee of regions that have been independently matched. Experimental results demonstrate that using 28 small regions on the face allow for the highest level of 3D face recognition. Score-based fusion is performed on the individual region match scores and experimental results show that the Borda count and consensus voting methods yield higher performance than the standard sum, product, and min fusion rules. In addition, results are reported that demonstrate the robustness of our algorithm by simulating large holes and artifacts in images. To our knowledge, no other work has been published that uses a large number of 3D face regions for high-performance face matching. Rank one recognition rates of 97.2% and verification rates of 93.2% at a 0.1% false accept rate are reported and compared to other methods published on the face recognition grand challenge v2 data set.

Due to its semi-rigid shape and robustness against change over time, the ear has become an increasingly popular biometric feature. It has been shown that combining individual biometric methods into multi-biometric systems improves recognition. What features should be used, how they should be captured, what algorithms should be used, and how they should be combined are all open questions. In this paper, we discuss several existing methods of combination and the recognition rates of each.

We experimentally evaluate bagging and seven other randomization-based approaches to creating an ensemble of decision tree classifiers. Statistical tests were performed on experimental results from 57 publicly available data sets. When cross-validation comparisons were tested for statistical significance, the best method was statistically more accurate than bagging on only eight of the 57 data sets. Alternatively, examining the average ranks of the algorithms across the group of data sets, we find that boosting, random forests, and randomized trees are statistically significantly better than bagging. Because our results suggest that using an appropriate ensemble size is important, we introduce an algorithm that decides when a sufficient number of classifiers has been created for an ensemble. Our algorithm uses the out-of-bag error estimate, and is shown to result in an accurate ensemble for those methods that incorporate bagging into the construction of the ensemble.

We note that the images in the CASIA version 1.0 iris dataset have been edited so that the pupil area is replaced by a circular region of uniform intensity. We recommend that this dataset is no longer used in iris biometrics research, unless there this a compelling reason that takes into account the nature of the images. In addition, based on our experience with the Iris Challenge Evaluation (ICE) 2005 technology development project, we make recommendations for reporting results of iris recognition experiments.

Previous works have shown that the ear is a promising candidate for biometric identification. However, in prior work, the preprocessing of ear images has had manual steps and algorithms have not necessarily handled problems caused by hair and earrings. We present a complete system for ear biometrics, including automated segmentation of the ear in a profile view image and 3D shape matching for recognition. We evaluated this system with the largest experimental study to date in ear biometrics, achieving a rank-one recognition rate of 97.8 percent for an identification scenario and an equal error rate of 1.2 percent for a verification scenario on a database of 415 subjects and 1,386 total probes.

We present an ensemble learning approach that achieves accurate predictions from arbitrarily partitioned data. The partitions come from the distributed processing requirements of a large scale simulation where the volume of the data is such that classifiers can train only on data local to a given partition. As a result of the partition reflecting the need for efficient simulation analysis, rather than the needs of data mining, the class statistics vary across partitions; indeed some classes will likely be absent from some partitions. We combine a fast ensemble learning algorithm with majority voting to generate an accurate working model of the simulation. Results from several simulations show that regions of interest are successfully identified in spite of training set class imbalances. Accuracy is analyzed both at the level of nodes in the simulation data structure, and in terms of higher-level regions of interest. It is shown that over 98% of salient regions are found in independent test sets. Hence, this approach will be a significant time saver for simulation users and developers

An algorithm is proposed for 3D face recognition in the presence of varied facial expressions. It is based on combining the match scores from matching multiple overlapping regions around the nose. Experimental results are presented using the largest database employed to date in 3D face recognition studies, over 4,000 scans of 449 subjects. Results show substantial improvement over matching the shape of a single larger frontal face region. This is the first approach to use multiple overlapping regions around the nose to handle the problem of expression variation.

This work examines face recognition using normal intensity images, infrared images, three-dimensional shape, and combinations of these. We compare the performance improvement obtained by combining three-dimensional or infrared with normal intensity images (a "multimodal" approach) to the performance improvement obtained by using multiple intensity images (a "multisample" approach). Combining results from different types of imagery gives significantly higher recognition rates than are obtained by using a single intensity image. However, significantly higher recognition rates are also obtained by combining results from multiple intensity images. Overall, initial results indicate that, using an "eigen-face" recognition algorithm and weighted score fusion, multisample techniques can result in a performance increase comparable to that of multimodal techniques

Random subspaces are a popular ensemble construction technique that improves the accuracy of weak classifiers. It has been shown, in different domains, that random subspaces combined with weak classifiers such as decision trees and nearest neighbor classifiers can provide an improvement in accuracy. In this paper, we apply the random subspace methodology to the 2-D face recognition task. The main goal of the paper is to see if the random subspace methodology can do as well, if not better, than the single classifier constructed on the tuned face space. We also propose the use of a validation set for tuning the face space, to avoid bias in the accuracy estimation. In addition, we also compare the random subspace methodology to an ensemble of subsamples of image data. This work shows that a random subspaces ensemble can outperform a well-tuned single classifier for a typical 2-D face recognition problem. The random subspaces approach has the added advantage of requiring less careful tweaking.

Over the last couple of years, face recognition researchers have been developing new techniques. These developments are being fueled by advances in computer vision techniques, computer design, sensor design, and interest in fielding face recognition systems. Such advances hold the promise of reducing the error rate in face recognition systems by an order of magnitude over Face Recognition Vendor Test (FRVT) 2002 results. The face recognition grand challenge (FRGC) is designed to achieve this performance goal by presenting to researchers a six-experiment challenge problem along with data corpus of 50,000 images. The data consists of 3D scans and high resolution still imagery taken under controlled and uncontrolled conditions. This paper describes the challenge problem, data corpus, and presents baseline performance and preliminary results on natural statistics of facial imagery.

Random subspaces are a popular ensemble construction technique that improves the accuracy of weak classifiers. It has been shown, in different domains, that random subspaces combined with weak classifiers such as decision trees and nearest neighbor classifiers can provide an improvement in accuracy. In this paper, we apply the random subspace methodology to the 2D face recognition task. The main goal of the paper is to see if the random subspace methodology can improve the performance of the face recognition system given the high dimensional data, temporal, and distribution variant data. We used two different datasets to evaluate the methodology. One dataset comprises of completely unique subjects for testing, and the other dataset comprises of the same subjects (both in training and testing) but images in the test set are captured at different times under different conditions.

We report on the largest experimental study to date in multimodal 2D+3D face recognition, involving 198 persons in the gallery and either 198 or 670 time-lapse probe images. PCA-based methods are used separately for each modality and match scores in the separate face spaces are combined for multimodal recognition. Major conclusions are: 1) 2D and 3D have similar recognition performance when considered individually, 2) combining 2D and 3D results using a simple weighting scheme outperforms either 2D or 3D alone, 3) combining results from two or more 2D images using a similar weighting scheme also outperforms a single 2D image, and 4) combined 2D+3D outperforms the multiimage 2D result. This is the first (so far, only) work to present such an experimental control to substantiate multimodal performance improvement.

Identification of people by analysis of gait patterns extracted from video has recently become a popular research problem. However, the conditions under which the problem is "solvable" are not understood or characterized. To provide a means for measuring progress and characterizing the properties of gait recognition, we introduce the HumanID Gait Challenge Problem. The challenge problem consists of a baseline algorithm, a set of 12 experiments, and a large data set. The baseline algorithm estimates silhouettes by background subtraction and performs recognition by temporal correlation of silhouettes. The 12 experiments are of increasing difficulty, as measured by the baseline algorithm, and examine the effects of five covariates on performance. The covariates are: change in viewing angle, change in shoe type, change in walking surface, carrying or not carrying a briefcase, and elapsed time between sequences being compared. Identification rates for the 12 experiments range from 78 percent on the easiest experiment to 3 percent on the hardest. All five covariates had statistically significant effects on performance, with walking surface and time difference having the greatest impact. The data set consists of 1,870 sequences from 122 subjects spanning five covariates (1.2 Gigabytes of data). The gait data, the source code of the baseline algorithm, and scripts to run, score, and analyze the challenge experiments are available at http://www.GaitChallenge.org. This infrastructure supports further development of gait recognition algorithms and additional experiments to understand the strengths and weaknesses of new algorithms. The more detailed the experimental results presented, the more detailed is the possible meta-analysis and greater is the understanding. It is this potential from the adoption of this challenge problem that represents a radical departure from traditional computer vision research methodology.

Collobert, Bengio, and Bengio (2002) recently introduced a novel approach to using a neural network to provide a class prediction from an ensemble of support vector machines (SVMs). This approach has the advantage that the required computation scales well to very large data sets. Experiments on the Forest Cover data set show that this parallel mixture is more accurate than a single SVM, with 90.72% accuracy reported on an independent test set. Although this accuracy is impressive, their article does not consider alternative types of classifiers. We show that a simple ensemble of decision trees results in a higher accuracy, 94.75%, and is computationally efficient. This result is somewhat surprising and illustrates the general value of experimental comparisons using different types of classifiers.

Previous performance evaluation of range image segmentation algorithms has depended on manual tuning of algorithm parameters, and has lacked a basis for a test of the significance of differences between algorithms. We present an automated framework for evaluating the performance of range image segmentation algorithms. Automated tuning of algorithm parameters in this framework results in performance as good as that previously obtained with careful manual tuning by the algorithm developers. Use of multiple training and test sets of images provides the basis for a test of the significance of performance differences between algorithms. The framework implementation includes range images, ground truth overlays, program source code, and shell scripts. This framework should a) make it possible to objectively and reliably compare the performance of range image segmentation algorithms; b) allow informed experimental feedback for the design of improved segmentation algorithms. The framework is demonstrated using range images, but in principle it could be used to evaluate region segmentation algorithms for any type of image.

We experimentally evaluate randomization-based approaches to creating an ensemble of decision-tree classifiers. Unlike methods related to boosting, all of the eight approaches considered here create each classifier in an ensemble independently of the other classifiers. Experiments were performed on 28 publicly available datasets, using C4.5 release 8 as the base classifier. While each of the other seven approaches has some strengths, we find that none of them is consistently more accurate than standard bagging when tested for statistical significance.

Bio-informatics data sets may be large in the number of examples and/or the number of features. Predicting the secondary structure of proteins from amino acid sequences is one example of high dimensional data for which large training sets exist. The data from the KDD Cup 2001 on the binding of compounds to thrombin is another example of a very high dimensional data set. This type of data set can require significant computing resources to train a neural network. In general, decision trees will require much less training time than neural networks. There have been a number of studies on the advantages of decision trees relative to neural networks for specific data sets. There are often statistically significant, though typically not very large, differences. Here, we examine one case in which a neural network greatly outperforms a decision tree; predicting the secondary structure of proteins. The hypothesis that the neural network learns important features of the data through its hidden units is explored by a using a neural network to transform data for decision tree training. Experiments show that this explains some of the performance difference, but not all. Ensembles of decision trees are compared with a single neural network. It is our conclusion that the problem of protein secondary structure prediction exhibits some characteristics that are fundamentally better exploited by a neural network model.

Researchers have suggested that the ear may have advantages over the face for biometric recognition. Our previous experiments with ear and face recognition, using the standard principal component analysis approach, showed lower recognition performance using ear images. We report results of similar experiments on larger data sets that are more rigorously controlled for relative quality of face and ear images. We find that recognition performance is not significantly different between the face and the ear, for example, 70.5 percent versus 71.6 percent, respectively, in one experiment. We also find that multimodal recognition using both the ear and face results in statistically significant improvement over either individual biometric, for example, 90.9 percent in the analogous experiment.

In this paper we present a method to compute the egomotion of a range camera using the space envelope. The space envelope is a geometric model that provides more information than a simple segmentation for correspondences and motion estimation. We describe a novel variation of the maximal matching algorithm that matches surface normals to find correspondences. These correspondences are used to compute rotation and translation estimates of the egomotion. We demonstrate our methods on two image sequences containing 70 images. We also discuss the cases where our methods fail, and additional possible methods for exploiting the space envelope.

It has been asserted that, using traditional pruning methods, growing decision trees with increasingly larger amounts of training data will result in larger tree sizes even when accuracy does not increase. With regard to error-based pruning, the experimental data used to illustrate this assertion have apparently been obtained using the default setting for pruning strength; in particular, using the default certainty factor of 25 in the C4.5 decision tree implementation. We show that, in general, an appropriate setting of the certainty factor for error-based pruning will cause decision tree size to plateau when accuracy is not increasing with more training data.

Open source software is making a large impact on many aspects of society including the business community, the computing industry, the entertainment industry and higher education. The computer science education community has been quiet about issues of open source versus closed source and the role of open source code in the advancement of information technology. A survey of recent issues of SIGCSE Bulletin and SIGCSE conference proceedings shows little attention to the role open source software should play in computer science education. We are here to raise the question: "What are the social and ethical responsibilities of computer science faculty regarding open source software?"One set of issues concerns the use of open source software in teaching and the use of open source development models in the teaching of software development. Some basic questions that arise include "Should analysis of open source (and possible contributions to it) be the subject of class assignments?" and "Should open source software development models be taught?"A second set of issues concerns the use of open source software in support of teaching (e.g., using Linux as your standard platform). Some basic questions that arise include "Should we use open source software to support teaching?" and "Are these faculty ethically obligated to make contributions to the open source software base?"In this panel we will identify many of the parties involved in the ethical and social issues surrounding the use of open source in teaching and in the support of teaching, and we will identify the rights and responsibilities we, as faculty, have to the various parties. This panel will initiate a discussion that will identify additional parties and our further professional obligations.

An approach to the construction of classifiers from imbalanced datasets is described. A dataset is imbalanced if the classification categories are not approximately equally represented. Often real-world data sets are predominately composed of ``normal'' examples with only a small percentage of ``abnormal'' or ``interesting'' examples. It is also the case that the cost of misclassifying an abnormal (interesting) example as a normal example is often much higher than the cost of the reverse error. Under-sampling of the majority (normal) class has been proposed as a good means of increasing the sensitivity of a classifier to the minority class. This paper shows that a combination of our method of over-sampling the minority (abnormal) class and under-sampling the majority (normal) class can achieve better classifier performance (in ROC space) than only under-sampling the majority class. This paper also shows that a combination of our method of over-sampling the minority class and under-sampling the majority class can achieve better classifier performance (in ROC space) than varying the loss ratios in Ripper or class priors in Naive Bayes. Our method of over-sampling the minority class involves creating synthetic minority class examples. Experiments are performed using C4.5, Ripper and a Naive Bayes classifier. The method is evaluated using the area under the Receiver Operating Characteristic curve (AUC) and the ROC convex hull strategy.

Bagging forms a committee of classifiers by bootstrap aggregation of training sets from a pool of training data. A simple alternative to bagging is to partition the data into disjoint subsets. Experiments on various datasets show that, given the same size partitions and bags, disjoint partitions result in better performance than bootstrap aggregates (bags). Many applications (e.g., protein structure prediction) involve the use of datasets that are too large to handle in the memory of a typical computer. Our results indicate that, in such applications, the simple approach of creating a committee of classifiers from disjoint partitions is preferred over the more complex approach of bagging.

Safety-critical software is a core topic in courses on "ethics and computing" or "computers and society," as well as in software engineering courses. The Reagan-era Strategic Defense initiative (SDI) was the focus of a great deal of technical argument relating to design and testing of safety-critical software. Most of today's students have no familiarity with the substance of the SDI arguments. However, with presidents Clinton and Bush considering various versions of a national missile defense system, the topic has again become quite relevant and motivated by current events. This paper describes a curriculum module developed around a Reagan-era SDI debate on the theme-"Star wars: can the computing requirements be met?" This module should be appropriate for use in ethics-related or software-engineering-related courses taught in undergraduate Information Systems, Information Technology, Computer Science, or Computer Engineering programs.

This paper presents an evaluation of edge detector performance. We use the task of structure from motion (SFM) as a "black box" through which to evaluate the performance of edge detection algorithms. Edge detector goodness is measured by how accurately the SFM could recover the known structure and motion from the edge detection of the image sequences. We use a variety of real image sequences with ground truth to evaluate eight different edge detectors from the literature. Our results suggest that ratings of edge detector performance based on pixel-level metrics and on the SFM are well correlated and that detectors such as the Canny detector and Heitger detector offer the best performance.

A range image segmentation contest was organized in conjunction with ICPR'2000. The goal is to continue the effort of experimentally evaluating range image segmentation algorithms initiated by Hoover et al. (1996) and Powell et al. (1998). This paper summarizes the results of the contest.

We have developed an automated framework for performance evaluation of curved-surface range image segmentation algorithms. Enhancements over our previous work include automated training of parameter values, correcting the artifact problem in K/sup 2/T scanner images, and acquisition of images of the same scenes from different range scanners. The image dataset includes planar, spherical, cylindrical, conical, and toroidal surfaces. We have evaluated the automated parameter tuning technique and found that it compares favorably with manual parameter tuning. We present initial results from comparing curved-surface segmenters by Besl and Jain (1988) and by Jiang and Bunke (1998).

Workshops on the theme of “Teaching Ethics and Computing” were sponsored by the National Science Foundation's Undergraduate Faculty Enhancement program. This paper outlines the results of the workshops, available at marathon. csee.usf.edu/~kwb/naf-ufe/. Preparation for the workshops included a survey of videos that are potentially useful in teaching ethics and computing. This paper reviews some of the “best of” these videos.

This seminar/workshop on ethics and the social impact in computer science, supported by studies of the applicability of alternative teaching and learning strategies, is targeted towards doctoral candidates in computer science whose life-goal is to teach in a university or college setting. Based on the concept of “ethics across the curriculum” the seminar/workshop will prepare future faculty to incorporate ethical and social impact concerns in their technical courses. At the same time they will be exposed to modern teaching and learning techniques that will assist them in making a good start in their teaching careers.

Simulation problems in the DOE ASCI program generate visualization datasets more than a terabyte in size. The practical difficulties in visualizing such datasets motivate the desire for automatic recognition of salient events. We have developed a parallel decision tree classifier for use in this context. Comparisons to ScalParC, a previous attempt to build a fast parallelization of a decision tree classifier, are provided. Our parallel classifier executes on the "ASCI Red" supercomputer. Experiments demonstrate that datasets too large to be processed on a single processor can be efficiently handled in parallel, and suggest that there need not be any decrease in accuracy relative to a monolithic classifier constructed on a single processor.

We have developed an automated framework for objectively evaluating the performance of region segmentation algorithms. This framework is demonstrated with range image data sets, but is applicable to any type of imagery. Parameters of the segmentation algorithm are tuned using training images. Images and source code for the training process care publicly available. The trained parameters are then used to evaluate the algorithm on a (sequestered) test set. The primary performance metric is the average number of correctly segmented regions. Statistical tests are used to determine the significance of performance improvement over a baseline algorithm.

A method is demonstrated to evaluate edge detector performance using receiver operating characteristic curves. It involves matching edges to manually specified ground truth to count true positive and false positive detections. Edge detector parameter settings are trained and tested on different images, and aggregate test ROC curves presented for two sets of 10 images. The performance of eight different edge detectors is compared. The Canny and Heitger detectors provide the best performance.

This paper presents a method of evaluating unsupervised texture segmentation algorithms. The control scheme of texture segmentation has been conceptualized as two modular processes: (1) feature computation and (2) segmentation of homogeneous regions based on the feature values. Three feature extraction methods are considered: gray level co-occurrence matrix, Laws' texture energy and Gabor multi-channel filtering. Three segmentation algorithms are considered: fuzzy c-means clustering, square-error clustering and split-and-merge. A set of 35 real scene images with manually-specified ground truth was compiled. Performance is measured against ground truth on real images using region-based and pixel-based performance metrics.

This paper presents a methodology and results of evaluating edge detection algorithms using an object recognition task. A dataset consisting of 37 real images with 5 different jeep-like vehicles is used. Five edge detectors are compared using ROC curve analysis. The Heitger detector gives the best results. The work is being extended to include more images and a train-and-test style evaluation.

——————————F—————————— omputer vision emerged as a subfield in computer science and in electrical engineering in the 1960s. Two main motivations for research in computer vision are to develop algorithms to solve vision problems and to understand and model the human visual system. It turns out that finding satisfactory answers to either motivation is significantly harder than common wisdom initially assumed. Research in computer vision has actively continued to the current time. Most of the research in the computer vision and pattern recognition community is focused on developing solutions to vision problems. With three decades of research behind current efforts and with the availability of powerful, inexpensive computers, there is a common belief that computer vision is poised to deliver reliable solutions. The area of empirical evaluation of computer vision algorithms is developing the methods and tools for measuring the ability of algorithms to meet requirements to be fielded, for determining the state-of-the-art, and for pointing out future research directions. The goal of this special theme section of IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) is to highlight progress in empirical evaluation and identify it as a maturing area of computer vision. Out of 18 submissions, three were accepted for this special section. In addition, one submission was accepted to appear in a regular issue, and two others are being revised for consideration as regular papers. “Filtering for Supervised Texture Segmentation: A Comparative Study” by T. Randen and J.H. Husoy presents a comparative study of methods for texture classification. The emphasis of the study is filtering methods from signal processing. Most major filtering approaches are evaluated. For reference, a statistical algorithm and a model-based algorithm are also evaluated. The paper presents performance results on a number of mosaic texture images. In a first for PAMI, the raw image files for these images are being made available as part of the electronic version of the paper. (The electronic version of the paper is part of the Computer Society’s digital library, accessible online at www.computer.org.) It is hoped that future papers on texture segmentation will take advantage of this in order to present directly comparable experimental results. “Performance Evaluation and Analysis of Monocular Building Extraction From Aerial Imagery” by J.A. Shufelt evaluates end-to-end performance of four systems on their ability to extract buildings from 83 aerial images of 18 sites. The methodology allows for an examination of traditional assumptions made in designing algorithms that extract buildings from monocular imagery. “Evaluation of Methods for Ridge and Valley Detection” by A.M. Lopez, F. Lumbreras, and J. Serrat evaluates ridge and valley detectors. The authors discuss what are desirable properties of ridge and valley detectors and the methods for measuring desirable properties. Then they present an evaluation using these methods. We hope the papers in this special section are interesting and present challenges for future researchers.

This paper presents a task-oriented evaluation methodology for edge detectors. Performance is measured based on the task of structure from motion. Eighteen real image sequences from 2 different scenes varying in the complexity and scenery types are used. The task-level ground truth for each image sequence is manually specified in terms of the 3D motion and structure. An automated tool computes the accuracy of the motion and structure achieved using the set of edge maps. Parameter sensitivity and execution speed are also analyzed. Four edge detectors are compared. All implementations and data sets are publicly available.

This work focuses on creating a framework for objectively evaluating the performance of range image segmentation algorithms. The algorithms are evaluated in terms of correct segmentation, over- and under-segmentation, missed and noise regions. A set of images with ground truth was created for this work. The images were captured using a structured light scanner. Images used in the evaluation contain planar, spherical, cylindrical, toroidal and conical surface patches. The different surface patches in each image were manually identified to establish ground truth for performance evaluation. Two segmentation algorithms from the literature are compared.

We present a method of evaluating edge detector performance based on empirical receiver-operating characteristic (ROC) curves. The approach is based on using real images, for which the ground truth has been manually specified. The ROC curve for a given edge detector summarizes its range of tradeoffs between true positive and false positive edge pixels, as determined by comparing the detected edge pixels to the specified ground truth. Aggregate ROC curves are constructed for groups of images. To give a fairer evaluation relative to typical application use, a form of "train and test" methodology has been developed.

Consider a labeled data set of 1 terabyte in size. A salient subset might depend upon the users interests. Clearly, browsing such a large data set to find interesting areas would be very time consuming. An intelligent agent which, for a given class of user, could provide hints on areas of the data that might interest the user would be very useful. Given large data sets having categories of salience for different user classes attached to the data in them, these labeled sets of data can be used to train a decision tree to label unseen data examples with a category of salience. The training set will be much larger than usual. This paper describes an approach to generating the rules for an agent from a large training set. A set of decision trees are built in parallel on tractable size training data sets which are a subset of the original data. Each learned decision tree will be reduced to a set of rules, conflicting rules resolved and the resultant rules merged into one set. Results from cross validation experiments on a data set suggest this approach may be effectively applied to large sets of data.

The construction of a surface model from range data may be undertaken at any point in a continuum of scales that reflects the level of detail of the resulting model. This continuum relates the construction parameters to the scale of the model. We propose methods to dynamically reprocess range data at different scales. The construction result from a single scale is automatically evaluated, causing reconstruction at a different scale when user-defined criteria are not met. We demonstrate our methods in constructing a planar b-rep space envelope (a scene representation) for over 400 range images. The experiments demonstrate the ability to construct 100 percent valid models, with the scale of detail within specified requirements.

A new method for evaluating edge detection algorithms is presented and applied to measure the relative performance of algorithms by Canny, Nalwa-Binford, Iverson-Zucker, Bergholm, and Rothwell. The basic measure of performance is a visual rating score which indicates the perceived quality of the edges for identifying an object. The process of evaluating edge detection algorithms with this performance measure requires the collection of a set of gray-scale images, optimizing the input parameters for each algorithm, conducting visual evaluation experiments and applying statistical analysis methods. The novel aspect of this work is the use of a visual task and real images of complex scenes in evaluating edge detectors. The method is appealing because, by definition, the results agree with visual evaluations of the edge images.

This paper presents a method for combining classifiers that uses estimates of each individual classifier's local accuracy in small regions of feature space surrounding an unknown test sample. An empirical evaluation using five real data sets confirms the validity of our approach compared to some other combination of multiple classifiers algorithms. We also suggest a methodology for determining the best mix of individual classifiers.

Receiver operating characteristic (ROC) analysis is an established method of measuring diagnostic performance in medical imaging studies. Traditionally, artificial neural networks (ANN's) have been applied as a classifier to find one "best" detection rate. Recently researchers have begun to report ROC curve results for ANN classifiers. The current standard method of generating ROC curves for an ANN is to vary the output node threshold for classification. In this work, we propose a different technique for generating ROC curves for a two-class ANN classifier. We show that this new technique generates better ROC curves in the sense of having greater area under the ROC curve (AUC), and in the sense of being composed of a better distribution of operating points.

The purpose of this paper is to describe a new (to computer vision) experimental framework which allows us to make quantitative comparisons using subjective ratings made by people. This approach avoids the issue of pixel-level ground truth. As a result, it does not allow us to make statements about the frequency of false positive and false negative errors at the pixel level. Instead, using experimental design and statistical techniques borrowed from Psychology, we make statements about whether the outputs of one edge detector are rated statistically significantly higher than the outputs of another. This approach offers itself as a nice complement to signal-based quantitative measures. Also, the evaluation paradigm in this paper is goal oriented; in particular, we consider edge detection in the context of object recognition. The human judges rate the edge, detectors based on how well the capture the salient features of real objects. So far, edge detection modules have been designed and evaluated in isolation, except for the recent work by Ramesh and Haralick (1992). The only prior work (that we are aware of) which also uses humans to rate image algorithms is that of Reeves and Higdon (1995). They use human ratings to decide on regularization parameters of image restoration. Fram and Deutch (1975) also used human subjects, however, the focus was on human versus machine performance rather than using human ratings to compare different edge detectors. The use of human judges to rate image outputs mist be approached systematically. Experiments must be designed and conducted carefully, and results interpreted with appropriate statistical tools. The use of statistical analysis in vision system performance characterization has been rare. The only prior work in the area that we are aware of is that of Nair et al. (1995), who used statistical ranking procedures to compare neural network based object recognition systems.

Combination of multiple classifiers (CMC) has recently drawn attention as a method of improving classification accuracy. This paper presents a method for combining classifiers that use estimates of each individual classifier's local accuracy in small regions of feature space surrounding an unknown test sample. Only the output of the most locally accurate classifier is considered. We address issues of (1) optimization of individual classifiers, and (2) the effect of varying the sensitivity of the individual classifiers on the CMC algorithm. Our algorithm performs better on data from a real problem in mammogram image analysis than do other recently proposed CMC techniques.

A methodology for evaluating range image segmentation algorithms is proposed. This methodology involves (1) a common set of 40 laser range finder images and 40 structured light scanner images that have manually specified ground truth and (2) a set of defined performance metrics for instances of correctly segmented, missed, and noise regions, over- and under-segmentation, and accuracy of the recovered geometry. A tool is used to objectively compare a machine generated segmentation against the specified ground truth. Four research groups have contributed to evaluate their own algorithm for segmenting a range image into planar patches.

Knowledge about required functionality of an object can be used as an effective representation for a generic object category (e.g. "chair", "cup", or "hammer"). This approach to object representation and recognition has recently become an active area of research. We explore a scenario in which a robot senses the environment to obtain an initial partial shape model of an object. If the information in this initial model is not sufficient to hypothesize a possible function for the object, then additional view(s) may be suggested. Once a possible function is hypothesized, a plan is formulated for interacting with the object to confirm that its material properties are compatible with the hypothesized function. The module for reasoning about partial shape models has been evaluated on over 200 shape models acquired from range images. The module for carrying out a function verification plan has been evaluated in a simulated environment using the ThingWorld (TW) system.

Functionality-based recognition systems recognize objects at the category level by reasoning about how well the objects support the expected function. Such systems naturally associate a ``measure of goodness'' or ``membership value'' with a recognized object. This measure of goodness is the result of combining individual measures, or membership values, from potentially many primitive evaluations of different properties of the object's shape. A membership function is used to compute the membership value when evaluating a primitive of a particular physical property of an object. In previous versions of a recognition system known as Gruff, the membership function for each of the primitive evaluations was hand-crafted by the system designer. In this paper, we provide a learning component for the Gruff system, called Omlet, that automatically learns membership functions given a set of example objects labeled with their desired category measure. The learning algorithm is generally applicable to any problem in which low-level membership values are combined through an and-or tree structure to give a final overall membership value.

A new approach is presented for extracting an explicit 3D shape model from a single range image. One novel aspect is that the model represents both observed object surfaces, and surfaces which bound the volume of occluded space. Another novel aspect is that the approach does not require that the range image segmentation be perfect. The low-level segmentation may be such that the model-building process encounters topology versus geometry conflicts. The model-building process is designed to be "fail soft" in the face of such problems. The portion of the 3D model where a problem presents itself is "glued" together in a manner meant to minimize the disturbance in the 3D shape. The goal is to produce a valid boundary-representation which can be processed by higher-level routines. A third novel aspect of this work is that the implementation has been evaluated on over 200 real range images of polyhedral objects, with no operator intervention and all parameters held constant, and obtained a 97% success rate in creating valid b-reps.>

This paper discusses the applications of hyperquadric models in computer vision and focuses on their recovery from range data. Hyperquadrics are volumetric shape models that include superquadrics as a special case. A hyperquadric model can be composed of any number of terms and its geometric bound is an arbitrary convex polytope. Thus, hyperquadrics can model more complex shapes than superquadrics. Hyperquadrics also possess many other advantageous properties (compactness, semilocal control, and intuitive meaning). Our proposed algorithm starts with a rough fit using only six terms in 3D (four in 2D) and adds additional terms as necessary to improve fitting. Suitable constraints are used to ensure proper convergence. Experimental results with real 2D and 3D data are presented.>

This paper examines techniques for the efficient use of high dimensional feature sets in the detection of micro-calcifications in mammograms. The paper focuses on techniques for dimensionality reduction and discriminant analysis. The paper examines the use of principal components and Fisher's linear discriminant for dimensionality reduction along with parametric and nonparametric statistical techniques for discriminant analysis.>

Receiver operating characteristic (ROC) analysis is an established method of measuring diagnostic performance in medical imaging studies. An ROC curve characterizes the inherent tradeoff between true positive and false positive detection rates in a classification system. Traditionally, artificial neural networks (ANNs) have been applied as a classifier to find one "best" partition of feature space, and therefore a single detection rate. This work proposes and evaluates a new technique for generating an ROC curve for a 2-class ANN classifier. We show that the new technique generates significantly better ROC curves than the method currently used to generate ROCs for ANNs.>

Previous work on the recognition of objects by reasoning about their functionality has not dealt with objects that have moving parts. In this paper we introduce a scenario in which object recognition is accomplished by first deriving an articulated shape model from an observed sequence of 3-D shapes and by then reasoning about the possible functionality of the articulated shape model.

This paper describes a definition of the range image segmentation (of polyhedral scenes) problem, a data set to use in evaluation, a method for specifying ground truth, and a set of metrics to classify segmentation results against ground truths.>

Superquadric is an implicit model which was recently introduced and successfully applied in computer vision research. The authors introduce its generalization, the use of the hyperquadric models, for computer vision applications, and focus on its utilization for shape recovery from range data. The hyperquadric model can be composed of any number of terms. Its geometric bound is an arbitrary convex polyhedron, and thus it can describe more complex shapes than the superquadric. A fitting method is proposed that starts with a rough fit with only two terms in the 2-D case or three terms in the 3-D case, and then adds additional terms to improve the fit. The experiments indicate that the use of hyperquadrics is a promising paradigm for shape representation and recovery in computer vision.>

Representation schemes traditionally used in model-based vision are contrasted with the “function-based” representation scheme. A system which utilizes function-based representation has been implemented and tested, using the object category “chair” for case study. Function-based description is used to recognize classes and identify subclasses of known categories of objects, even if the specific object has never been encountered previously. Interpretation of the functionality of an object is accomplished through qualitative reasoning about its 3-D shape. During the recognition process, evidence is gathered as to how well the functional requirements are satisfied by the input shape. An investigation of different types of operators used in the combination of the functional evidence has been made. Three pairs of conjunctive and disjunctive operators have been used in the recognition process of more than 100 object shapes. The results are compared and differences are discussed.

Algorithms for computing the aspect graph representation are generalized to include a larger, more realistic domain of objects known as articulated assemblies those objects composed of rigid parts with articulated connections allowed between parts. The generalization suggests two slightly different representations: one that directly summarizes the possible general views of the object and another (hierarchical) form summarizing the possible general configurations and their respective views. Algorithms are outlined for computing both representations. The generalized aspect graphs of assemblies formed using translational connections are examined.>

An algorithm for computing the aspect graph for a class of curved-surface objects based on an exact parcellation of 3-D viewpoint space is presented. The object class considered is solids of revolution. A detailed analysis of the visual events for this object class is given, as well as an algorithm for constructing the aspect graph. Numerical search techniques, based on a geometric interpretation of the visual events, have been devised to determine those visual event surfaces that cannot be calculated directly. The worst-case complexity of the number of cells in the parcellation of 3-D viewpoint space, and, hence, the number of nodes in the aspect graph, is O(N/sup 4/), where N is the degree of a polynomial that defines the object shape. A summary of the results for 20 different object descriptions is presented, along with a detailed example for a flower vase.>

Currently the aspect graph is computed from the theoretical standpoint of perfect resolution in object shape, the viewpoint and the projected image. This means that the aspect graph may include details that an observer could never see in practice. Introducing the notion of scale into the aspect graph framework provides a mechanism for selecting a level of detail that is "large enough" to merit explicit representation. This effectively allows control over the number of nodes retained in the aspect graph. This paper introduces the concept of the scale space aspect graph, defines three different interpretations of the scale dimension, and presents a detailed example for a simple class of objects, with scale defined in terms of the spatial extent of features in the image.>

Currently the aspect graph is computed under the assumption of perfect resolution in the viewpoint, the projected image, and the object shape. Visual detail is represented that an observer might never see in practice. By introducing scale into this framework, a mechanism is provided for selecting levels of detail that are large enough to merit explicit representation, effectively allowing control over the size of the aspect graph. To this end the scale space aspect graph is introduced, and an interpretation of the scale dimension in terms of the spatial extent of image features is considered. A brief example is given for polygons in a plane.>

The authors report the implementation and evaluation of a function-based recognition system that takes an uninterrupted 3-D object shape as its input and reasons to determine if the object belongs to the superordinate category furniture, and if so, into which (sub)category it falls. The system has analyzed over 250 input objects, and the results largely agree with intuitive human interpretation of the objects. The study confirms that a relatively small number of knowledge primitives may be used as the basis for defining a relatively broad range of object categories. The greatest derivation from intuitive human interpretation occurs with objects that humans would not typically label as one of the known categories defined, but which have some novel orientation in which they could serve the function of one of these categories. This is because the system uses a purely function-based definition of the object category.>

The work which demonstrates the feasibility of a different approach to 3-D object recognition is described. The authors construct a definition of a generic object category, such as a chair, in terms of the function required of the object. This definition is based on qualitative reasoning about 3-D shape, and does not imply any particular geometric or structural model for an object. Thus, this approach has the potential to lead to recognition systems of much greater generality than current CAD-based or model-based approaches.>

An attempt is made to demonstrate the feasibility of defining an object category in terms of the functional properties shared by all objects in the category. This form of representation should allow much greater generality. A complete system has been implemented that takes the boundary surface description of a 3D object as its input and attempts to recognize whether the object belongs to the category 'chair' and, if so, into which subcategory if falls. This is, to the authors' knowledge, the first implemented system to explore the use of a purely function-based definition of an object category (that is, no explicit geometric or structural model) to recognize 3D objects. System competence has been evaluated on a database of over 100 objects, and the results largely agree with human interpretation of the objects.>

Consideration is given to a generalization of the aspect graph concept which is applicable to objects which may have articulated connection between their (rigid) parts. This generalization is a major step forward in terms of applying the aspect graph concept to a larger and more realistic domain of objects. An algorithm for computing the generalized aspect graph is described, and an example is analyzed in detail.>

Several researchers have previously described approaches to 3-D object recognition which use nonlinear optimization to control the matching of features of a 3-D object niodel to features found in an image. Recognition, in this context, includes estimating the parameters of translation and orientation of the object. A major problem acknowledged by previous researchers is how to efficiently choose a set of starting paranleter estimates which will avoid recognition errors due to local minima. The unique contribution of this paper is that it outlines an approach for using the perspective projection aspect gruph representation to alleviate the problems encountered by previous researchers, describes a particular implementation of this general approach, and presents data to illustrate the effectiveness (of the approac!].

An algorithm is presented for constructing the perspective projection aspect graph of polyhedra. The first phase determines the set of surfaces involved in the parcellation of viewing space around the object. Three types of surfaces are involved: object planes, in which the faces of the object lie, auxiliary planes, defined by the visual interaction of edge-vertex pairs, and auxiliary quadric surfaces, defined by the visual interaction of edge triplets. Auxiliary planes and surfaces in a sense enumerate the potential self-occlusions of the object. A list is constructed for each type of surface. Next the algorithm constructs the geometric incidence lattice representing the arrangement of object planes, and distinguishes each 3-face (node) as representing either object or viewing space. The auxiliary planes and surfaces are then added to the lattice. Each 3-face (node) in the lattice has visibility attributes which identify the potentially visible object features from viewpoints within it. These attributes are updated throughout the process of building the lattice. Finally, the lattice is traversed to make a final update of the visibility attributes and merge neighboring 3-faces which have the same visible features. The resulting structure represents the perspective projection aspect graph.

This paper has been accepted for publication in the proceedings, but the photo-ready form was not received in time. Copies of the paper should be available upon request at the presentation. It may appear in a later issue of the SIGCSE Bulletin.

article Free Access Share on Duke university computer kamp 1982 Authors: Kevin Bowyer Computer Science Department Computer Science DepartmentView Profile , Mel Ray Vice Chancellor for Data Processing Vice Chancellor for Data ProcessingView Profile , Cary Laxer Duke University, Durham, North Carolina Duke University, Durham, North CarolinaView Profile Authors Info & Claims ACM SIGCSE BulletinVolume 15Issue 1February 1983pp 233–236https://doi.org/10.1145/952978.801053Published:01 February 1983Publication History 0citation245DownloadsMetricsTotal Citations0Total Downloads245Last 12 Months60Last 6 weeks8 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Publisher SiteeReaderPDF

Contiguous element region selection is a fundamental step in many image analysis applications. This application is often run on computers with virtual memory systems. The page fault behavior of the contiguous element selection algorithm is greatly influenced by the ordering of the nearest neighbor search pattern. The execution time of the algorithm can be cut in half for important instances of real world data.