Medical datasets play a crucial role in advancing healthcare research and supporting clinical decision-making. At the same time, the reliability of responsible and accountable AI systems is directly dependent on the integrity and transparency of the datasets on which they are built. The data subcard implements the System Cards framework's data assessment dimension to evaluate tabular medical datasets across four criteria: privacy, fairness, quality, and protection. It combines data-level profiling with optional model-based diagnostics, selected to fit each dataset, to assess completeness, duplication, outliers, demographic dispar-ities, re-identification risk, and compliance readiness. Applied to the UCI Heart Disease and Diabetes Readmission datasets, the method flags privacy risks, fairness imbalances, quality defects, and protection gaps that warrant review before modeling. The data subcard produces quantitative scores and visual summaries, providing a structured and interpretable mechanism for dataset accountability within the System Cards framework.

Electronic health record (EHR) datasets come in various schemas and can contain a range of data types, measurement units, and variables that share duplicate semantic content. The process of bringing such datasets into a common schema with consistent values, so that it is possible to perform queries uniformly, is known as harmonization. However, performing this process manually can be both time-consuming and prone to errors. In this work, we present a web-based platform that semi-automates the harmonization and linking of EHR datasets through a human-in-the-loop framework, guiding users with the use of large language models (LLMs). Our solution is a two-stage harmonization pipeline that keeps schema metadata processing online while handling patient-level data locally, to align with HIPAA data privacy principles. In the first stage, users harmonize and link only non-identifiable schema information. In the second stage, sensitive value-level harmonization occurs entirely on the user's system, so no private and protected health information ever leaves their environment. Throughout both stages, we expect that LLM-powered suggestions could potentially speed up the harmonization and linking processes.

This paper presents AI-SNIPS (AI Support for Network Intelligence-based Pharmaceutical Security), a production-ready platform that enables stakeholder decision-making, secure data sharing, and interdisciplinary research in the fight against Illicit, Substandard, and Falsified Medical Products (ISFMP). AI-SNIPS takes as input cases: a case consists of one or more URLs suspected of ISFMP activity. Cases can be supplemented with ground-truth structured data (labeled keywords) such as seller PII or case notes. First, AI-SNIPS scrapes and stores relevant images and text from the provided URLs without any user intervention. Salient features for predicting case similarity are extracted from the aggregated data using a combination of rule-based and machine-learning techniques and used to construct a seller network, with the nodes representing cases (sellers) and the edges representing the similarity between two sellers. Network analysis and community detection techniques are applied to extract seller clusters ranked by profitability and their potential to harm society. Lastly, AI-SNIPS provides interpretability by distilling common word/image similarities for each cluster into signature vectors. We validate the importance of AI-SNIPS's features for distinguishing large pharmaceutical affiliate networks from small ISFMP operations using an actual ISFMP lead sheet.

Algorithms play an essential and expanding role in public policy decisions, including those in criminal justice. This short paper reports on the first author’s summer research project characterizing the tradeoff between accuracy and fairness in parole decision predictions. The dataset employed in this study contains over 30,000 parole decisions made by the New York State Division of Criminal Justice Services. Each decision contains information on the subject, such as sex, race/ethnicity, and parole decision, as well as predictive features describing the crime committed by the subject and the parole interview held. Logistic regression, decision tree, support vector machine, and random forest models are trained and utilized to analyze parole decision predictions based on the available features. Most models fail to pass standard fairness tests for most fairness metrics. Moreover, while there may be an overall tradeoff between fairness and accuracy, the obtained differences in accuracy are too small to make a well-supported claim. Future research may enhance the preliminary work introduced in this paper by using multiple real-world datasets to investigate the tradeoff between accuracy and fairness.

Criminal recidivism models are tools that have gained widespread adoption by parole boards across the United States to assist with parole decisions. These models take in large amounts of data about an individual and then predict whether an individual would commit a crime if released on parole. Although such models are not the only or primary factor in making the final parole decision, questions have been raised about their accuracy, fairness, and interpretability. In this paper, various machine learning-based criminal recidivism models are created based on a real-world parole decision dataset from the state of Georgia in the United States. The recidivism models are comparatively evaluated for their accuracy, fairness, and interpretability. It is found that there are noted differences and trade-offs between accuracy, fairness, and being inherently interpretable. Therefore, choosing the best model depends on the desired balance between accuracy, fairness, and interpretability, as no model is perfect or consistently the best across different criteria.

Deep learning-based domain-invariant feature learning methods are advancing in near-infrared and visible (NIR-VIS) heterogeneous face recognition. However, these methods are prone to overfitting due to the large intra-class variation and the lack of NIR images for training. In this paper, we introduce Domain-Based Label Face (DBLFace), a learning approach based on the assumption that a subject is not represented by a single label but by a set of labels. Each label represents images of a specific domain. In particular, a set of two labels per subject, one for the NIR images and one for the VIS images, are used for training a NIR-VIS face recognition model. The classification of images into different domains reduces the intra-class variation and lessens the negative impact of data imbalance in training. To train a network with sets of labels, we introduce a domain-based angular margin loss and a maximum angular loss to maintain the inter-class discrepancy and to enforce the close relationship of labels in a set. Quantitative experiments confirm that DBLFace significantly improves the rank-1 identification rate by 6.7% on the EDGE20 dataset and achieves state-of-the-art performance on the CASIA NIR-VIS 2.0 dataset.

Pedestrian detection remains a challenging task due to the problems caused by occlusion variance. Visible-body bounding boxes are typically used as an extra supervision signal to improve the performance of pedestrian detection to predict the full-body. However, visible-body assisted approaches produce a large number of false positives, which result from a lack of adequate and discriminative full-body contextual information. In this paper, we propose a new network, dubbed DVRNet, based on the representative visible-body assisted pedestrian detector named Bi-box. Specifically, we extend Bi-box by adding three modules named the attention-based feature interleaver module (AFIM), the binary mask learning module (BMLM), and the head-aware feature enhancement module (HFEM), which play important roles in employing features learned by the visible-body and the head supervision signals to enrich high discriminative contextual information of the full-body and enhance the power of feature representation. Experimental results indicate that the DVRNet achieves promising results on the CityPersons and the CrowdHuman datasets.

Surveillance-related datasets that have been released in recent years focus only on one specific problem at a time (e.g., pedestrian detection, face detection, or face recognition), while most of them were collected using visible spectrum (VIS) cameras. Even though some cross-spectral datasets were presented in the past, they were acquired in a constrained setup, which limited the performance of methods for the aforementioned problems under a cross-spectral setting. This work introduces a new dataset, named EDGE19, that can be used in addressing the problems of pedestrian detection, face detection, and face recognition in images captured using trail cameras under the VIS and NIR spectra. Data acquisition was performed in an outdoor environment, during both day and night, under unconstrained acquisition conditions. The collection of images is accompanied by a rich set of annotations, consisting of person and facial bounding boxes, unique subject identifiers, and labels that characterize facial images as frontal, profile, or back faces. Moreover, the performance of several state-of-the-art methods was evaluated for each of the scenarios covered by our dataset. The baseline results we obtained highlight the difficulty of current methods in the tasks of cross-spectral pedestrian detection, face detection, and face recognition due to unconstrained conditions, including low resolution, pose variation, illumination variation, occlusions, and motion blur.

For many computer vision applications such as image captioning, visual question answering, and person search, learning discriminative feature representations at both image and text level is an essential yet challenging problem. Its challenges originate from the large word variance in the text domain as well as the difficulty of accurately measuring the distance between the features of the two modalities. Most prior work focuses on the latter challenge, by introducing loss functions that help the network learn better feature representations but fail to account for the complexity of the textual input. With that in mind, we introduce TIMAM: a Text-Image Modality Adversarial Matching approach that learns modality-invariant feature representations using adversarial and cross-modal matching objectives. In addition, we demonstrate that BERT, a publicly-available language model that extracts word embeddings, can successfully be applied in the text-to-image matching domain. The proposed approach achieves state-of-the-art cross-modal matching performance on four widely-used publicly-available datasets resulting in absolute improvements ranging from 2% to 5% in terms of rank-1 accuracy.

Biometrics-related research has been accelerated significantly by deep learning technology. However, there are limited open-source resources to help researchers evaluate their deep learning-based biometrics algorithms efficiently, especially for the face recognition tasks. In this work, we design, implement, and evaluate a computationally lightweight, maintainable, scalable, generalizable, and extendable face recognition evaluation toolbox named FaRE that supports both online and offline evaluation to provide feedback to algorithm development and accelerate biometricsrelated research. FaRE includes a set of evaluation metrics and provides various APIs for commonly-used face recognition datasets including LFW, CFP, UHDB31, and IJBseries datasets. FaRE can be easily extended to include other datasets. The package is publically available for research use at https://github.com/uh-cbl/FaRE.

Many approaches for action recognition focus on general actions, such as "running" or "walking". This work presents a method for recognizing carrying actions in single images, by utilizing privileged information, such as annotation, available only during training, following the learning using privileged information paradigm. In addition, we introduce a dataset for carrying actions, formed using images extracted from YouTube videos depicting several scenarios. We accompany the dataset with a variety of different annotation types that include human pose, object and scene attributes. The experimental results demonstrate that our method, boosted sample averaged F1 score performance by 15.4% and 4.15%, respectively, in the validation and testing partitions of our dataset, when compared to an end-to-end CNN model, trained only with the observable information.

Uncontrolled illumination is one of the most significant challenges in face recognition. The performance of state-of-the-art face recognition algorithms drops drastically when measured on datasets with large illumination variations. In this paper, we propose a deep face relighting algorithm and employ it as a data augmentation method to enrich training data with illumination variations. For an input image, the proposed face relighting as data augmentation (FRADA) approach first estimates its 3D morphable model coefficients and spherical harmonic lighting coefficients. Then, it extracts the face normals, face mask, face shading, and face albedo, and renders new face images under random lighting conditions following physically-based image formation theory. Qualitative results demonstrate that FRADA produces more realistic images than the state-of-the-art face relighting algorithm. Quantitative experiments confirm the effectiveness of our relighting approach for face recognition. We successfully enhance the robustness of face templates to illumination variations simply by training face recognition algorithms with our relit images.

The goal of this work is to seek principles of designing a deep neural network for 3D face reconstruction from a single image. To make the evaluation simple, we generated a synthetic dataset and used it for evaluation. We conducted extensive experiments using an end-to-end face reconstruction algorithm using E2FAR and its variations, and analyzed the reason why it can be successfully applied for 3D face reconstruction. From the comparative studies, we conclude that feature aggregation from different layers is a key point to training better neural networks for 3D face reconstruction. Based on these observations, a face reconstruction feature aggregation network (FR-FAN) is proposed, which obtains significant improvements compared with baselines on the synthetic validation set. We evaluate our model on existing popular indoor and in-the-wild 2D-3D datasets. Extensive experiments demonstrate that FR-FAN performs 16.50% and 9.54% better than E2FAR on BU-3DFE and JNU-3D, respectively. Finally, the sensitivity analysis we performed on controlled datasets demonstrates that our designed network is robust to large variations of pose, illumination, and expressions.

Recent advances in deep learning have spurred 3D hand pose estimation, as convolutional network (ConvNet) based methods outperformed random forests. However, in the state of the art, ConvNet based methods employ only depth images of the hand without leveraging color and texture information from the RGB domain. In this paper, we investigate whether ConvNets can learn more rich and discriminative em-beddings, by combining RGB and depth information. To answer this question, we propose the fusion of RGB and depth information in a double-stream architecture. More specifically, RGB and depth images are fed into two separate networks by extracting features, which are subsequently fused at an intermediate layer of the ConvNet, implementing input-level fusion, feature-level fusion and score-level fusion. The double-stream scheme is coupled with a deep ConvNet, contrary to the shallow networks that are mostly proposed in the literature. Experimental results show that while the depth of the network is crucial for hand pose estimation, the double-stream nets perform very similarly with the net trained only with depth images. This may suggest that training double-stream architectures purely with supervision may be insufficient for hand pose estimation with RGB-D fusion.

The goal of point-to-set matching is to match a single image with a set of images from a subject. Within an image set, different images contain various levels of discriminative information and thus should contribute differently to the results. However, the discriminative level is not accessible directly. To this end, we propose a confidence driven network to perform point-to-set matching. The proposed system comprises a feature extraction network (FEN) and a performance prediction network (PPN). Given an input image, the FEN generates a template, while the PPN generates a confidence score which measures the discriminative level of the template. At matching time, the template is used to compute a point-to-point similarity. The similarity scores from different samples in the set are integrated at a score level, weighted by the predicted confidence scores. Extensive multi-probe face recognition experiments on the IJB-A and UHDB-31 datasets demonstrate performance improvements over state of the art algorithms.

Monocular 3D facial shape reconstruction from a single 2D facial image has been an active research area due to its wide applications. Inspired by the success of deep neural networks (DNN), we propose a DNN-based approach for End-to-End 3D FAce Reconstruction (UH-E2FAR) from a single 2D image. Different from recent works that reconstruct and refine the 3D face in an iterative manner using both an RGB image and an initial 3D facial shape rendering, our DNN model is end-to-end, and thus the complicated 3D rendering process can be avoided. Moreover, we integrate in the DNN architecture two components, namely a multi-task loss function and a fusion convolutional neural network (CNN) to improve facial expression reconstruction. With the multi-task loss function, 3D face reconstruction is divided into neutral 3D facial shape reconstruction and expressive 3D facial shape reconstruction. The neutral 3D facial shape is class-specific. Therefore, higher layer features are useful. In comparison, the expressive 3D facial shape favors lower or intermediate layer features. With the fusion-CNN, features from different intermediate layers are fused and transformed for predicting the 3D expressive facial shape. Through extensive experiments, we demonstrate the superiority of our end-to-end framework in improving the accuracy of 3D face reconstruction.

In this paper, we explore global and local features obtained from Convolutional Neural Networks (CNN) for learning to estimate head pose and localize landmarks jointly. Because there is a high correlation between head pose and landmark locations, the head pose distributions from a reference database and learned local deep patch features are used to reduce the error in the head pose estimation and face alignment tasks. First, we train GNet on the detected face region to obtain a rough estimate of the pose and to localize the seven primary landmarks. The most similar shape is selected for initialization from a reference shape pool constructed from the training samples according to the estimated head pose. Starting from the initial pose and shape, LNet is used to learn local CNN features and predict the shape and pose residuals. We demonstrate that our algorithm, named JFA, improves both the head pose estimation and face alignment. To the best of our knowledge, this is the first system that explores the use of the global and local CNN features to solve head pose estimation and landmark detection tasks jointly.

Image-based 3D face reconstruction has great potential in different areas, such as facial recognition, facial analysis, and facial animation. Due to the variations in image quality, single-image-based 3D face reconstruction might not be sufficient to accurately reconstruct a 3D face. To overcome this limitation, multi-view 3D face reconstruction uses multiple images of the same subject and aggregates complementary information for better accuracy. Though theoretically appealing, there are multiple challenges in practice. Among these challenges, the most significant is that it is difficult to establish coherent and accurate correspondence among a set of images, especially when these images are captured in different conditions. In this paper, we propose a method, Deep Recurrent 3D FAce Reconstruction (DRFAR), to solve the task ofmulti-view 3D face reconstruction using a subspace representation of the 3D facial shape and a deep recurrent neural network that consists of both a deep con-volutional neural network (DCNN) and a recurrent neural network (RNN). The DCNN disentangles the facial identity and the facial expression components for each single image independently, while the RNN fuses identity-related features from the DCNN and aggregates the identity specific contextual information, or the identity signal, from the whole set of images to predict the facial identity parameter, which is robust to variations in image quality and is consistent over the whole set of images. Through extensive experiments, we evaluate our proposed method and demonstrate its superiority over existing methods.

Registering a 3D facial model to a 2D image under occlusion is difficult. First, not all of the detected facial landmarks are accurate under occlusions. Second, the number of reliable landmarks may not be enough to constrain the problem. We propose a method to synthesize additional points (Sensible Points) to create pose hypotheses. The visual clues extracted from the fiducial points, non-fiducial points, and facial contour are jointly employed to verify the hypotheses. We define a reward function to measure whether the projected dense 3D model is well-aligned with the confidence maps generated by two fully convolutional networks, and use the function to train recurrent policy networks to move the Sensible Points. The same reward function is employed in testing to select the best hypothesis from a candidate pool of hypotheses. Experimentation demonstrates that the proposed approach is very promising in solving the facial model registration problem under occlusion.

A few well-developed face recognition pipelines have been reported in recent years. Most of the face-related work focuses on a specific module or demonstrates a research idea. In this paper, we present a pose-invariant 3D-aided 2D face recognition system (3D2D-PIFR) that is robust to pose variations as large as 90° by leveraging deep learning technology. We describe the architecture and the interface of 3D2D-PIFR, and introduce each module in detail. Experiments are conducted on the UHDB31 and IJB-A, demonstrating that 3D2D-PIFR outperforms existing 2D face recognition systems such as VGG-Face, FaceNet, and a commercial off-the-shelf software (COTS) by at least 9% on UHDB31 and 3% on IJB-A dataset on average. It fills a gap by providing a 3D-aided 2D face recognition system that has compatible results with 2D face recognition systems using deep learning techniques.

This paper focuses on improving the performance of current convolutional neural networks in face recognition without changing the network architecture. We propose a hierarchical framework that builds chains of local binary neural networks after one global neural network over all the class labels, Local Classifier Chains based Convolutional Neural Networks (LCC-CNN). Two different criteria based on a similarity matrix and confusion matrix are introduced to select binary label pairs to create local deep networks. To avoid error propagation, each testing sample travels through one global model and a local classifier chain to obtain its final prediction. The proposed framework has been evaluated with UHDB31 and CASIA-WebFace datasets. The experimental results indicate that our framework achieves better performance when compared with using only baseline methods as the global deep network. The accuracy is improved by 2.7% and 0.7% on the two datasets, respectively.

In this paper, a human behavior recognition method using multimodal features is presented. We focus on modeling individual and social behaviors of a subject (e.g., friendly/aggressive or hugging/kissing behaviors) with a hidden conditional random field (HCRF) in a supervised framework. Each video is represented by a vector of spatio-temporal visual features (STIP, head orientation and proxemic features) along with audio features (MFCCs). We propose a feature pruning method for removing irrelevant and redundant features based on the spatio-temporal neighborhood of each feature in a video sequence. The proposed framework assumes that human movements are highly correlated with sound emissions. For this reason, canonical correlation analysis (CCA) is employed to find correlation between the audio and video features prior to fusion. The experimental results, performed in two human behavior recognition datasets including political speeches and human interactions from TV shows, attest the advantages of the proposed method compared with several baseline and alternative human behavior recognition methods.

In this paper, we first offer an overview of advances in the field of distance metric learning. Then, we empirically compare selected methods using a common experimental protocol. The number of distance metric learning algorithms proposed keeps growing due to their effectiveness and wide application. However, existing surveys are either outdated or they focus only on a few methods. As a result, there is an increasing need to summarize the obtained knowledge in a concise, yet informative manner. Moreover, existing surveys do not conduct comprehensive experimental comparisons. On the other hand, individual distance metric learning papers compare the performance of the proposed approach with only a few related methods and under different settings. This highlights the need for an experimental evaluation using a common and challenging protocol. To this end, we conduct face verification experiments, as this task poses significant challenges due to varying conditions during data acquisition. In addition, face verification is a natural application for distance metric learning because the encountered challenge is to define a distance function that: 1) accurately expresses the notion of similarity for verification; 2) is robust to noisy data; 3) generalizes well to unseen subjects; and 4) scales well with the dimensionality and number of training samples. In particular, we utilize well-tested features to assess the performance of selected methods following the experimental protocol of the state-of-the-art database labeled faces in the wild. A summary of the results is presented along with a discussion of the insights obtained and lessons learned by employing the corresponding algorithms.

In this work, we investigate the problem of predicting gender from still images using human metrology. Since the values of the anthropometric measurements are difficult to be estimated accurately from state-of-the-art computer vision algorithms, ratios of anthropometric measurements were used as features. Additionally, since several measurements will not be available at test time in a real-life scenario, we opted for the Learning Using Privileged Information (LUPI) paradigm. During training, we used as features, ratios from all the available anthropometric measurements, whereas at test time only ratios of measurable (i.e., observable) quantities were used. We show that by using the LUPI framework, the estimation of soft biometric characteristics such as gender is possible. Gender classification from human metrology is also tested on real images with promising results.

In many human activity recognition systems the size of the unlabeled training data may be significantly large due to expensive human effort required for data annotation. Moreover, the insufficient data collection process from heterogenous sources may cause dissimilarities between training and testing data. To address these limitations, a novel probabilistic approach that combines learning using privileged information (LUPI) and active learning is proposed. A pool-based privileged active learning approach is presented for semi-supervising learning of human activities from multimodal labeled and unlabeled data. Both uncertainty and distance from the decision boundary are used as query inference strategies for selecting an unlabeled observation and querying its label. Experimental results in four publicly available datasets demonstrate that the proposed method can identify complex human activities with high accuracy.

Existing distance metric learning methods define an objective function and seek a distance metric (or equivalently a projection) that minimizes it. In this paper, we propose a different approach that illustrates how to formulate distance metric learning as a regression problem. First, the objective function is minimized to learn target representations. Then, a regression method is employed to learn a projection that maps the input to the target representations. This global projection function is the single output of the proposed algorithm. Our contribution is a different perspective on how to train a distance metric learning algorithm. The advantages are: (i) this approach has the potential to simplify the optimization process; and (ii) it allows researchers to leverage the power of existing regression methods and those to be invented. Experimental results on several publicly available datasets illustrate that the proposed framework can learn a distance metric with discriminative properties.

In this paper, we propose a novel regression-based method for employing privileged information to estimate the height using human metrology. The actual values of the anthropometric measurements are difficult to estimate accurately using state-of-the-art computer vision algorithms. Hence, we use ratios of anthropometric measurements as features. Since many anthropometric measurements are not available at test time in real-life scenarios, we employ a learning using privileged information (LUPI) framework in a regression setup. Instead of using the LUPI paradigm for regression in its original form (i.e., ε-SVR+), we train regression models that predict the privileged information at test time. The predictions are then used, along with observable features, to perform height estimation. Once the height is estimated, a mapping to classes is performed. We demonstrate that the proposed approach can estimate the height better and faster than the ε-SVR+ algorithm and report results for different genders and quartiles of humans.

People with low vision, Alzheimer's disease, and autism spectrum disorder experience difficulties in perceiving or interpreting facial expression of emotion in their social lives. Though automatic facial expression recognition (FER) methods on 2-D videos have been extensively investigated, their performance was constrained by challenges in head pose and lighting conditions. The shape information in 3-D facial data can reduce or even overcome these challenges. However, high expenses of 3-D cameras prevent their widespread use. Fortunately, 2.5-D facial data from emerging portable RGB-D cameras provide a good balance for this dilemma. In this paper, we propose an automatic emotion annotation solution on 2.5-D facial data collected from RGB-D cameras. The solution consists of a facial landmarking method and a FER method. Specifically, we propose building a deformable partial face model and fit the model to a 2.5-D face for localizing facial landmarks automatically. In FER, a novel action unit (AU) space-based FER method has been proposed. Facial features are extracted using landmarks and further represented as coordinates in the AU space, which are classified into facial expressions. Evaluated on three publicly accessible facial databases, namely EURECOM, FRGC, and Bosphorus databases, the proposed facial landmarking and expression recognition methods have achieved satisfactory results. Possible real-world applications using our algorithms have also been discussed.

Uncontrolled illumination is one of the most widely researched and most encountered face recognition challenges in recent years. In this study, the authors propose the division of algorithms into two categories: (i) relighting and (ii) unlighting. Relighting methods try to match the probe's illumination conditions using a subset of representative gallery images, while unlighting methods seek to suppress the variations. A total of 64 state‐of‐the‐art methods are summarised and categorised in each of the groups. To make this work concise and easy to follow, they restricted themselves to selected conferences/journals and they limited the number of approaches reviewed. Also, eight past state‐of‐the‐art approaches are used in both identification and verification experiments. However, only significant reported results from all methods were compared and organised in tables. The author's main objective is not to provide an exhaustive analysis of each category, but to present a collection of papers that can be useful in identifying research directions. Results indicate that unlighting methods are a better and a practical solution to address illumination challenges.

Biometric systems use score normalization techniques and fusion rules to improve recognition performance. The large amount of research on score fusion for multimodal systems raises an important question: can we utilize the available information from unimodal systems more effectively? In this paper, we present a rank-based score normalization framework that addresses this problem. Specifically, our approach consists of three algorithms: 1) partition the matching scores into subsets and normalize each subset independently; 2) utilize the gallery versus gallery matching scores matrix (i.e., gallery-based information); and 3) dynamically augment the gallery in an online fashion. We invoke the theory of stochastic dominance along with results of prior research to demonstrate when and why our approach yields increased performance. Our framework: 1) can be used in conjunction with any score normalization technique and any fusion rule; 2) is amenable to parallel programming; and 3) is suitable for both verification and open-set identification. To assess the performance of our framework, we use the UHDB11 and FRGC v2 face datasets. Specifically, the statistical hypothesis tests performed illustrate that the performance of our framework improves as we increase the number of samples per subject. Furthermore, the corresponding statistical analysis demonstrates that increased separation between match and nonmatch scores is obtained for each probe. Besides the benefits and limitations highlighted by our experimental evaluation, results under optimal and pessimal conditions are also presented to offer better insights.

In this paper, we propose a method for matching biometric data from disparate domains. Specifically, we focus on the problem of comparing a low-resolution (LR) image with a high-resolution (HR) one. Existing coupled mapping methods do not fully exploit the HR information or they do not simultaneously use samples from both domains during training. To this end, we propose a method that learns coupled distance metrics in two steps. In addition, we propose to jointly learn two semi-coupled bases that yield optimal representations. In particular, the HR images are used to learn a basis and a distance metric that result in increased class-separation. The LR images are then used to learn a basis and a distance metric that map the LR data to their class-discriminated HR pairs. Finally, the two distance metrics are refined to simultaneously enhance the class-separation of both HR class-discriminated and LR projected images. We illustrate that different distance metric learning approaches can be employed in conjunction with our framework. Experimental results on Multi-PIE and SCface, along with the relevant hypothesis tests, provide evidence of the effectiveness of the proposed approach.

Diabetes is a life-altering medical condition that affects millions of people and results in many hospitalizations per year. Consequently, predicting the length of stay of in-hospital diabetic patients has become increasingly important for staffing and resource planning. Although statistical methods have been used to predict length of stay in hospitalized patients, many powerful machine learning techniques have not yet been explored. In this paper, we compare and discuss the performance of various supervised machine learning algorithms (i.e., Multiple linear regression, support vector machines, multi-task learning, and random forests) for predicting long versus short-term length of stay of hospitalized diabetic patients.

We present a method for the filtering of over-sampled data using non-cooperative game theory (NEATER) to address the imbalanced data problem using game theory. Specifically, the problem is formulated as a non-cooperative game where all the data are players and the goal is to uniformly and consistently label all of the synthetic data created by any over-sampling technique. We present extensive experimental results which demonstrate the advantages of our method.

3D-Model-Aided 2D face recognition (MaFR) has attracted a lot of attention in recent years. By registering a 3D model, facial textures of the gallery and the probe can be lifted and aligned in a common space, thus alleviating the challenge of pose variations. One obstacle preventing accurate registration is the 3D-2D pose estimation, which is easily affected by landmarks. In this work, we present the performance that state-of-the-art pose estimation algorithms could reach using state-of-the-art automatic landmark localization methods. We generated an application-specific dataset with more than 59,000 synthetic face images and ground truth camera pose and landmarks, covering 45 poses and six illumination conditions. Our experiments compared four recently proposed pose estimation algorithms using 2D landmarks detected by two automatic methods. Our results highlight one near-real-time landmark detection method and a highly accurate pose estimation algorithm, which would potentially boost the 3D-Model-Aided 2D face recognition performance.

Asymmetric 3D to 2D face recognition has gained attention from the research community since the real-world application of 3D to 3D recognition is limited by the unavailability of inexpensive 3D data acquisition equipment. A 3D to 2D face recognition system explicitly relies on 3D facial data to account for uncontrolled image conditions related to head pose or illumination. We build upon such a system, which matches relit gallery textures with pose-normalized probe images, using the gallery facial meshes. The relighting process, however, is based on an assumption of indoor lighting conditions and limits recognition performance on outdoor images. In this paper, we propose a novel method for minimizing illumination difference by unlighting a 3D face texture via albedo estimation using lighting maps. The algorithm is evaluated on challenging databases (UHDB30, UHDB11, FRGC v2.0) with drastic lighting and pose variations. The experimental results demonstrate the robustness of our method for estimating the albedo from both indoor and outdoor captured images, and the effectiveness and efficiency for illumination normalization in face recognition.

Behavioral biometrics have recently begun to gain attention for mobile user authentication. The feasibility of touch gestures as a novel modality for behavioral biometrics has been investigated. In this paper, we propose applying a statistical touch dynamics image (aka statistical feature model) trained from graphic touch gesture features to retain discriminative power for user authentication while significantly reducing computational time during online authentication. Systematic evaluation and comparisons with state-of-the-art methods have been performed on touch gesture data sets. Implemented as an Android App, the usability and effectiveness of the proposed method have also been evaluated.

Image segmentation is, in general, an ill-posed problem and additional constraints need to be imposed in order to achieve the desired segmentation result. While segmenting organs in medical images, which is the topic of this paper, a significant amount of prior knowledge about the shape, appearance, and location of the organs is available that can be used to constrain the solution space of the segmentation problem. Among the various types of prior information, the incorporation of prior information about shape, in particular, is very challenging. In this paper, we present an explicit shape-constrained MAP-MRF-based contour evolution method for the segmentation of organs in 2-D medical images. Specifically, we represent the segmentation contour explicitly as a chain of control points. We then cast the segmentation problem as a contour evolution problem, wherein the evolution of the contour is performed by iteratively solving a MAP-MRF labeling problem. The evolution of the contour is governed by three types of prior information, namely: (i) appearance prior, (ii) boundary-edgeness prior, and (iii) shape prior, each of which is incorporated as clique potentials into the MAP-MRF problem. We use the master-slave dual decomposition framework to solve the MAP-MRF labeling problem in each iteration. In our experiments, we demonstrate the application of the proposed method to the challenging problem of heart segmentation in non-contrast computed tomography data.

Statistical shape models, such as Active Shape Models (ASMs), suffer from their inability to represent a large range of variations of a complex shape and to account for the large errors in detection of model points. We propose a novel method (dubbed PDM-ENLOR) that overcomes these limitations by locating each shape model point individually using an ensemble of local regression models and appearance cues from selected model points. Our method first detects a set of reference points which were selected based on their saliency during training. For each model point, an ensemble of regressors is built. From the locations of the detected reference points, each regressor infers a candidate location for that model point using local geometric constraints, encoded by a point distribution model (PDM). The final location of that point is determined as a weighted linear combination, whose coefficients are learnt from the training data, of candidates proposed from its ensemble's component regressors. We use different subsets of reference points as explanatory variables for the component regressors to provide varying degrees of locality for the models in each ensemble. This helps our ensemble model to capture a larger range of shape variations as compared to a single PDM. We demonstrate the advantages of our method on the challenging problem of segmenting gene expression images of mouse brain.

We present a Markov Random Field model for the analysis of lattices (e.g., images or 3D meshes) in terms of the discriminative information of their vertices. The proposed method provides a measure field that estimates the probability of each vertex being "discriminative" or "nondiscriminative" for a given classification task. To illustrate the applicability and generality of our framework, we use the estimated probabilities as feature scoring to define compact signatures for three different classification tasks: 1) 3D Face Recognition, 2) 3D Facial Expression Recognition, and 3) Ethnicity-based Subject Retrieval, obtaining very competitive results. The main contribution of this work lies in the development of a novel framework for feature selection in scenaria in which the most discriminative information is smoothly distributed along a lattice.

A 3D landmark detection method for 3D facial scans is presented and thoroughly evaluated. The main contribution of the presented method is the automatic and pose-invariant detection of landmarks on 3D facial scans under large yaw variations (that often result in missing facial data), and its robustness against large facial expressions. Three-dimensional information is exploited by using 3D local shape descriptors to extract candidate landmark points. The shape descriptors include the shape index, a continuous map of principal curvature values of a 3D object's surface, and spin images, local descriptors of the object's 3D point distribution. The candidate landmarks are identified and labeled by matching them with a Facial Landmark Model (FLM) of facial anatomical landmarks. The presented method is extensively evaluated against a variety of 3D facial databases and achieves state-of-the-art accuracy (4.5-6.3 mm mean landmark localization error), considerably outperforming previous methods, even when tested with the most challenging data.

Studies have shown that aortic calcification is associated with cardiovascular disease. In this study, a method for localization, centerline extraction, and segmentation of the thoracic aorta in noncontrast cardiac-computed tomography (CT) images, toward the detection of aortic calcification, is presented. The localization of the right coronary artery ostium slice is formulated as a regression problem whose input variables are obtained from simple intensity features computed from a pyramid representation of the slice. The localization, centerline extraction, and segmentation of the aorta are formulated as optimal path detection problems. Dynamic programming is applied in the Hough space for localizing key center points in the aorta which guide the centerline tracing using a fast marching-based minimal path extraction framework. The input volume is then resampled into a stack of 2-D cross-sectional planes orthogonal to the obtained centerline. Dynamic programming is again applied for the segmentation of the aorta in each slice of the resampled volume. The obtained segmentation is finally mapped back to its original volume space. The performance of the proposed method was assessed on cardiac noncontrast CT scans and promising results were obtained.

Face recognition is a challenging task, especially when low-resolution images or image sequences are used. A decrease in image resolution results in a loss of facial high frequency components leading to a decrease in recognition rates. In this paper, we propose a new method for super-resolution by building a dictionary of high-frequency components in the facial data, which are added to a low-resolution input image to create a super-resolved image. Our method is different from existing methods as we estimate the high-frequency components, rather than studying the direct relationship between the high- and low-resolution images. Quantitative and qualitative results are reported for both synthetic and surveillance facial image databases.

This paper presents a simple and computationally efficient framework for human action recognition based on modeling the motion of human body parts. Intuitively, a collective understanding of human part movements can lead to better understanding and representation of any human action. In this paper, we propose a generative representation of the motion of the human body parts to learn and classify human actions. The proposed representation combines the advantages of both local and global representations, encoding the relevant motion information as well as being robust to local appearance changes. Our work is motivated by the pictorial structures model and the framework of sparse representations for recognition. Human part movements are represented efficiently through quantization in the polar space. The key discrimination within each action is efficiently encoded by sparse representation to perform classification. The proposed method is evaluated on both the KTH and the UCF action datasets and the results are compared against other state-of-the-art methods.

Human interaction dynamics are known to play an important role in the development of robust pedestrian trackers that are applicable to a variety of applications in video surveillance. Traditional approaches to pedestrian tracking assume that each pedestrian walks independently and the tracker predicts the location based on an underlying motion model, such as a constant velocity or autoregressive model. Recent approaches have begun to leverage interaction, especially by modeling the repulsion force, among pedestrians to improve motion predictions. However, human interaction is more complex and is influenced by both repulsion and attraction effects. This motivates the use of a more complex human interaction model for pedestrian tracking. In this paper, we propose a novel visual tracking method by leveraging complex social interactions. We present an algorithm that decomposes social interactions into multiple potential interaction modes. We integrate these multiple social interaction modes into an interactive Markov Chain Monte Carlo tracker. We demonstrate how the developed method translates into a more informed motion prediction, resulting in a robust tracking performance. We test our method on videos from unconstrained outdoor environments and compare it against popular multi-object trackers.

Analysis of gene expression patterns in brain images obtained from high-throughput in situ hybridization requires accurate and consistent annotations of anatomical regions/subregions. Such annotations are obtained by mapping an anatomical atlas onto the gene expression images through intensity- and/or landmark-based registration methods or deformable model-based segmentation methods. Due to the complex appearance of the gene expression images, these approaches require a pre-processing step to determine landmark correspondences in order to incorporate landmark-based geometric constraints. In this paper, we propose a novel method for landmark-constrained, intensity-based registration without determining landmark correspondences a priori. The proposed method performs dense image registration and identifies the landmark correspondences, simultaneously, using a single higher-order Markov Random Field model. In addition, a machine learning technique is used to improve the discriminating properties of local descriptors for landmark matching by projecting them in a Hamming space of lower dimension. We qualitatively show that our method achieves promising results and also compares well, quantitatively, with the expert's annotations, outperforming previous methods.

We present a Markov Random Field model for the analysis of lattices (e.g., images or 3D meshes) in terms of the discriminative information of their vertices. The proposed method provides a measure field that estimates the probability of each vertex to be “discriminative” or “non-discriminative”. As an application of the proposed framework, we present a method for the selection of compact and robust features for 3D face recognition. The resulting signature consists of 360 coefficients, based on which we are able to build a classifier yielding better recognition rates than currently reported in the literature. The main contribution of this work lies in the development of a novel framework for feature selection in scenarios in which the most discriminative information is known to be concentrated along piece-wise smooth regions of a lattice.

Face recognition is a challenging problem, especially when low resolution images or image sequences are used for the task. Many methods have been proposed that can combine multiple low resolution images to realize a higher resolution or super-resolved image. Nonetheless, their utility and limitations for use in face recognition are not well understood. In this paper, we present a quantitative and comparative evaluation of wavelet transform based methods for image super-resolution. We evaluate different basis functions, varying levels of decomposition, and multiple methods for coefficient fusion to maximize the benefit of the super-resolved image for the task of face recognition. We have used a Discrete Wavelet Transform and the shift-invariant Dual-Tree Complex Wavelet Transform. Results are reported across both manually generated datasets and data from a surveillance system.

Landmark detection has proven to be a very challenging task in biometrics. In this paper, we address the task of facial component-landmark detection. By “component” we refer to a rectangular subregion of the face, containing an anatomical component (e.g., “eye”). We present a fully-automated system for facial component-landmark detection based on multi-resolution isotropic analysis and adaptive bag-of-words descriptors incorporated into a cascade of boosted classifiers. Specifically, first each component-landmark detector is applied independently and then the information obtained is used to make inferences for the localization of multiple components. The advantage of our approach is that it has robustness to pose as well as illumination. Our method has a failure rate lower than that of commercial software. Additionally, we demonstrate that using our method for the initialization of a point landmark detector results in performance comparable with that of state-of-the-art methods. All of our experiments are carried out using data from a publicly available database.

This survey focuses on discrete expression classification and facial action unit recognition performed using 3D face data, possibly including a corresponding 2D texture image. Research trends to date are summarized and the limitations of current methods are discussed. The challenges towards the development of more accurate and automated 3D facial expression recognition methods are identified. We also call for standardized experimental protocols in order to draw fair and meaningful comparisons between different systems.

Face recognition is a challenging task, especially when low-resolution images or image sequences are used. A de crease in image resolution typically results in loss of facial component details leading to a decrease in recognition rates. In this paper, we propose a new method for super resolution by first learning the high-frequency components in the facial data that can be added to a low-resolution in put image to create a super-resolved image. Our method is different from conventional methods as we estimate the high-frequency components, that are not used in other methods, to reconstruct a higher-resolution image, rather than studying the direct relationship between the high and low resolution images. Quantitative and qualitative results are reported for both synthetic and surveillance facial image databases.

Facial landmark detection is a fundamental step for many tasks in computer vision such as expression recognition and face alignment. In this paper, we focus on the detection of landmarks under realistic scenarios that include pose, illumination and expression challenges as well as blur and low-resolution input. In our approach, an n-point shape of point-landmarks is represented as a union of simpler polygonal sub-shapes. The core idea of our method is to find the sequence of deformation parameters simultaneously for all sub-shapes that transform each point-landmark into its target landmark location. To accomplish this task, we introduce an agglomerate of fern regressors. To optimize the convergence speed and accuracy we take advantage of search localization using component-landmark detectors, multi-scale analysis and learning of point cloud dynamics. Results from extensive experiments on facial images from several challenging publicly available databases demonstrate that our method (ACFeR) can reliably detect landmarks with accuracy comparable to commercial soft ware and other state-of-the-art methods.

We present a novel approach for computing a compact and highly discriminant biometric signature for 3D face recognition using linear dimensionality reduction techniques. Initially, a geometry-image representation is used to effectively resample the raw 3D data. Subsequently, a wavelet transform is applied and a biometric signature composed of 7,200 wavelet coefficients is extracted. Finally, we apply a second linear dimensionality reduction step to the wavelet coefficients using Linear Discriminant Analysis and compute a compact biometric signature. Although this biometric signature consists of just 57 coefficients, it is highly discriminant. Our approach, UR3D-C, is experimentally validated using four publicly available databases (FRGC vl, FRGC v2, Bosphorus and BU-3DFE). State-of-the-art performance is reported in all of the above databases.

Existing 3D face recognition algorithms have achieved high enough performances against public datasets like FRGC v2, that it is difficult to achieve further significant increases in recognition performance. However, the 3D TEC dataset is a more challenging dataset which consists of 3D scans of 107 pairs of twins that were acquired in a single session, with each subject having a scan of a neutral expression and a smiling expression. The combination of factors related to the facial similarity of identical twins and the variation in facial expression makes this a challenging dataset. We conduct experiments using state of the art face recognition algorithms and present the results. Our results indicate that 3D face recognition of identical twins in the presence of varying facial expressions is far from a solved problem, but that good performance is possible.

An accurate labeling of a multi-part, complex anatomical structure (e.g., brain) is required in order to compare data across images for spatial analysis. It can be achieved by fitting an object-specific geometric atlas that is constructed using a partitioned, high-resolution deformable mesh and tagging each of its polygons with a region label. Subdivision meshes have been used to construct such an atlas because they can provide a compact representation of a partitioned, multi-resolution, object-specific mesh structure using only a few control points. However, automated fitting of a subdivision mesh-based geometric atlas to an anatomical structure in an image is a difficult problem and has not been sufficiently addressed. In this paper, we propose a novel Markov Random Field-based method for fitting a planar, multi-part subdivision mesh to anatomical data. The optimal fitting of the atlas is obtained by determining the optimal locations of the control points. We also tackle the problem of landmark matching in tandem with atlas fitting by constructing a single graphical model to impose pose-invariant, landmark-based geometric constraints on atlas deformation. The atlas deformation is also governed by additional constraints imposed by the mesh's geometric properties and the object boundary. We demonstrate the potential of the proposed method on the difficult problem of segmenting a mouse brain and its interior regions in gene expression images which exhibit large intensity and shape variability. We obtain promising results when compared with manual annotations and prior methods.

In this paper, we propose a novel one-shot optimization approach to simultaneously determine both the optimal 3D landmark model and the corresponding 2D projections without explicit estimation of the camera viewpoint, which is also able to deal with misdetections as well as partial occlusions. To this end, a 3D shape manifold is built upon fourth-order interactions of landmarks from a training set where pose-invariant statistics are obtained in this space. The 3D-2D consistency is also encoded in such high-order interactions, which eliminate the necessity of viewpoint estimation. Furthermore, the modeling of visibility improves further the performance of the method by handling missing correspondences and occlusions. The inference is addressed through a MAP formulation which is naturally transformed into a higher-order MRF optimization problem and is solved using a dual-decomposition-based method. Promising results on standard face benchmarks demonstrate the potential of our approach.

Facial landmark detection has proved to be a very challenging task in biometrics due to the numerous sources of variation. In this work, we present an algorithm for robust detection of facial component-landmarks. Specifically, we address the variation due to extreme pose and illumination. To achieve robust detection for extreme poses, we use a set of independent pose and landmark specific detectors. Each component-landmark detector is applied independently and the information obtained is used to make inferences about the layout of multiple components. In addition, we incorporate a multi-view representation based on an aspect graph approach. The performance of our algorithm is assessed using data from a publicly available database. The failure rate of our method is lower than that of commercially available software.

The uncontrolled conditions of real-world biometric applications pose a great challenge to any face recognition approach. The unconstrained acquisition of data from uncooperative subjects may result in facial scans with significant pose variations along the yaw axis. Such pose variations can cause extensive occlusions, resulting in missing data. In this paper, a novel 3D face recognition method is proposed that uses facial symmetry to handle pose variations. It employs an automatic landmark detector that estimates pose and detects occluded areas for each facial scan. Subsequently, an Annotated Face Model is registered and fitted to the scan. During fitting, facial symmetry is used to overcome the challenges of missing data. The result is a pose invariant geometry image. Unlike existing methods that require frontal scans, the proposed method performs comparisons among interpose scans using a wavelet-based biometric signature. It is suitable for real-world applications as it only requires half of the face to be visible to the sensor. The proposed method was evaluated using databases from the University of Notre Dame and the University of Houston that, to the best of our knowledge, include the most challenging pose variations publicly available. The average rank-one recognition rate of the proposed method in these databases was 83.7 percent.

Three-dimensional face landmarking aims at automatically localizing facial landmarks and has a wide range of applications (e.g., face recognition, face tracking, and facial expression analysis). Existing methods assume neutral facial expressions and unoccluded faces. In this paper, we propose a general learning-based framework for reliable landmark localization on 3-D facial data under challenging conditions (i.e., facial expressions and occlusions). Our approach relies on a statistical model, called 3-D statistical facial feature model, which learns both the global variations in configurational relationships between landmarks and the local variations of texture and geometry around each landmark. Based on this model, we further propose an occlusion classifier and a fitting algorithm. Results from experiments on three publicly available 3-D face databases (FRGC, BU-3-DFE, and Bosphorus) demonstrate the effectiveness of our approach, in terms of landmarking accuracy and robustness, in the presence of expressions and occlusions.

In this paper, we present a graph-based image segmentation method (patch-cuts) that incorporates features and spatial relations obtained from image patches. In the first step, patch-cuts extracts a set of patches that can assume arbitrary shape and size. Patches are determined by a combination of intensity quantization and morphological operations and render the proposed method robust against noise. Upon patch extraction, a set of intensity, texture and shape features are computed for each patch. These features are integrated and minimized simultaneously in a tunable energy function. Patch-cuts explores the benefit of information theory-based measures such as the Kullback-Leibler and the Jensen-Shannon divergence in its energy terms. In our experiments, we applied patchcuts to general images as well as to non-contrast Computed Tomography heart scans. 1

In this paper, we present a knowledge-driven Markov Random Field (MRF) model for the segmentation of organs in medical images with particular emphasis on the incorporation of shape constraints into the segmentation problem. We cast the problem of image segmentation as the Maximum A Posteriori (MAP) estimation of a Markov Random Field which, in essence, is equivalent to the minimization of the corresponding Gibbs energy function. We then incorporate a set of constraints into the Gibbs energy function that collectively force the resulting segmentation contour/surface to have a shape similar to that of a given shape template. In particular, we introduce a flux-maximization constraint and a generalized template-based star-shape constraint that are encoded into the first- and second-order clique potentials of the Gibbs energy function, respectively. Our main contribution is in the translation of a set of global notions about the shape of the desired segmentation contour into a set of local measures that can be conveniently encoded into the Gibbs energy function and used in combination with other traditionally used constraints derived from image information. In our experiments, we demonstrate the application of the proposed method to the challenging problem of heart segmentation in non-contrast computed tomography (CT) data.

In this paper, we present a new method for bidirectional relighting for 3D-aided 2D face recognition under large pose and illumination changes. During subject enrollment, we build subject-specific 3D annotated models by using the subjects' raw 3D data and 2D texture. During authentication, the probe 2D images are projected onto a normalized image space using the subject-specific 3D model in the gallery. Then, a bidirectional relighting algorithm and two similarity metrics (a view-dependent complex wavelet structural similarity and a global similarity) are employed to compare the gallery and probe. We tested our algorithms on the UHDB11 and UHDB12 databases that contain 3D data with probe images under large lighting and pose variations. The experimental results show the robustness of our approach in recognizing faces in difficult situations.

Image segmentation is, in general, an ill-posed problem and additional constraints need to be imposed in order to achieve the desired result. Particularly in the field of medical image segmentation, a significant amount of prior knowledge is available that can be used to constrain the solution space of the segmentation problem. However, most of this prior knowledge is, in general, vague or imprecise in nature, which makes it very difficult to model. This is the problem that is addressed in this paper. Specifically, in this paper, we present fuzzy-cuts, a novel, knowledge-driven, graph-based method for medical image segmentation. We cast the problem of image segmentation as the maximum a posteriori (MAP) estimation of a Markov random field (MRF) which, in essence, is equivalent to the minimization of the corresponding Gibbs energy function. Considering the inherent imprecision that is common in the a priori description of objects in medical images, we propose a fuzzy theoretic model to incorporate knowledge-driven constraints into the MAP-MRF formulation. In particular, we focus on prior information about the object's location, appearance and spatial connectivity to a known seed region inside the object. To that end, we introduce fuzzy connectivity and fuzzy location priors that are used in combination to define the first-order clique potential of the Gibbs energy function. In our experiments, we demonstrate the application of the proposed method to the challenging problem of heart segmentation in non-contrast computed tomography (CT) data.

Level sets and graph cuts are two state-of-the-art image segmentation methods in use today. The two methods are apparently different from each other not only because they originate from different theory foundations but also because they employ image information in different ways — level sets typically use image information in a point-wise way, whereas graph cuts use image information in a pairwise way. In this paper, we derive an equivalence relationship between the two methods through kernel technology. In particular, we show that the kernelization of the Chan-Vese (CV) functional — a functional widely used in the level set community — is exactly the energy optimized in the average association — a well-known graph cut criterion. We refer to the level sets method using the kernelized version of the CV functional as kernel active contour. The kernel active contour has computational complexity O(n2) due to the involved kernel technology. We propose a fast implementation for kernel active contour with computational complexity only O(n) using random projection. The kernel active contour is evaluated on synthetic and real images and compared with several existing level set and graph cut methods for image segmentation.

The silhouette of the face profile is a well-known biometric that is already in use in face recognition research. One of the challenges for successful employment of this biometric is the sensitivity of its geometry to face rotation. In this paper, we introduce a new method that improves robustness to rotation. We achieve this by exploring the feature space of profiles under various rotations with the aid of a 3D face model. Based on fiducial points on the profile silhouette, we extract a set of rotation-, translation- and scale-invariant features which are used to design and train a hierarchical pose-identity classifier. In our experiments the classifier is used for the identification of a driver using his/her side-view image. We present our results on a publicly available database.

Approximately 1.5 million heart attacks are suffered annually by Americans, and about half of them prove fatal, despite a host of new public health initiatives targeting heart disease and its aggravating factors such as obesity. The case of Former US President Bill Clinton, who underwent quadruple bypass surgery, demonstrates that even a former president with access to the best medical care available can have undiagnosed heart disease. Clinton himself blamed ldquoinsufficient vigilancerdquo and stressed the importance of repeated testing as a means of heart disease prevention. Considering the large amounts of data that a comprehensive vascular health screening will produce, there is an urgent need for biomedical image analysis tools (segmentation, shape and motion estimation) to assist in screening for the conditions that underlie sudden cardiac events. In this talk, we present biomedical image analysis tools for the mining of information from cardiovascular imaging for the detection of persons with a high likelihood of developing a heart attack in the near future (vulnerable patients).

In spite of the advancement and proliferation of cardiovascular imaging, the rate of deaths due to unpredicted heart attack remains high. Thus, it becomes imperative to develop novel computational tools to mine quantitative parameters from the imaging data for early detection of asymptomatic cardiovascular disease. Coronary calcification burden has been reported to be a significant and independent predictor of the atherosclerosis disease, and is associated with future cardiac events. Additionally, increased neovascularization of the plaque has been identified as a common feature of coronary plaque inflammation and has been defined as a plaque vulnerability index. In this paper, we present methods to extract and quantify coronary calcifications in the non-contrast cardiac CT scans and to detect neovascularization in the coronary vessels using contrast-enhanced intra-vascular ultrasound imaging.

Intravascular ultrasound (IVUS) is a catheter-based medical imaging technique that produces cross-sectional images of blood vessels and is particularly useful for studying atherosclerosis. In this paper, we present a probabilistic approach for the semi-automatic identification of the luminal border on IVUS images. Specifically, we parameterize the lumen contour using a mixture of Gaussian that is deformed by the minimization of a cost function formulated using a probabilistic approach. For the optimization of the cost function, we introduce a novel method that linearly combines the descent directions of the steepest descent and BFGS optimization methods within a trust region that improves convergence. Results of our proposed method on 20 MHz IVUS images are presented and discussed in order to demonstrate the effectiveness of our approach.

In this paper, we introduce a new system for profile-based face recognition. The specific scenario involves a driver entering a gated area and using his/her side-view image (the driver remains seated in the vehicle) as identification. The system has two modes: enrollment and identification. In the enrollment mode, 3D face models of subjects are acquired and profiles extracted under different poses and stored to form a gallery database. In the identification mode, 2D images are acquired and the corresponding planar profiles are extracted and used as probes. Then, probes are matched to the gallery profiles to determine identity. The matching is accomplished using implicit shape registration via the vector distance functions. In our experiments, the approach using implicit registration exhibited higher accuracy than the iterative closest point methodology due to the use of more general transformations. The performance of our system is illustrated using a variety of databases.

In this paper, we present a novel frame-based denoising algorithm for photon-limited 3-D images. We first construct a new 3-D nonseparable filterbank by adding elements to an existing frame in a structurally stable way. In contrast with the traditional 3-D separable wavelet system, the new filterbank is capable of using edge information in multiple directions. We then propose a data-adaptive hysteresis thresholding algorithm based on this new 3-D nonseparable filterbank. In addition, we develop a new validation strategy for denoising of photon-limited images containing sparse structures, such as neurons (the structure of interest is less than 5% of total volume). The validation method, based on tubular neighborhoods around the structure, is used to determine the optimal threshold of the proposed denoising algorithm. We compare our method with other state-of-the-art methods and report very encouraging results on applications utilizing both synthetic and real data.

Prior models play an important role in the wavelet-based Bayesian image estimation problem. Although it is well known that a residual dependency structure always remains among natural image wavelet coefficients, only few multivariate prior models with a closed parametric form are available in the literature. In this paper, we develop new multivariate prior models that not only match well with the observed statistics of the wavelet coefficients of natural images, but also have a simple parametric form. These prior models are very effective for Bayesian image estimation and lead to an improved estimation performance over related earlier techniques.

Intravascularultrasound (IVUS) sequences recorded in vivo are subject to a wide array of motion artifacts as the majority of these studies are performed within the coronary arteries of a beating heart. To eliminate these artifacts, an electrocardiogram (ECG) signal is typically used to gate (collect) those frames recorded at the points in time associated with a particular fraction of the cardiac cycle. However, this technique may be suboptimal for a number of reasons, among which is the difficulty of determining the optimal fraction at which to gate. This value is generally nonobvious. To circumvent this problem, we introduce a frame-gating method for IVUS pullbacks that mimics ECG (i.e., in the sense that it selects only one frame per cardiac cycle), but will automatically choose the fraction of the cycle that renders the most stable gated frame set. Stability here is gauged by measuring interframe similarity. Our method operates exclusively on the imagery data and does not require ECG or any form of image segmentation or other high-level image analysis. To validate our algorithm, we compare its behavior versus true ECG gating.

The ten papers in this special section focus on computer vision for intravascular and intracardiac imaging. The papers are summarized here.

Three-dimensional data are increasingly being used for biometric purposes as they offer resilience to problems common mon in two-dimensional data. They have been successfully applied to face recognition and more recently to ear recognition. However, real-life biometric applications require algorithms that are both robust and efficient so that they scale well with the size of the databases. A novel ear recognition method is presented that uses a generic annotated ear model to register and fit each ear dataset. Then a compact biometric signature is extracted that retains 3D information. The proposed method is evaluated using the largest publicly available 3D ear database appended with our own database, resulting in a database containing data from multiple 3D sensor types. Using this database it is shown that the proposed method is not only robust, accurate and sensor invariant but also extremely efficient, thus making it suitable for real-life biometric applications.

Intravascular ultrasound (IVUS) is a catheter-based modality which is used to produce high-resolution, cross-sectional images of the interior of blood vessels. By capturing 2-D IVUS images continually while translating the catheter, a volumetric image of a vessel may be digitally reconstructed. To improve the quality of these volumes, electrocardiogram (ECG)-based frame gating is often applied to alleviate motion artifacts caused by the beating heart. However, there are several issues surrounding the use of ECG signals which make their use for this purpose potentially suboptimal. We introduce a method which gates pullback sequences by examining the imaging data alone, without requiring synchronous ECG, and guarantees that frames will be collected at those points in time when the heart is maximally motionless (i.e., regardless of the fraction of cardiac phase associated with those points). We compare the results of our method and of ECG on pullbacks captured in vivo in swine.

In this paper, we present the computational tools and a hardware prototype for 3D face recognition. Full automation is provided through the use of advanced multistage alignment algorithms, resilience to facial expressions by employing a deformable model framework, and invariance to 3D capture devices through suitable preprocessing steps. In addition, scalability in both time and space is achieved by converting 3D facial scans into compact metadata. We present our results on the largest known, and now publicly available, Face Recognition Grand Challenge 3D facial database consisting of several thousand scans. To the best of our knowledge, this is the highest performance reported on the FRGC v2 database for the 3D modality.

As the size of the available collections of 3D objects grows, database transactions become essential for their management with the key operation being retrieval (query). Large collections are also precategorized into classes so that a single class contains objects of the same type (e.g., human faces, cars, four-legged animals). It is shown that general object retrieval methods are inadequate for intraclass retrieval tasks. We advocate that such intraclass problems require a specialized method that can exploit the basic class characteristics in order to achieve higher accuracy. A novel 3D object retrieval method is presented which uses a parameterized annotated model of the shape of the class objects, incorporating its main characteristics. The annotated subdivision-based model is fitted onto objects of the class using a deformable model framework, converted to a geometry image and transformed into the wavelet domain. Object retrieval takes place in the wavelet domain. The method does not require user interaction, achieves high accuracy, is efficient for use with large databases, and is suitable for nonrigid object classes. We apply our method to the face recognition domain, one of the most challenging intraclass retrieval tasks. We used the Face Recognition Grand Challenge v2 database, yielding an average verification rate of 95.2 percent at 10-3 false accept rate. The latest results of our work can be found at http://www.cbl.uh.edu/UR8D/.

Associating specific gene activity with functional locations in the brain results in a greater understanding of the role of the gene. To perform such an association for the more than 20 000 genes in the mammalian genome, reliable automated methods that characterize the distribution of gene expression in relation to a standard anatomical model are required. In this paper, we propose a new automatic method that results in the segmentation of gene expression images into distinct anatomical regions in which the expression can be quantified and compared with other images. Our contribution is a novel hybrid atlas that utilizes a statistical shape model based on a subdivision mesh, texture differentiation at region boundaries, and features of anatomical landmarks to delineate boundaries of anatomical regions in gene expression images. This atlas, which provides a common coordinate system for internal brain data, is being used to create a searchable database of gene expression patterns in the adult mouse brain. Our framework annotates the images about four times faster and has achieved a median spatial overlap of up to 0.92 compared with expert segmentation in 64 images tested. This tool is intended to help scientists interpret large-scale gene expression patterns more efficiently.

In this paper, we present a new 3D face recognition approach. Full automation is provided through the use of advanced multi-stage alignment algorithms, resilience to facial expressions by employing a deformable model framework, and invariance to 3D capture devices through suitable preprocessing steps. In addition, scalability in both time and space is achieved by converting 3D facial scans into compact wavelet metadata. We present results on the largest known, and now publicly-available, Face Recognition Grand Challenge 3D facial database consisting of several thousand scans. To the best of our knowledge, our approach has achieved the highest accuracy on this dataset. 1

In this paper, we present a novel frames-based denoising algorithm. Using a general result on lifting frames, we construct a non-separable 3D frame capable of robust edge detection. This frame detects edge information by ensemble thresholding of the filtered data. The denoising uses a hysteresis thresholding step and an affine thresholding function, which are filter-adaptive and take full advantage of the threshold bounds. The threshold bounds are statistically determined from the given data for each directional filter. We compare our denoising method with other methods based on separable 3D wavelets and 3D median filtering, and report very encouraging results on applications to both synthetic and real confocal microscopy data

Traditional segmentation techniques do not quite meet the challenges posed by inherently fuzzy medical images. Image segmentation based on fuzzy connectedness addresses this problem by attempting to capture both closeness, based on characteristic intensity, and "hanging togetherness," based on intensity homogeneity, of image elements to the target object. This paper presents a modification and extension of previously published image segmentation algorithms based on fuzzy connectedness, which is computed as a linear combination of an object-feature-based and a homogeneity-based component using fixed weights. We provide a method, called fuzzy connectedness using dynamic weights (DyW), to introduce directional sensitivity to the homogeneity-based component and to dynamically adjust the linear weights in the functional form of fuzzy connectedness. Dynamic computation of the weights relieves the user of the exhaustive search process to find the best combination of weights suited to a particular application. This is critical in applications such as analysis of cardiac cine magnetic resonance (MR) images, where the optimal combination of affinity component weights can vary for each slice, each phase, and each subject, in spite of data being acquired from the same MR scanner with identical protocols. We present selected results of applying DyW to segment phantom images and actual MR, computed tomography, and infrared data. The accuracy of DyW is assessed by comparing it to two different formulations of fuzzy connectedness. Our method consistently achieves accuracy of more than 99.15% for a range of image complexities: contrast 5%-65%, noise-to-contrast ratio of 6%-18%, and bias field of four types with maximum gain factor of up to 10%.

We present a general mathematical theory for lifting frames that allows us to modify existing filters to construct new ones that form Parseval frames. We apply our theory to design nonseparable Parseval frames from separable (tensor) products of a piecewise linear spline tight frame. These new frame systems incorporate the weighted average operator, the Sobel operator, and the Laplacian operator in directions that are integer multiples of 45 degrees. A new image denoising algorithm is then proposed, tailored to the specific properties of these new frame filters. We demonstrate the performance of our algorithm on a diverse set of images with very encouraging results.

It is becoming increasingly important to be able to credential and identify authorized personnel at key points of entry. Such identity management systems commonly employ biometric identifiers. In this paper, we present a novel multimodal facial recognition approach that employs data from both visible spectrum and thermal infrared sensors. Data from multiple cameras is used to construct a three-dimensional mesh representing the face and a facial thermal texture map. An annotated face model with explicit two-dimensional parameterization (UV) is then fitted to this data to construct: 1) a three-channel UV deformation image encoding geometry, and 2) a one-channel UV vasculature image encoding facial vasculature. Recognition is accomplished by comparing: 1) the parametric deformation images, 2) the parametric vasculature images, and 3) the visible spectrum texture maps. The novelty of our work lies in the use of deformation images and physiological information as means for comparison. We have performed extensive tests on the Face Recognition Grand Challenge v1.0 dataset and on our own multimodal database with very encouraging results.

Biometrics-based technologies in the area of identity management are gaining increasing importance, as a means of establishing non-falsifiable credentials for end users. However, in the three-way tug-of-war between convenient, unobtrusive data collection (required for user acceptance), accuracy in results (required for justifying deployment), and speed (required for widespread use in practice), no single biometric to date has managed to hold the middle ground that would allow for its ready adoption. The overall goal of our project is to develop the theoretical framework and computational tools that will lead to the development of a practical, unobtrusive, and accurate face recognition system for convenient and effective access control. This framework encompasses 8D characteristics of the face (3D geometry+2D visible texture+2D infrared texture, over time). In this paper, we present a novel multi-modal facial recognition approach that employs data from both visible spectrum and thermal infrared sensors. From the fitted parametric model we extract two images corresponding to the subject's face and process these images to extract biometric signatures. Specifically, the deformation image is compressed using a wavelet transform and the vasculature graph is extracted from the parametric thermal image.

We present a general mathematical theory for lifting frames that allows us to modify existing filters to construct new ones that form Parseval frames. We apply our theory to design non-separable Parseval frames from separable (tensor) products of a piecewise linear spline tight frame. These new frame systems incorporate the weighted average operator and the Sobel operator in directions that are integer multiples of 45/spl deg/. A new image denoising algorithm is then proposed tailored to the specific properties of these new frame filters. We demonstrate the performance of our algorithm on a diverse set of images with very encouraging results.

We propose a new method to denoise a surface. This method is motivated by the Laplacian flow and the theory of tight frames. Mesh denoising is achieved by building local Wiener filtering into the detail representation of the surface. We have performed a number of experiments to assess the accuracy, advantages and limitations of our approach with very encouraging results.

This paper presentes a novel algorithm for the voxelization of surface models of arbitrary topology. Our algorithm uses the depth and stencil buffers, available in most commercial graphics hardware, to achieve high performance. It is suitable for both polygonal meshes and parametric surfaces. Experiments highlight the advantages and limitations of our approach.

In this paper, we construct a new class of deformable models using new biorthogonal wavelets, named Generalized Hermite Distributed Approximating Functional (g-HDAF) Wavelets. The scaling functions of this new family are symmetric and the corresponding wavelets op-timize their smoothness for a given number of vanishing moments. In addition, we embed these multiresolution deformable models to the physics-based deformable model framework and use them for fitting 3D range data. We have performed a number of experiments with both synthetic and real data with very encouraging results. 1

In this paper, we present a non-intrusive method for human motion estima-tion from a monocular video camera for the teleoperation of ROBONAUT (ROBOtic astroNAUT). ROBONAUT is an anthropomorphic robot devel-oped at NASA- JSC, which is capable of dextrous, human- like maneuvers to handle common extravehicular activity tools. The human operator is repre-sented using an articulated three-dimensional model consisting of rigid links connected by spherical joints. The shape of a link is described by a triangu-lar mesh and its motion by six parameters: one three-dimensional translation vector and three rotation angles. The motion parameters of the links are estimated by maximizing the conditional probability of the frame-to-frame intensity differences at observation points. The algorithm was applied to real test sequences of a moving arm with very encouraging results. Specifically, the mean error for the derived wrist position (using the estimated motion pa-rameters) was 0.570.31 cm. The motion estimates were used to remotely command a robonaut simulation developed at NASA- JSC. 1

In this paper we present a computational framework that provides the automatic analysis of spotted DNA microarray image data. The challenges are in providing an accurate representation of microarray hybridization observations while minimizing user interaction. To obtain this, we need to segment the observation data and subsequent correction for true hybridization level measurements must be accomplished against the backdrop of signal noise, background signal variation, and spatial non-uniformity in the array layout. With the requirements of automation and accuracy, an approach based on data-driven denoising, array addressing, background estimation, and spot segmentation was developed We proceeded to validate our approach on synthetic data as well as the publicly available raw and analyzed microarray data from the published Stanford yeast cell cycle analysis project. Spot mean and total intensities were examined as well as spot background estimates. By minimizing the user role, a main bottleneck in microarray data analysis is removed, allowing for more immediate analysis of large observation data sets. Our implementation has proven to be relatively fast, and the results of our approach have been encouraging.

We present a technique for the automatic adaptation of a deformable model's elastic parameters within a Kalman filter framework for shape estimation applications. The novelty of the technique is that the model's elastic parameters are not constant, but spatio-temporally varying. The variation of the elastic parameters depends on the distance of the model from the data and the rate of change of this distance. Each pass of the algorithm uses physics-based modeling techniques to iteratively adjust both the geometric and the elastic degrees of freedom of the model in response to forces that are computed from the discrepancy between the model and the data. By augmenting the state equations of an extended Kalman filter to incorporate these additional variables, we are able to significantly improve the quality of the shape estimation. Therefore, the model's elastic parameters are always initialized to the same value and they are subsequently modified depending on the data and the noise distribution. We present results demonstrating the effectiveness of our method for both two-dimensional and three-dimensional data.

In this paper, we present a four-step technique for simultaneously estimating a human's anthropometric measurements (up to a scale parameter) and pose from a single image. The user initially selects a set of image points that constitute the projection of selected landmark. Using this information, along with a priori statistical information about the human body, a set of plausible segment length estimates are generated. The third step produces a set of plausible poses based on joint limit constraints using a geometric method. In the fourth step, pose and anthropometric measurements are obtained by minimizing an appropriate cost function subject to the associated constraints. The novelty of our approach is the use of anthropometric statistics to constrain the estimation process that allows the simultaneous estimation of both anthropometry and pose. We demonstrate the accuracy, advantages and limitations of our method for various classes of both synthetic and real input data.

This paper presents the formulations and techniques that we have developed for the 3D model-based, motion estimation of human movement from multiple cameras. Our method is based on the spatio-temporal analysis of the subject's silhouette and it has the advantage that the subject does not have to wear markers or other devices. We present tracking results from experiments involving the recovery of complex motions in the presence of significant occlusion.

The paper presents a system for animating customized virtual humans using motion parameters estimated from multi-view image sequences. The advantage of the method is that the subject does not have to wear markers or other devices. The authors offer a detailed analysis of the singularities that might arise during tracking humans using multiple cameras and provide an analysis of the criteria that allow the ordering of the cameras that provide the most information for tracking. They demonstrate their system using image sequences requiring the recovery of complex three-dimensional motions in the presence of significant occlusion, and present a quantitative analysis of the accuracy of the estimated motion parameters.

We present a new integrated approach to the two-dimensional part segmentation, shape, and motion estimation of moving multipart objects. Our technique exploits the relationship between the geometry and the observed deformations of the apparent contour of a moving multipart object and its structure. The novelty of the technique is that no prior model of the object or of its parts is employed. We develop aPart Segmentation Algorithm(PSA) that recursively recovers all the moving parts of an object by monitoring and reasoning over the changes of its deforming apparent contour. To parameterize and segment over time a deforming apparent contour, we fit initially a single deformable model whose global and local deformations over time allow us to hypothesize an underlying part structure. This hypothesis is verified by further monitoring the relative motion among the model's parts and the satisfaction of certain criteria. Upon verifying the part hypothesis, the initial deformable model is split into two or more models that better fit the apparent contour. This recursive operation allows the refinement over time of the number and shape of the extracted parts. When multiple deformable models are used to model the apparent contour of a multipart object, there is an uncertainty concerning the deformable model to which the data points should apply forces to. To address this problem, we present a new algorithm for force assignment that assigns forces from the data to multiple models. This algorithm allows partial overlap between the parts' models and the determination of their joint location. Finally, the effectiveness of the approach is demonstrated through a series of experiments involving a variety of objects.

We present a new method for the 3D model-based tracking of human body parts. To mitigate the difficulties arising due to occlusion among body parts, we employ multiple calibrated cameras in a mutually orthogonal configuration. In addition, we develop criteria for a time varying active selection of a set of cameras to track the motion of a particular human part. In particular, at every frame, each camera tracks a number of parts depending on the visibility of these parts and the observability of their predicted motion from the specific camera. To relate points on the occluding contours of the parts to points on their models we apply concepts from projective geometry. Then, within the physics-based framework we compute the generalized forces applied from the parts' occluding contours to model points of the body parts. These forces update the translational and rotational degrees of freedom of the model, such as to minimize the discrepancy between the sensory data and the estimated model state. We present initial tracking results from a series of experiments involving the recovery of complex 3D motions in the presence of significant occlusion.

We present a novel motion-based approach for the part determination and shape estimation of a human's body parts. The novelty of the technique is that neither a prior model of the human body is employed nor prior body part segmentation is assumed. We present a human body part identification strategy (HBPIS) that recovers all the body parts of a moving human based on the spatiotemporal analysis of its deforming silhouette. We formalize the process of simultaneous part determination and 2D shape estimation by employing the supervisory control theory of discrete event systems. In addition, in order to acquire the 3D shape of the body parts, we present a new algorithm which selectively integrates the (segmented by the HBPIS) apparent contours, from three mutually orthogonal views. The effectiveness of the approach is demonstrated through a series of experiments, where a subject performs a set of movements according to a protocol that reveals the structure of the human body.>

We present a novel, robust, integrated approach to segmentation shape and motion estimation of articulated objects. Initially, we assume the object consists of a single part, and we fit a deformable model to the given data using our physics-based framework. As the object attains new postures, we decide based on certain criteria if and when to replace the initial model with two new models. These criteria are based on the model's state and the given data. We then fit the models to the data using a novel algorithm for assigning forces from the data to the two models, which allows partial overlap between them and determination of joint location. This approach is applied iteratively until all the object's moving parts are identified. Furthermore, we define new global deformations and we demonstrate our technique in a series of experiments, where Kalman filtering is employed to account for noise and occlusion.>