In this work, we aim to enhance model-based face reconstruction by avoiding fitting the model to outliers, i.e. regions that cannot be well-expressed by the model such as occluders or makeup. The core challenge for localizing outliers is that they are highly variable and difficult to annotate. To overcome this challenging problem, we introduce a joint Face-autoencoder and outlier segmentation approach (FOCUS). In particular, we exploit the fact that the outliers cannot be fitted well by the face model and hence can be localized well given a high-quality model fitting. The main challenge is that the model fitting and the outlier segmentation are mutually dependent on each other, and need to be inferred jointly. We resolve this chicken-and-egg problem with an EM-type training strategy, where a face autoencoder is trained jointly with an outlier segmentation network. This leads to a synergistic effect, in which the segmentation network prevents the face encoder from fitting to the outliers, enhancing the reconstruction quality. The improved 3D face reconstruction, in turn, enables the segmentation network to better predict the outliers. To resolve the ambiguity between outliers and regions that are difficult to fit, such as eyebrows, we build a statistical prior from synthetic data that measures the systematic bias in model fitting. Experiments on the NoW testset demonstrate that FOCUS achieves SOTA 3D face reconstruction performance among all baselines trained without 3D annotation. Moreover, our results on CelebA-HQ and AR database show that the segmentation network can localize occluders accurately despite being trained without any segmentation annotation.

In this article, we provide a detailed survey of 3D Morphable Face Models over the 20 years since they were first proposed. The challenges in building and applying these models, namely, capture, modeling, image formation, and image analysis, are still active research topics, and we review the state-of-the-art in each of these areas. We also look ahead, identifying unsolved challenges, proposing directions for future research, and highlighting the broad range of current and future applications.

In this work, we consider the problem of learning a hierarchical generative model of an object from a set of images which show examples of the object in the presence of variable background clutter. Existing approaches to this problem are limited by making strong a-priori assumptions about the object’s geometric structure and require seg- mented training data for learning. In this paper, we propose a novel framework for learning hierarchical compositional models (HCMs) which do not suffer from the mentioned limitations. We present a generalized formulation of HCMs and describe a greedy structure learning framework that consists of two phases: Bottom-up part learning and top-down model composition. Our framework integrates the foreground-background segmentation problem into the structure learning task via a background model. As a result, we can jointly optimize for the number of layers in the hierarchy, the number of parts per layer and a foreground- background segmentation based on class labels only. We show that the learned HCMs are semantically meaningful and achieve competitive results when compared to other generative object models at object classification on a standard transfer learning dataset.

We propose a photo-realistic method for artificially ageing facial photographs by combining learned shape deformations with skin detail transfer between a donor and a receiver face. Facial ageing is a complicated process that most existing face models, such as 3d Morphable Models fail to express, due to lacking correspondence between the wrinkles of different individuals. We propose an exemplar-based approach to face ageing, where we transfer high-frequency details from an older face texture to a younger receiver. By warping the resulting image according to a learned shape ageing deformation, we obtain photo-realistic aged photographs. We evaluate the simulator with human perception experiments showing that we can indeed create results that are perceived to be real.

We present a novel method for co-registration of two independent statistical shape models. We solve the problem of aligning a face model to a skull model with stochastic optimization based on Markov Chain Monte Carlo (MCMC). We create a probabilistic joint face-skull model and show how to obtain a distribution of plausible face shapes given a skull shape. Due to environmental and genetic factors, there exists a distribution of possible face shapes arising from the same skull. We pose facial reconstruction as a conditional distribution of plausible face shapes given a skull shape. Because it is very difficult to obtain the distribution directly from MRI or CT data, we create a dataset of artificial face-skull pairs. To do this, we propose to combine three data sources of independent origin to model the joint face-skull distribution: a face shape model, a skull shape model and tissue depth marker information. For a given skull, we compute the posterior distribution of faces matching the tissue depth distribution with Metropolis-Hastings. We estimate the joint face-skull distribution from samples of the posterior. To find faces matching to an unknown skull, we estimate the probability of the face under the joint face-skull model. To our knowledge, we are the first to provide a whole distribution of plausible faces arising from a skull instead of only a single reconstruction. We show how the face-skull model can be used to rank a face dataset and on average successfully identify the correct match in top 30%. The face ranking even works when obtaining the face shapes from 2D images. We furthermore show how the face-skull model can be useful to estimate the skull position in an MR-image.

In this paper, we present a novel open-source pipeline for face registration based on Gaussian processes as well as an application to face image analysis. Non-rigid registration of faces is significant for many applications in computer vision, such as the construction of 3D Morphable face models (3DMMs). Gaussian Process Morphable Models (GPMMs) unify a variety of non-rigid deformation models with B-splines and PCA models as examples. GPMM separate problem specific requirements from the registration algorithm by incorporating domain-specific adaptions as a prior model. The novelties of this paper are the following: (i) We present a strategy and modeling technique for face registration that considers symmetry, multi-scale and spatially-varying details. The registration is applied to neutral faces and facial expressions. (ii) We release an open-source software framework for registration model-building demonstrated on the publicly available BU3D-FE database. The released pipeline also contains an implementation of an Analysis-by-Synthesis model adaption of 2D face images, tested on the Multi-PIE and LFW database. This enables the community to reproduce, evaluate and compare the individual steps of registration to model-building and 3D/2D model fitting. (iii) Along with the framework release, we publish a new version of the Basel Face Model (BFM-2017) with an improved age distribution and an additional facial expression model.

We propose a novel stochastic generative parametric freckle model for the analysis and synthesis of human faces. Morphable Models are the state-of-the-art generative parametric face models. However, they are unable to synthesize freckles which are part of atural face variation. The deficiency lies in requiring point-to-point correspondence on the texture pixels. We propose to assume a correspondence between freckle density and not the freckles themselves. We propose a model that is stochastic, generative, and parametric and generates freckles with a point process according to a density and size distribution. The resulting model can synthesize photo-realistic freckles according to observations as well as add freckles to existing faces. We create more realistic faces than with Morphable Models alone and allow for detailed face pigment analysis.

Models of shape variations have become a central component for the automated analysis of images. An important class of shape models are point distribution models (PDMs). These models represent a class of shapes as a normal distribution of point variations, whose parameters are estimated from example shapes. Principal component analysis (PCA) is applied to obtain a low-dimensional representation of the shape variation in terms of the leading principal components. In this paper, we propose a generalization of PDMs, which we refer to as Gaussian Process Morphable Models (GPMMs). We model the shape variations with a Gaussian process, which we represent using the leading components of its Karhunen-Loève expansion. To compute the expansion, we make use of an approximation scheme based on the Nyström method. The resulting model can be seen as a continuous analog of a standard PDM. However, while for PDMs the shape variation is restricted to the linear span of the example data, with GPMMs we can define the shape variation using any Gaussian process. For example, we can build shape models that correspond to classical spline models and thus do not require any example data. Furthermore, Gaussian processes make it possible to combine different models. For example, a PDM can be extended with a spline model, to obtain a model that incorporates learned shape characteristics but is flexible enough to explain shapes that cannot be represented by the PDM. We introduce a simple algorithm for fitting a GPMM to a surface or image. This results in a non-rigid registration approach whose regularization properties are defined by a GPMM. We show how we can obtain different registration schemes, including methods for multi-scale or hybrid registration, by constructing an appropriate GPMM. As our approach strictly separates modeling from the fitting process, this is all achieved without changes to the fitting algorithm. To demonstrate the applicability and versatility of GPMMs, we perform a set of experiments in typical usage scenarios in medical image analysis and computer vision: The model-based segmentation of 3D forearm images and the building of a statistical model of the face. To complement the paper, we have made all our methods available as open source.

We propose an efficient self-shadowing illumination model for Morphable Models. Simulating self-shadowing with ray casting is computationally expensive which makes them impractical in Analysis-by-Synthesis methods for object reconstruction from single images. Therefore, we propose to learn self-shadowing for Morphable Model parameters directly with a linear model. Radiance transfer functions are a powerful way to represent self-shadowing used within the precomputed radiance transfer framework (PRT). We build on PRT to render deforming objects with self-shadowing at interactive frame rates. It can be illuminated efficiently by environment maps represented with spherical harmonics. The result is an efficient global illumination method for Morphable Models, exploiting an approximated radiance transfer. We apply the method to fitting Morphable Model parameters to a single image of a face and demonstrate that considering self-shadowing improves shape reconstruction.

We propose a novel narrow baseline multiview stereo surface reconstruction method that is specifically aimed at complex shapes of biological origin that show many thin protrusions and curved occluding contours. Our method is built around fitting local quadrics to occluding contours and it thus avoids any planarity assumptions that are common to other state of the art methods. We describe a complete pipeline that begins with calibrated noisy images and produces a final watertight surface. We present a novel technique to detect pixel precise internal contours and to fit local quadrics to them. This procedure is designed to deal with curved occluding contours, and it is very robust to noise and to slow changes in surface radiance. Our method can even reconstruct shapes from sequences where the illumination is attached to the observer and not the scene. We demonstrate the potential of our method by reconstructing the intricate shape of a tiny insect from images taken under a scanning electron microscope.

We present a novel image segmentation method based on statistical shape model fitting. Instead of fitting the model to raw intensity values we consider object probabilities. The abstraction from the plain intensity images to probability maps makes the segmentation more robust against misleading texture inside the object or surrounding background. The target object probability is predicted based on random forest regression trained with neighborhood dependent features of sample images. In contrast to similar approaches, both, the object boundary as well as the whole object and background region are considered for segmentation. We apply our approach to a 3D cone beam computed tomography image dataset of the jaw region where we segment the wisdom tooth shape. Compared to a boundary-and a region-based method we obtain superior segmentation performance.

In video post-production it is often necessary to track interest points in the video. This is called off-line tracking, because the complete video is available to the algorithm and can be contrasted with on-line tracking, where an incoming stream is tracked in real time. Off-line tracking should be accurate and - if used interactively - needs to be fast, preferably faster than real-time. We describe a 50 to 100 frames per second off-line tracking algorithm, which globally maximizes the probability of the track given the complete video. The algorithm is more reliable than previous methods because it explains the complete frames, not only the patches of the final track, making as much use of the data as possible. It achieves efficiency by using a greedy search strategy with deferred cost evaluation, focusing the computational effort on the most promising track candidates while finding the globally optimal track.

Fitting statistical 2D and 3D shape models to images is necessary for a variety of tasks, such as video editing and face recognition. Much progress has been made on local fitting from an initial guess, but determining a close enough initial guess is still an open problem. One approach is to detect distinct landmarks in the image and initalize the model fit from these correspondences. This is difficult, because detection of landmarks based only on the local appearance is inherently ambiguous. This makes it necessary to use global shape information for the detections. We propose a method to solve the combinatorial problem of selecting out of a large number of candidate landmark detections the configuration which is best supported by a shape model. Our method, as opposed to previous approaches, always finds the globally optimal configuration. The algorithm can be applied to a very general class of shape models and is independent of the underlying feature point detector. Its theoretic optimality is shown, and it is evaluated on a large face dataset.

Generative 3D face models are a powerful tool in computer vision. They provide pose and illumination invariance by modeling the space of 3D faces and the imaging process. The power of these models comes at the cost of an expensive and tedious construction process, which has led the community to focus on more easily constructed but less powerful models. With this paper we publish a generative 3D shape and texture model, the Basel face model (BFM), and demonstrate its application to several face recognition task. We improve on previous models by offering higher shape and texture accuracy due to a better scanning device and less correspondence artifacts due to an improved registration algorithm. The same 3D face model can be fit to 2D or 3D images acquired under different situations and with different sensors using an analysis by synthesis method. The resulting model parameters separate pose, lighting, imaging and identity parameters, which facilitates invariant face recognition across sensors and data sets by comparing only the identity parameters. We hope that the availability of this registered face model will spur research in generative models. Together with the model we publish a set of detailed recognition and reconstruction results on standard databases to allow complete algorithm comparisons.

Efficient and accurate fitting of active appearance models (AAM) is a key requirement for many applications. The most efficient fitting algorithm today is inverse compositional image alignment (ICIA). While ICIA is extremely fast, it is also known to have a small convergence radius. Convergence is especially bad when training and testing images differ strongly, as in multi-person AAMs. We describe “forward” compositional image alignment in a consistent framework which also incorporates methods previously termed “inverse” compositional, and use it to develop two novel fitting methods. The first method, compositional gradient descent (CoDe), is approximately four times slower than ICIA, while having a convergence radius which is even larger than that achievable by direct quasi-Newton descent. An intermediate convergence range with the same speed as ICIA is achieved by LinCoDe, the second new method. The success rate of the novel methods is 10 to 20 times higher than that of the original ICIA method.

Non-rigid registration is central to many problems in computer vision and medical image analysis. We propose a registration algorithm which is regularized by prior knowledge in the form of a statistical deformation model. This model is obtained from previous registrations performed on a set of noise-free training examples given by images, or shapes represented by level set functions. Contrary to similar approaches, our method does not strictly constrain the result to lie in the span of the statistical model but rather uses the model for Tikhonov regularization. Therefore, our method can be used to reduce the influence of noise and artifacts even when the model contains only a few typical examples. This automatically gives rise to a bootstrapping strategy for building statistical models from noisy data sets requiring only a limited number of high quality examples. We demonstrate the effectiveness of the approach on synthetic and medical images.

We describe an expression-invariant method for face recognition by fitting an identity/expression separated 3D Morphable Model to shape data. The expression model greatly improves recognition and retrieval rates in the uncooperative setting, while achieving recognition rates on par with the best recognition algorithms in the face recognition great vendor test. The fitting is performed with a robust nonrigid ICP algorithm. It is able to perform face recognition in a fully automated scenario and on noisy data. The system was evaluated on two datasets, one with a high noise level and strong expressions, and the standard UND range scan database, showing that while expression invariance increases recognition and retrieval performance for the expression dataset, it does not decrease performance on the neutral dataset. The high recognition rates are achieved even with a purely shape based method, without taking image data into account.

We present a method for face recognition by fitting a 3D Morphable Model to shape data. Fitting is done with a a robust nonrigid ICP algorithm. For recognition, it is possible to use either the fitted model parameters, or the correspondences induced by the model. We compare different similarity measures, and show that a 3D Morphable Model allows very robust retrieval results.

In this paper, a novel method for reducing the runtime complexity of a support vector machine classifier is presented. The new training algorithm is fast and simple. This is achieved by an over-complete wavelet transform that finds the optimal approximation of the support vectors. The presented derivation shows that the wavelet theory provides an upper bound on the distance between the decision function of the support vector machine and our classifier. The obtained classifier is fast, since a Haar wavelet approximation of the support vectors is used, enabling efficient integral image-based kernel evaluations. This provides a set of cascaded classifiers of increasing complexity for an early rejection of vectors easy to discriminate. This excellent runtime performance is achieved by using a hierarchical evaluation over the number of incorporated and additional over the approximation accuracy of the reduced set vectors. Here, this algorithm is applied to the problem of face detection, but it can also be used for other image-based classifications. The algorithm presented, provides a 530-fold speedup over the support vector machine, enabling face detection at more than 25 fps on a standard PC.

We show how to extend the ICP framework to nonrigid registration, while retaining the convergence properties of the original algorithm. The resulting optimal step nonrigid ICP framework allows the use of different regularisations, as long as they have an adjustable stiffness parameter. The registration loops over a series of decreasing stiffness weights, and incrementally deforms the template towards the target, recovering the whole range of global and local deformations. To find the optimal deformation for a given stiffness, optimal iterative closest point steps are used. Preliminary correspondences are estimated by a nearest-point search. Then the optimal deformation of the template for these fixed correspondences and the active stiffness is calculated. Afterwards the process continues with new correspondences found by searching from the displaced template vertices. We present an algorithm using a locally affine regularisation which assigns an affine transformation to each vertex and minimises the difference in the transformation of neighbouring vertices. It is shown that for this regularisation the optimal deformation for fixed correspondences and fixed stiffness can be determined exactly and efficiently. The method succeeds for a wide range of initial conditions, and handles missing data robustly. It is compared qualitatively and quantitatively to other algorithms using synthetic examples and real world data.

This paper presents a novel framework to localize in a photograph prominent irregularities in facial skin, in particular nevi (moles, birthmarks). Their characteristic configuration over a face is used to encode the person's identity independent of pose and illumination. This approach extends conventional recognition methods, which usually disregard such small scale variations and thereby miss potentially highly discriminative features. Our system detects potential nevi with a very sensitive multi scale template matching procedure. The candidate points are filtered according to their discriminative potential, using two complementary methods. One is a novel skin segmentation scheme based on gray scale texture analysis that we developed to perform outlier detection in the face. Unlike most other skin detection/segmentation methods it does not require color input. The second is a local saliency measure to express a point's uniqueness and confidence taking the neighborhood's texture characteristics into account. We experimentally evaluate the suitability of the detected features for identification under different poses and illumination on a subset of the FERET face database.

This paper presents a novel statistical shape model that can be used to detect and localise feature points of a class of objects in images. The shape model is inspired from the 3D morphable model (3DMM) and has the property to be viewpoint invariant. This shape model is used to estimate the probability of the position of a feature point given the position of reference feature points, accounting for the uncertainty of the position of the reference points and of the intrinsic variability of the class of objects. The viewpoint invariant detection algorithm maximises a foreground/background likelihood ratio of the relative position of the feature points, their appearance, scale, orientation and occlusion state. Computational efficiency is obtained by using the Bellman principle and an early rejection rule based on 3D to 2D projection constraints. Evaluations of the detection algorithm on the CMU-P1E face images and on a large set of non-face images show high levels of accuracy (zero false alarms for more than 90% detection rate). As well as locating feature points, the detection algorithm also estimates the pose of the object and a few shape parameters. It is shown that it can be used to initialise a 3DMM fitting algorithm and thus enables a fully automatic viewpoint and lighting invariant image analysis solution.

We present a novel model based stereo system, which accurately extracts the 3D shape and pose of faces from multiple images taken simultaneously. Extracting the 3D shape from images is important in areas such as pose-invariant face recognition and image manipulation. The method is based on a 3D morphable face model learned from a database of facial scans. The use of a strong face prior allows us to extract high precision surfaces from stereo data of faces, where traditional correlation based stereo methods fail because of the mostly textureless input images. The method uses two or more uncalibrated images of arbitrary baseline, estimating calibration and shape simultaneously. Results using two and three input images are presented. We replace the lighting and albedo estimation of a monocular method with the use of stereo information, making the system more accurate and robust. We evaluate the method using ground truth data and the standard PIE image dataset. A comparison with the state of the art monocular system shows that the new method has a significantly higher accuracy.

New product development involves people with different backgrounds. Designers, engineers, and consumers all have different criteria, and these criteria interact. Early concepts evolve in this kind of collaborative context, and there is a need for dynamic visualization of the interaction between design shape and other shape-related design criteria. In this paper, a Morphable Model is defined from simplified representations of suitably chosen real cars, providing a continuous shape space to navigate, manipulate, and visualize. Physical properties and consumer-provided scores for the real cars (such as 'weight' and 'sportiness') are estimated for new designs across the shape space. This coupling allows one to manipulate the shape directly while reviewing the impact on estimated criteria, or conversely, to manipulate the criterial values of the current design to produce a new shape with more desirable attributes.

Unconstrained illumination and pose variation lead to significant variation in the photographs of faces and constitute a major hurdle preventing the widespread use of face recognition systems. The challenge is to generalize from a limited number of images of an individual to a broad range of conditions. Recently, advances in modeling the effects of illumination and pose have been accomplished using three-dimensional (3-D) shape information coupled with reflectance models. Notable developments in understanding the effects of illumination include the nonexistence of illumination invariants, a characterization of the set of images of objects in fixed pose under variable illumination (the illumination cone), and the introduction of spherical harmonics and low-dimensional linear subspaces for modeling illumination. To generalize to novel conditions, either multiple images must be available to reconstruct 3-D shape or, if only a single image is accessible, prior information about the 3-D shape and appearance of faces in general must be used. The 3-D Morphable Model was introduced as a generative model to predict the appearances of an individual while using a statistical prior on shape and texture allowing its parameters to be estimated from single image. Based on these new understandings, face recognition algorithms have been developed to address the joint challenges of pose and lighting. In this paper, we review these developments and provide a brief survey of the resulting face recognition algorithms and their performance

This paper presents a method for face recognition across large changes in viewpoint. Our method is based on a morphable model of 3D faces that represents face-specific information extracted from a dataset of 3D scans. For non-frontal face recognition in 2D still images, the morphable model can be incorporated in two different approaches: in the first, it serves as a preprocessing step by estimating the 3D shape of novel faces from the non-frontal input images, and generating frontal views of the reconstructed faces at a standard illumination using 3D computer graphics. The transformed images are then fed into state-of-the-art face recognition systems that are optimized for frontal views. This method was shown to be extremely effective in the Face Recognition Vendor Test FRVT 2002. In the process of estimating the 3D shape of a face from an image, a set of model coefficients are estimated. In the second method, face recognition is performed directly from these coefficients. In this paper we explain the algorithm used to preprocess the images in FRVT 2002, present additional FRVT 2002 results, and compare these results to recognition from the model coefficients.

We present a novel algorithm aiming to estimate the 3D shape, the texture of a human face, along with the 3D pose and the light direction from a single photograph by recovering the parameters of a 3D morphable model. Generally, the algorithms tackling the problem of 3D shape estimation from image data use only the pixels intensity as input to drive the estimation process. This was previously achieved using either a simple model, such as the Lambertian reflectance model, leading to a linear fitting algorithm. Alternatively, this problem was addressed using a more precise model and minimizing a non-convex cost function with many local minima. One way to reduce the local minima problem is to use a stochastic optimization algorithm. However, the convergence properties (such as the radius of convergence) of such algorithms, are limited. Here, as well as the pixel intensity, we use various image features such as the edges or the location of the specular highlights. The 3D shape, texture and imaging parameters are then estimated by maximizing the posterior of the parameters given these image features. The overall cost function obtained is smoother and, hence, a stochastic optimization algorithm is not needed to avoid the local minima problem. This leads to the multi-features fitting algorithm that has a wider radius of convergence and a higher level of precision. This is shown on some example photographs, and on a recognition experiment performed on the CMU-PIE image database.

Abstract Pasting somebody's face into an existing image with traditional photo retouching and digital image processing tools has only been possible if both images show the face from the same viewpoint and with the same illumination. However, this is rarely the case for given pairs of images. We present a system that exchanges faces across large differences in viewpoint and illumination. It is based on an algorithm that estimates 3D shape and texture along with all relevant scene parameters, such as pose and lighting, from single images. Manual interaction is reduced to clicking on a set of about 7 feature points, and marking the hairline in the target image. The system can be used for image processing, virtual try‐on of hairstyles, and face recognition. By separating face identity from imaging conditions, our approach provides an abstract representation of images and a novel, high‐level tool for image manipulation. Categories and Subject Descriptors (according to ACM CCS): I.3.7 [Computer Graphics]: Animation

3D morphable models, as a means to generate images of a class of objects and to analyze them, have become increasingly popular. The problematic part of this framework is the registration of the model to an image, a.k.a. the fitting. The characteristic features of a fitting algorithm are its efficiency, robustness, accuracy and automation. Many accurate algorithms based on gradient descent techniques exist which are unfortunately short on the other features. Recently, an efficient algorithm called inverse compositional image alignment (ICIA) algorithm, able to fit 2D images, was introduced. We extent this algorithm to fit 3D morphable models using a novel mathematical notation which facilitates the formulation of the fitting problem. This formulation enables us to avoid a simplification so far used in the ICIA, being as efficient and leading to improved fitting precision. Additionally, the algorithm is robust without sacrificing its efficiency and accuracy, thereby conforming to three of the four characteristics of a good fitting algorithm.

Abstract This paper presents a method for photo‐realistic animation that can be applied to any face shown in a single imageor a video. The technique does not require example data of the person's mouth movements, and the image to beanimated is not restricted in pose or illumination. Video reanimation allows for head rotations and speech in theoriginal sequence, but neither of these motions is required. In order to animate novel faces, the system transfers mouth movements and expressions across individuals, basedon a common representation of different faces and facial expressions in a vector space of 3D shapes and textures.This space is computed from 3D scans of neutral faces, and scans of facial expressions. The 3D model's versatility with respect to pose and illumination is conveyed to photo‐realistic image and videoprocessing by a framework of analysis and synthesis algorithms: The system automatically estimates 3D shape andall relevant rendering parameters, such as pose, from single images. In video, head pose and mouth movements aretracked automatically. Reanimated with new mouth movements, the 3D face is rendered into the original images. Categories and Subject Descriptors (according to ACM CCS): I.3.7 [Computer Graphics]: Animation

This paper presents a method for face recognition across variations in pose, ranging from frontal to profile views, and across a wide range of illuminations, including cast shadows and specular reflections. To account for these variations, the algorithm simulates the process of image formation in 3D space, using computer graphics, and it estimates 3D shape and texture of faces from single images. The estimate is achieved by fitting a statistical, morphable model of 3D faces to images. The model is learned from a set of textured 3D scans of heads. We describe the construction of the morphable model, an algorithm to fit the model to images, and a framework for face identification. In this framework, faces are represented by model parameters for 3D shape and texture. We present results obtained with 4,488 images from the publicly available CMU-PIE database and 1,940 images from the FERET database.

We describe an algorithm for automatically learning discriminative com- ponents of objects with SVM classifiers. It is based on growing image parts by minimizing theoretical bounds on the error probability of an SVM. Component-based face classifiers are then combined in a second stage to yield a hierarchical SVM classifier. Experimental results in face classification show considerable robustness against rotations in depth and suggest performance at significantly better level than other face detection systems. Novel aspects of our approach are: a) an algorithm to learn component-based classification experts and their combination, b) the use of 3-D morphable models for training, and c) a maximum operation on the output of each component classifier which may be relevant for bio- logical models of visual recognition.

This paper proposes a method for face reconstruction that makes use of only a small set of feature points. Faces can be modeled by forming linear combinations of prototypes of shape and texture information. With the shape and future information at the feature points alone, we can achieve only an approximation to the deformation required. In such an underdetermined condition, we find an optimal solution using a simple least square minimization method. As experimental results, we show well-reconstructed 2D faces even from a small number of feature points.

Flexible models of object classes, based on linear combinations of prototypical images, are capable of matching novel images of the same class and have been shown to be a powerful tool to solve several fundamental vision tasks such as recognition, synthesis and correspondence. The key problem in creating a specific flexible model is the computation of pixelwise correspondence between the prototypes, a task done until now in a semiautomatic way. In this paper we describe an algorithm that automatically bootstraps the correspondence between the prototypes. The algorithm -which can be used for 2D images as well as for 3D models-is shown to synthesize successfully a flexible model of frontal face images and a flexible model of handwritten digits.

A new technique is described for recognizing faces from new viewpoints. From a single 2D image of a face synthetic images from new viewpoints are generated and compared to stored views. A novel 2D image of a face can be computed without knowledge about the 3D structure of the head. The technique draws on prior knowledge of faces based on example images of other faces seen in different poses and on a single generic 3D model of a human head. The example images are used to learn a pose-invariant shape and texture description of a new face. The 3D model is used to solve the correspondence problem between images showing faces in different poses. The performance of the technique is tested on a data set of 200 faces of known orientation for rotations up to 90/spl deg/.

The need to generate new views of a 3D object from a single real image arises in several fields, including graphics and object recognition. While the traditional approach relies on the use of 3D models, simpler techniques are applicable under restricted conditions. The approach exploits image transformations that are specific to the relevant object class, and learnable from example views of other "prototypical" objects of the same class. In this paper, we introduce such a technique by extending the notion of linear class proposed by the authors (1992). For linear object classes, it is shown that linear transformations can be learned exactly from a basis set of 2D prototypical views. We demonstrate the approach on artificial objects and then show preliminary evidence that the technique can effectively "rotate" high-resolution face images from a single 2D view.

A new technique is described for synthesizing images of faces from new viewpoints, when only a single 2D image from a known viewpoint is available. A novel 2D image of a face can be computed without knowledge about the 3D structure of the head. The technique draws on prior knowledge of faces based on example images of other faces seen in different poses and on a single generic 3D model of a human head. The example images are used to learn a pose-invariant shape and texture description of a new face. The 3D model is used to solve the correspondence problem between images showing faces in different poses. Examples of synthetic "rotations" over 24/spl deg/ based on a training set of 100 faces are shown.