Defect classification and detection have been explored using convolutional neural networks (CNNs). Normally, a large set of training images containing defects and the associated annotation data are required by these approaches. However, such a large set of images is usually difficult to collect because defects are rare and annotation is time-consuming and expensive. To address these issues, we propose to use a multitask deep one-class CNN for defect classification. Compared with supervised classification methods, this CNN does not require abnormal images and annotated data for training. Specifically, we build a stacked encoder–decoder autoencoder for learning feature representation from normal images. The encoder is used as a feature extractor based on the hard sharing scheme of multitask learning. A one-class classification (OCC) objective learned as a hypersphere using minimum volume estimation is appended to it. Together the encoder and the OCC objective lead to a deep one-class classifier. To train both the autoencoder and one-class classifier end-to-end, a multitask loss function is built. Given an unknown sample, the distance between its feature representation and the center of the hypersphere is used as the anomaly score. Furthermore, defect detection is implemented using a moving-window scanning method on top of the deep one-class classifier. The proposed approach achieves better performance than its counterparts trained using a two-stage method. For defect detection, our approach achieves results almost as good as the supervised method even without using any annotated data. We attribute the promising results to the advantages of multitask learning.Note to Practitioners—Building and evaluating vision-based nondestructive testing (NDT) techniques require many examples of abnormal images, which may not be easy to acquire. This article describes a method that does not require abnormal images for training a convolutional neural network (CNN) in order to perform one-class defect classification (outlier detection). We also applied the method to defect detection with promising results. We include results of experiments demonstrating that better performance can be obtained using our method compared to a set of baselines. Although the proposed method does not use abnormal images for training, it still produces results that are almost as good as the supervised learning-based CNN approaches. This study provides a solution to the challenge encountered by the industrial inspection community when enough abnormal samples are hard to obtain.

Abstract We introduce a new architecture called ChoiceNet where each layer of the network is highly connected with skip connections and channelwise concatenations. This enables the network to alleviate the problem of vanishing gradients, reduces the number of parameters without sacrificing performance and encourages feature reuse. We evaluate our proposed architecture on three independent tasks: classification, segmentation and facial landmark localisation. For this, we use benchmark datasets such as ImageNet, CIFAR-10, CIFAR-100, SVHN CamVid and 300W.

In the aerospace manufacturing industry, nondestructive evaluation (NDE) of components plays an important role. Porosities and other defects usually occur in the welds of these components. If such defects end up in the aircraft, the fatigue life of components is lessened, which may cause disastrous accidents. At present, those welds are manually evaluated by human inspectors via reviewing X-ray images. To reduce the workload of inspectors, we have developed an automatic inspection system for identifying defects in linear thin welds. For an X-ray image, this system starts with localizing the central line of the weld using a random forest (RF) regressor. A region surrounding the line is then investigated using an RF classifier in order to detect defects. After extensive experiments, the results demonstrate that the weld can be precisely localized from X-ray images, and the defect detection module can find 80% of defects that have been identified by human inspectors (i.e., true positives), while fewer than 1.6 false positives per image are returned. It is suggested that the system may be beneficial to human inspectors by reducing their workload. In addition, our system produces encouraging results on the publicly available weld X-ray image data set and a magnetic tile image data set.Note to Practitioners—This work was motivated by the challenge of inspecting aerospace components, which is almost entirely done manually at present. Rather than replacing human inspectors, this work aims at reducing their workload by providing them with an initial inspection result for each component. Especially, the proposed system is able to first localize the Region of Interest (RoI) from an X-ray image of a component and then identify potential defects contained in the RoI. To the best of our knowledge, few existing studies perform defect detection on raw component images. Normally, researchers manually cropped an RoI from these images. The output of our system is the pixelwise location information on potential defects. Our results demonstrate that the proposed system is able to accurately localize the weld and identify 80% of defects contained in abnormal weld images with very few false positives. Given that large weld images ($2304\times1920$pixels) were processed, our system located the weld in 6.6± 1.3 s/image and fulfilled defect detection on each localized weld region in 0.8± 0.1 s. The proposed system was also tested with the publicly available X-ray weld image data set: GDXray and a magnetic tile image data set. Although only a small number of training images were available, promising results were obtained. This suggests that our system is suitable for both X-ray weld images and other images though more work is needed to reduce false positives.

The non-destructive testing (NDT) of components is very important to the aerospace industry. Welds in these components may contain porosities and other defects. These reduce the fatigue life of components and may result in catastrophic accidents if they end up in the aircraft. Currently such welds are inspected by humans studying radiographs of the welds. We describe an automatic system for detecting defects in welds, with the aim of creating a triage system to reduce the workload on human inspectors. Given an X-ray image of the aerospace weld, the system locates the weld line, then analyses the region around the line to identify abnormalities. Our results show that the weld can be precisely extracted from X-ray images and the defect detection operation can identify 83% of defects with fewer than 3 false positives per image, and thus may be useful for prompting human inspectors to reduce their workload.

Osteoarthritis (OA) is the most common form of arthritis, affecting millions of people around the world. Since no cure has been discovered and considering the financial impact on health systems, any attempt to understand more of this disease could reveal new insights that would help develop new therapies. Lateral knee radiographs are often ignored both by clinicians and the research community when trying to diagnose OA or other diseases that affect the knee joint. Our goal is to show that this view has a considerable potential. We present a fully automated method based on a Random Forest Regression Voting Constrained Local Model (RFCLM) to discriminate radiographs of people that have developed OA from people who have not. The experiments involved models built on different combinations of the four shapes (patella, tibia, medial and lateral femoral condyles) of the knee joint. We show that automated analysis of the lateral view achieves classification performance comparable if not better than similar techniques applied to the frontal view.

Dental radiography plays an important role in clinical diagnosis, treatment and surgery. In recent years, efforts have been made on developing computerized dental X-ray image analysis systems for clinical usages. A novel framework for objective evaluation of automatic dental radiography analysis algorithms has been established under the auspices of the IEEE International Symposium on Biomedical Imaging 2015 Bitewing Radiography Caries Detection Challenge and Cephalometric X-ray Image Analysis Challenge. In this article, we present the datasets, methods and results of the challenge and lay down the principles for future uses of this benchmark. The main contributions of the challenge include the creation of the dental anatomy data repository of bitewing radiographs, the creation of the anatomical abnormality classification data repository of cephalometric radiographs, and the definition of objective quantitative evaluation for comparison and ranking of the algorithms. With this benchmark, seven automatic methods for analysing cephalometric X-ray image and two automatic methods for detecting bitewing radiography caries have been compared, and detailed quantitative evaluation results are presented in this paper. Based on the quantitative evaluation results, we believe automatic dental radiography analysis is still a challenging and unsolved problem. The datasets and the evaluation software will be made available to the research community, further encouraging future developments in this field. (http://www-o.ntust.edu.tw/~cweiwang/ISBI2015/).

A widely used approach for locating points on deformable objects in images is to generate feature response images for each point, and then to fit a shape model to these response images. We demonstrate that Random Forest regression-voting can be used to generate high quality response images quickly. Rather than using a generative or a discriminative model to evaluate each pixel, a regressor is used to cast votes for the optimal position of each point. We show that this leads to fast and accurate shape model matching when applied in the Constrained Local Model framework. We evaluate the technique in detail, and compare it with a range of commonly used alternatives across application areas: the annotation of the joints of the hands in radiographs and the detection of feature points in facial images. We show that our approach outperforms alternative techniques, achieving what we believe to be the most accurate results yet published for hand joint annotation and state-of-the-art performance for facial feature point detection.

Extraction of bone contours from radiographs plays an important role in disease diagnosis, preoperative planning, and treatment analysis. We present a fully automatic method to accurately segment the proximal femur in anteroposterior pelvic radiographs. A number of candidate positions are produced by a global search with a detector. Each is then refined using a statistical shape model together with local detectors for each model point. Both global and local models use Random Forest regression to vote for the optimal positions, leading to robust and accurate results. The performance of the system is evaluated using a set of 839 images of mixed quality. We show that the local search significantly outperforms a range of alternative matching techniques, and that the fully automated system is able to achieve a mean point-to-curve error of less than 0.9 mm for 99% of all 839 images. To the best of our knowledge, this is the most accurate automatic method for segmenting the proximal femur in radiographs yet reported.

Groupwise nonrigid image registration is a powerful tool to automatically establish correspondences across sets of images. Such correspondences are widely used for constructing statistical models of shape and appearance. As existing techniques usually treat registration as an optimization problem, a good initialization is required. Although the standard initialization-affine transformation-generally works well, it is often inadequate when registering images of complex structures. In this paper we present a more sophisticated method that uses the sparse matches of a parts+geometry model as the initialization. We show that both the model and its matches can be automatically obtained, and that the matches are able to effectively initialize a groupwise nonrigid registration algorithm, leading to accurate dense correspondences. We also show that the dense mesh models constructed during the groupwise registration process can be used to accurately annotate new images. We demonstrate the efficacy of the approach on three datasets of increasing difficulty, and report on a detailed quantitative evaluation of its performance.

Active Appearance Models (AAMs) are widely used to fit shape models to new images.Recently it has been demonstrated that non-linear regression methods and sequences of AAMs can significantly improve performance over the original linear formulation.In this paper we focus on the ability of a model trained on one dataset to generalise to other sets with different conditions.In particular we compare two non-linear, discriminative regression strategies for predicting shape updates, a boosting approach and variants of Random Forest regression.We investigate the use of these regression methods within a sequential model fitting framework, where each stage in the sequence consists of a shape model and a corresponding regression model.The performance of the framework is assessed by both testing on unseen data taken from within the training databases, as well as by investigating the more difficult task of generalising to unrelated datasets.We present results that show that (a) the generalisation performance of the Random Forest is superior to that of the linear or boosted regression procedure and that (b) using a simple feature selection procedure, the Random Forest can be made to be as efficient as the boosting procedure without significant reduction in accuracy.

This paper tackles the problem of accurately matching a 3D deformable face model to sequences of images in challenging real-world scenarios with large amounts of head movement, occlusion, and difficult lighting conditions. A baseline system involves searching with a set of view-dependent local patches to locate image features, and using these to update the face shape model parameters. We show here two modifications that lead to improvements in performance and can be applied in other similar systems. These are: explicitly searching for occluding boundaries, which prevents the model from rotating rather than changing shape; and a simple method for weighting the relative importance of each located match for model fit. We demonstrate the improvements on both standard test sets and on a series of difficult in-car driver videos, showing more accurate matching and fewer search failures. © 2010. The copyright of this document resides with its authors.

The Active Appearance Model (AAM) provides an efficient method for localizing objects that vary in both shape and texture, and uses a linear regressor to predict updates to model parameters based on current image residuals. This study investigates using additive (or 'boosted') predictors, both linear and non-linear, as a substitute for the linear predictor in order to improve accuracy and efficiency. We demonstrate: (a) a method for training additive models that is several times faster than the standard approach without sacrificing accuracy; (b) that linear additive models can serve as an effective substitute for linear regression; (c) that linear models are as effective as non-linear models when close to the true solution. Based on these observations, we compare a 'hybrid' AAM to the standard AAM for both the XM2VTS and BioID datasets, including cross-dataset evaluations. © 2010. The copyright of this document resides with its authors.

The `displacement expert' has recently proven popular for rapid tracking applications. In this paper, we note that experts are typically constrained only to produce approximately correct parameter updates at training locations. However, we show that incorporating constraints on the gradient of the displacement field within the learning framework results in an expert with better convergence and fewer local minima. We demonstrate this proposal for facial feature localization in static images and object tracking over a sequence.

Groupwise image registration algorithms seek to establish dense correspondences between sets of images. Typically, they involve iteratively improving the registration between each image and an evolving mean. A variety of methods have been proposed, which differ in their choice of objective function, representation of deformation field, and optimization methods. Given the complexity of the task, the final accuracy is significantly affected by the choices made for each component. Here, we present a groupwise registration algorithm which can take advantage of the statistics of both the image intensities and the range of shapes across the group to achieve accurate matching. By testing on large sets of images (in both 2D and 3D), we explore the effects of using different image representations and different statistical shape constraints. We demonstrate that careful choice of such representations can lead to significant improvements in overall performance.

Statistical shape models are powerful tools for image interpretation and shape analysis. A simple, yet effective, way of building such models is to capture the statistics of sampled point coordinates over a training set of example shapes. However, a major drawback of this approach is the need to establish a correspondence across the training set. In 2-D, a correspondence is often defined using a set of manually placed 'landmarks' and linear interpolation to sample the shape in between. Such annotation is, however, time-consuming and subjective, particularly when extended to 3-D. In this paper, we show that it is possible to establish a dense correspondence across the whole training set automatically by treating correspondence as an optimization problem. The objective function we use for the optimization is based on the minimum description length principle, which we argue is a criterion that leads to models with good compactness, specificity, and generalization ability. We manipulate correspondence by reparameterizing each training shape. We describe an explicit representation of reparameterization for surfaces in 3-D that makes it impossible to generate an illegal (i.e., not one-to-one) correspondence. We also describe several large-scale optimization strategies for model building, and perform a detailed analysis of each approach. Finally, we derive quantitative measures of model quality, allowing meaningful comparison between models built using different methods. Results are given for several different training sets of 3-D shapes, which show that the minimum description length models perform significantly better than other approaches.

We describe a method for modelling and locating deformable objects using a combination of global and local shape models. An object is represented as a set of patches together with a geometric model of their relative positions. The geometry is modelled with a global pose and linear shape model, together with a Markov Random Field (MRF) model of local displacements from the global model. Matching to a new image involves an alternating scheme in which an MRF inference technique selects the best candidates for each point, which are then used to update the parameters of the global pose and shape model. A cascade of increasingly complex models is used to achieve robust matching to new images. We explore the effect of model parameters on system performance and show that the proposed method achieves better accuracy than other widely used methods on standard datasets. © 2009. The copyright of this document resides with its authors.

We present an efficient method of fitting a set of local feature models to an image within the popular Active Shape Model (ASM) framework [3]. We compare two different types of non-linear boosted feature models trained using GentleBoost [9]. The first type is a conventional feature detector classifier, which learns a discrimination function between the appearance of a feature and the local neighbourhood. The second local model type is a boosted regression predictor which learns the relationship between the local neighbourhood appearance and the displacement from the true feature location. At run-time the second regression model is much more efficient as only the current feature patch needs to be processed. We show that within the local iterative search of the ASM the local feature regression provides improved localisation on two publicly available human face test sets as well as increasing the search speed by a factor of eight. 1

We describe an approach for automated analysis of deformable objects which extracts structure information from groups of images containing different examples of the object with a particular application to human imaging. The proposed analysis framework simultaneously segments and registers a set of images, incrementally constructing a model of the composition of the object. By fitting an appropriate intensity distribution model to the image we obtain a soft segmentation which allows us to explicitly model the construction of each pixel from constituent image segments, rather than its expected intensity. This effectively decouples the model from the effects of the imaging system and varying statistics in different examples. When estimating the optimal deformation field for each example, the original image is compared to a reconstruction, generated using the composition model and its intensity distribution parameters for each segment (i.e. an estimate of how the model would appear given the imaging conditions for that image). In the paper we describe the algorithm in detail and show results of applying it to two sets of medical images of different anatomies taken with different imaging modalities. We present quantitative results demonstrating that the proposed algorithm is more powerful than current state of the art methods at extracting structural information such as spatial correspondences across groups of images with varying statistics. 1

Active appearance models (AAMs) are widely used to fit statistical models of shape and appearance to images, and have applications in segmentation, tracking, and classification of structures. A limitation of AAMs is that they are not robust to a large set of gross outliers. Using a robust kernel can help, but there are potential problems in determining the correct kernel scaling parameters. We describe a method of learning two sets of scaling parameters during AAM training: a coarse and a fine scale set. Our algorithm initially applies the coarse scale and then uses a form of deterministic annealing to reduce to the fine outlier rejection scaling as the AAM converges. The algorithm was assessed on two large datasets consisting of a set of faces, and a medical dataset of images of the spine. A significant improvement in accuracy and robustness was observed in cases which were difficult for a standard AAM. 1

This paper deals with the problem of objective evaluation of dynamic, multi-sensor image fusion. For this purpose an established static image fusion evaluation framework, based on gradient information preservation between the inputs and the fused image, is extended to deal with additional scene and object motion information present in multi-sensor sequences. In particular formulations for dynamic, multi-sensor information preservation models are proposed to provide space-time localised fusion performance estimates. Perceptual importance distribution models are derived to accommodate temporal data and provide a natural generalisation of localised performance estimates into both global and continuous dynamic fusion performance scores. The proposed system is described in detail and shown to exhibit better evaluation accuracy, robustness and sensitivity when compared to existing dynamic fusion metrics on an evaluation of several established image fusion algorithms applied to multi- sensor sequences from an array of dynamic fusion scenarios.

The present work aims to model the correspondence between facial motion and speech. The face and sound are modelled separately, with phonemes being the link between both. We propose a sequential model and evaluate its suitability for the generation of the facial animation from a sequence of phonemes, which we obtain from speech. We evaluate the results both by computing the error between generated sequences and real video, as well as with a rigorous double-blind test with human subjects. Experiments show that our model compares favourably to other existing methods and that the sequences generated are comparable to real video sequences.

Active Appearance Models [5] are widely used to match statistical models of shape and appearance to new images rapidly. They work by finding model parameters which minimise the sum of squares of residual differences between model and target image. Their efficiency is achieved by pre-computing the Jacobian describing how the residuals are expected to change as the parameters vary. This leads to a method of predicting the position of the minima based on a single measurement of the residuals (though in practise the algorithm is iterated to refine the estimate). However, the estimate of the Jacobian from the training set will only be an approximation for any given target image, and may be a poor one if the target image is significantly different from the training images. This paper describes a simple method of updating a representation of the Jacobian as the search progresses. This allows us to tune the AAM to the current example. Though useful for matching to a single image, it is particularly powerful when tracking objects through sequences, as it gives a method of tuning the AAM as the search progresses. We demonstrate the power of the technique on a variety of datasets. 1

We present an efficient and robust model matching method which uses a joint shape and texture appearance model to generate a set of region template detectors. The model is fitted to an unseen image in an iterative manner by generating templates using the joint model and the current parameter estimates, correlating the templates with the target image to generate response images and optimising the shape parameters so as to maximise the sum of responses. The appearance model is similar to that used in the AAM [1]. However in our approach the appearance model is used to generate likely feature templates, instead of trying to approximate the image pixels directly. We show that when applied to human faces, our Constrained Local Model (CLM) algorithm is more robust and more accurate than the original AAM search method, which relies on the image reconstruction error to update the model parameters. We demonstrate improved localisation accuracy on two publicly available face data sets and improved tracking on a challenging set of in-car face sequences. 1

The considerable number of image fusion algorithms available today vary widely in terms of fusion performance and robust fusion assessment tools have become a target of considerable research. Based on a variety of localised or global evaluations of image statistics and structure between the inputs and the fused image, available objective fusion evaluation metrics use a number of different information representation and information loss models. This paper explores the definition of an optimal information representation for the evaluation of multisensor image fusion and how it can be defined based on the actual application of fused information. Extensive evaluations of a considerable data set of subjectively annotated fused images with a variety of information representation approaches implemented using three different global fusion evaluation frameworks are presented. The results show that if used with correct information representation models global, statistical approaches can yield significantly better fusion evaluation performance than existing methods.

A plethora of image fusion algorithms have been proposed recently, yet what are optimal fusion parameters that should be used for any multi-sensor dataset cannot be defined a priori. They could be learned by evaluating all available fusion strategies on large, representative datasets, but this is not practical and provides no guarantee that fusion performance will remain optimal should real input conditions differ from sample data. This paper proposes and examines the viability of a powerful framework for objectively optimal image fusion that explicitly optimises fusion performance for any set of input conditions. The idea is to integrate proven concepts used in objective image fusion evaluation metrics to optimally adapt the fusion process to the input conditions. Specific focus is on fusion for display, which has a broad appeal in a wide range of fusion applications as only metrics shown to be subjectively relevant are considered. The results show that the proposed framework achieves a considerable improvement in both the level and robustness of fusion performance for a wide array of multi-sensor images.

We describe an algorithm for obtaining correspondences across a group of images of deformable objects. The approach is to construct a statistical model of appearance which can encode the training images as compactly as possible (a Minimum Description Length framework). Correspondences are defined by piece-wise linear interpolation between a set of control points defined on each image. Given such points a model can be constructed, which can approximate every image in the set. The description length encodes the cost of the model, the parameters and most importantly, the residuals not explained by the model. By modifying the positions of the control points we can optimise the description length, leading to good correspondence. We describe the algorithm in detail and give examples of its application to MR brain images and to faces. We also describe experiments which use a recently-introduced specificity measure to evaluate the performance of different components of the algorithm. 1

We describe a novel shape constraint technique which is incorporated into a multi-stage algorithm to automatically locate features on the human face. The method is coarse-to-fine. First a face detector is applied to find the approximate scale and location of the face in the image. Then individual feature detectors are applied and combined using a novel algorithm known as Pairwise Reinforcement of Feature Responses (PRFR). The points predicted by this method are then refined using a version of the Active Appearance Model (AAM) search, which is tuned to edge and corner features. The final output of the three stage algorithm is shown to give much better results than any other combination of methods. The method outperforms previous published results on the BIOID test set [11]. 1

-

Recently a fast and efficient face detection method has been devised [11], which relies on the AdaBoost algorithm and a set of Haar Wavelet like features. A natural extension of this approach is to use the same technique to locate individual features within the face region. However, we find that there is insufficient local structure to reliably locate each feature in every image, and thus local models can give many false positive responses. We demonstrate that the performance of such feature detectors can be significantly improved by using global shape constraints. We describe an algorithm capable of accurately and reliably detecting facial features and present quantitative results on both high and low resolution image sets.

Statistical models of shape and appearance are powerful tools for interpreting medical and other images. However there can remain problems with under-trained models being too constrained. We have combined a global model with a sequence of partially overlapping sub-models, in a manner that exploits all the statistical information, whilst mitigating the under-training problem. Instead of applying one global model, we use a global model to apply iteratively-updated soft constraints on a sequence of sub-models. These sub-models may also partially overlap, and thus previously fit sub-models can also impose soft constraints on the next iteration. The algorithm has been applied to dual x-ray absorptiometry scans of the spine in order to automate vertebral morphometry measurements, using overlapping triplets of vertebrae as the sub-models, together with a global model of the entire spine. Combining a global model in this way with a sequence of sub-models gives substantially better results than using the former alone.

We consider the problem of learning how a person's face behaves in a long video sequence, with the aim of synthesising convincing sequences demonstrating the same behaviours. We describe a novel approach to segment a sequence into short sections, each representing a distinct action (or a part of an action). These sections are grouped and a model of the variability of the action learnt. A variable length Markov model is trained on the sequence of such actions to learn the temporal relationships. The result is a system that can generate realistic sequences of an individual face.

The Active Appearance Model (AAM) algorithm has proved to be a suc-cessful method for matching statistical models of appearance to new images. Since the original algorithm was described there have been a variety of sug-gested modifications to the basic algorithm, each typically claiming to be in some way superior. We review these algorithms and report the results of ex-periments comparing their performance. We also investigate the effects of different methods of estimating the update matrix used in the algorithm. We find that careful choice of the latter has at least as much effect as the choice of updating technique. 1

Statistical models of shape and appearance have been successfully used in face modeling, tracking and synthesis. In this paper we describe ex-periments using appearance models for face verification. We compare a variety of different algorithms on a standard face database (XM2VTS). We demonstrate that simple methods of correction for head pose and face expression can significantly improve results. 1

The process of aging causes significant alterations in the facial appearance of individuals. When compared with other sources of variation in face images, appearance variation due to aging displays some unique characteristics. Changes in facial appearance due to aging can even affect discriminatory facial features, resulting in deterioration of the ability of humans and machines to identify aged individuals. We describe how the effects of aging on facial appearance can be explained using learned age transformations and present experimental results to show that reasonably accurate estimates of age can be made for unseen images. We also show that we can improve our results by taking into account the fact that different individuals age in different ways and by considering the effect of lifestyle. Our proposed framework can be used for simulating aging effects on new face images in order to predict how an individual might look like in the future or how he/she used to look in the past. The methodology presented has also been used for designing a face recognition system, robust to aging variation. In this context, the perceived age of the subjects in the training and test images is normalized before the training and classification procedure so that aging variation is eliminated. Experimental results demonstrate that, when age normalization is used, the performance of our face recognition system can be improved.

The multimodal nature of speech is often ignored in human-computer interaction, but lip deformations and other body motion, such as those of the head, convey additional information. We integrate speech cues from many sources and this improves intelligibility, especially when the acoustic signal is degraded. The paper shows how this additional, often complementary, visual speech information can be used for speech recognition. Three methods for parameterizing lip image sequences for recognition using hidden Markov models are compared. Two of these are top-down approaches that fit a model of the inner and outer lip contours and derive lipreading features from a principal component analysis of shape or shape and appearance, respectively. The third, bottom-up, method uses a nonlinear scale-space analysis to form features directly from the pixel intensity. All methods are compared on a multitalker visual speech recognition task of isolated letters.

We describe a method for automatically building statistical shape models from a training set of example boundaries/surfaces. These models show considerable promise as a basis for segmenting and interpreting images. One of the drawbacks of the approach is, however, the need to establish a set of dense correspondences between all members of a set of training shapes. Often this is achieved by locating a set of "landmarks" manually on each training image, which is time consuming and subjective in two dimensions and almost impossible in three dimensions. We describe how shape models can be built automatically by posing the correspondence problem as one of finding the parameterization for each shape in the training set. We select the set of parameterizations that build the "best" model. We define "best" as that which minimizes the description length of the training set, arguing that this leads to models with good compactness, specificity and generalization ability. We show how a set of shape parameterizations can be represented and manipulated in order to build a minimum description length model. Results are given for several different training sets of two-dimensional boundaries, showing that the proposed method constructs better models than other approaches including manual landmarking-the current gold standard. We also show that the method can be extended straightforwardly to three dimensions.

When attempting to code faces for modelling or recognition, estimates of dimensions are typically obtained from an ensemble. These tend to be significantly sub-optimal. Each face contains both predictable and non-predictable qualities

Statistical shape models have been used widely as a basis for segmenting and interpreting images. A major drawback of the approach is the need to establish a set of dense correspondences across a training set of segmented shapes. By posing the problem as one of minimising the description length of the model, we develop an efficient method that automatically defines correspondences across a set of shapes. Results are given for several different training sets of shapes, showing that the automatic method constructs significantly better models than those built by hand- the current gold standard. 1

We show how a novel, non-linear representation of edge structure can be used to improve the performance of model matching algorithms and object verification/recognition tasks. Rather than represent the image structure using intensity values or gradients, we use a measure which indicates the orientation of structures at each pixel, together with an indication of how reliable the orientation estimate is. Orientations in flat, noisy regions tend to be penalised whereas those near strong edges are favoured. We demonstrate that this representation leads to more accurate and reliable matching between models and new images, and leads to better recognition/verification of faces in an access control task.

. We demonstrate a novel method of interpreting images using an Active Appearance Model (AAM). An AAM contains a statistical model of the shape and grey-level appearance of the object of interest which can generalise to almost any valid example. During a training phase we learn the relationship between model parameter displacements and the residual errors induced between a training image and a synthesised model example. To match to an image we measure the current residuals and use the model to predict changes to the current parameters, leading to a better fit. A good overall match is obtained in a few iterations, even from poor starting estimates. We describe the technique in detail and give results of quantitative performance tests. We anticipate that the AAM algorithm will be an important method for locating deformable objects in many applications. 1 Introduction Model-based approaches to the interpretation of images of variable objects are now attracting considerable interest [6][8...

This paper describes building models which represent the appearance of an object (in particular, a face) as seen from two or more dierent viewpoints simultaneously. A small number of 2D linear statistical models are suÆcient to capture the shape and appearance of a face from a wide range of viewpoints. Given multiple images of the same face we can learn a coupled model describing the relationship between the frontal appearance and the prole of a face. This relationship can be used to predict new views of a face seen from one view. Such a coupled model can be used to constrain search algorithms which seek to locate a face in multiple views simultaneously, leading to more robust results than searching each view independently. 1

When attempting to code faces for modeling or recognition, estimates of di-mensions are typically obtained from an ensemble. These tend to be signif-icantly sub-optimal. Firstly, ensembles are rarely balanced with regard to identity and expression. This can be overcome by dividing the ensemble by type of variation and rotating sub-spaces relative to one another. Secondly, each face contains both predictable and non-predictable qualities; only the predictable aspects are useful for defining coding systems for other faces. Variance-based methods of defining codes (PCA) will provide eigenvectors which are themselves potential faces. Predictable aspects will induce eigen-vectors with comparable levels of spatial redundancy to the ensemble. We show that this gives relatively short and consistent codes, and allows fast and accurate fitting of codes to faces. 1

We demonstrate that a small number of 2D statistical models are sufficient to capture the shape and appearance of a face from any viewpoint (full profile to front-to-parallel). Each model is linear and can be matched rapidly to new images using the active appearance model algorithm. We show how such a set of models can be used to estimate head pose, to track faces through large angles of head rotation and to synthesize faces from unseen viewpoints.

In order to build a statistical model of facial appearance we require a set of images, each with a consistent set of landmarks. We address the problem of automatically placing a set of landmarks to define the correspondences across an image set. We can estimate correspondences between any pair of images by locating salient points on one and finding their corresponding position in the second. However, we wish to determine a globally consistent set of correspondences across all the images. We present an iterative scheme in which these pairwise correspondences are used to determine a global correspondence across the entire set. We show results on several training sets, and demonstrate that an appearance model trained on the correspondences is of higher quality than one built from hand-marked images.

In recent years there has been much progress in the development of facial recognition systems. The FERET series of tests reported the black box performance of several such systems working on stored face images. Much less effort has been spent in studying the behaviour of systems under realistic conditions of use. We describe and analyse the result of a trial of a door access control system based on a model-based approach. The trial consisted of 10 registered users making over 200 accesses during a 2 week period. We describe the internal failure modes and the performance characteristics of the system, identify inter- and intra-person dependencies and make recommendations for future work.

Statistical models of the shape and appearance of image structures can be matched to new images using both the Active Shape Model [7] algorithm and the Active Appearance Model algorithm [2]. The former searches along profiles about the current model point positions to update the current estimate of the shape of the object. The latter samples the image data under the current instance and uses the difference between model and sample to update the appearance model parameters. In this paper we compare and contrast the two algorithms, giving the results of experiments testing their performance on two data sets, one of faces, the other of structures in MR brain sections. We find that the ASM is faster and achieves more accurate feature point location than the AAM, but the AAM gives a better match to the texture. 1 Introduction Interpretting images containing objects whose appearance can vary is difficult. A powerful approach has been to use deformable models, which can represent the variati...

Facial variation divides into a number of functional subspaces, and ensemble-specific variation. An improved method of measuring these is presented, within the space defined by an Appearance Model. Initial estimates of the subspaces (lighting, pose, identity and expression) are obtained by Principal Components Analysis on appropriate groups of faces. An expectation-maximization algorithm is applied to image codings to maximise the probability of coding across these non-orthogonal subspaces. Ensemble specific variation is then removed by measuring the spatial predictability of the eigenvectors excluding those which are less predictable than the ensemble. These procedures significantly enhance identity recognition for a disjoint test set.

We address the problem of automatically placing landmarks across an image sequence to define correspondences between frames. The marked up sequence is then used to build a statistical model of the appearance of the object within the sequence. We locate the most salient object features from within the first frame and attempt to track them throughout the sequence. Salient features are those which have a low probability of being mis-classified as any other feature, and are therefore more likely to be robustly tracked throughout the sequence. The method automatically builds statistical models of the objects shape and the salient features appearance as the sequence is tracked. These models are used in subsequent frames to further improve the probability of finding accurate matches. Results are shown for several face image sequences. The quality of the model is comparable with that generated from hand labelled images.

Facial variation divides into a number of functional subspaces. An improved method of measuring these was designed within the space defined by an Appearance Model. Initial estimates of the subspaces (lighting, pose, identity, expression) were obtained by Principal Components Analysis on appropriate groups of faces. An iterative algorithm was applied to image codings to maximise the probability of coding across these non-orthogonal subspaces before obtaining the projection on each sub-space and recalculating the spaces. This procedure enhances identity recognition, reduces overall sub-space variance and produces Principal Components with greater span and less contamination.

We present a quantitative evaluation of an algorithm for model-based face recognition. The algorithm actively learns how individual faces vary through video sequences, providing on-line suppression of confounding factors such as expression, lighting and pose. By actively decoupling sources of image variation, the algorithm provides a framework in which identity evidence can be integrated over a sequence. We demonstrate that face recognition can be considerably improved by the analysis of video sequences. The method presented is widely applicable in many multi-class interpretation problems.

This paper presents advances in the construction and use of Active Appearance Models (AAMs) for image interpretation. AAMs are photo-realistic generative models of object appearance that can be used to rapidly locate deformable objects in images. We extend the AAM method to include coloured texture and present an enhanced search algorithm with the ability to locate partially occluded objects. Previously, AAMs have been limited by the need for good manual initialisation. In this paper, we describe a hierarchical search algorithm that overcomes this drawback. The extended AAM method provides a complete, unified scheme for model based image interpretation. We demonstrate the application of the scheme to the task of locating faces in images.

The process of ageing causes significant alterations in the facial appearance of individuals. When compared with other sources of variation in face images (e.g. variation due to changes in pose and expression), appearance variation due to ageing displays some unique characteristics. For example ageing variation is specific to a given individual, it occurs slowly and it is affected significantly by other factors, such as the health, gender and the lifestyle of the individual. In this paper we describe how the effects of ageing on facial appearance can be explained using a parameterized statistical model. We present experimental results to show that reasonably accurate estimates of age can be made for unseen images. We also show that we can improve our results significantly by taking into account the fact that different individuals age in different ways and by considering the effect of lifestyle. We also demonstrate how the proposed framework can be used for simulating ageing effects on new face images, in order to predict how an individual might look like in the future, or how he/she used to look in the past. Experimental and visual results on simulation of age effects are presented.

An Active Appearance Model (AAM) allows complex models of shape and appearance to be matched to new images rapidly. An AAM contains a statistical model of the shape and grey-level appearance of an object of interest The associated search algorithm exploits the locally linear relationship between model parameter displacements and the residual errors between model instance and image. This relationship can be learnt during a training phase. To match to an image we measure the current residuals and use the model to predict changes to the current parameters. The algorithm converges in a few iterations. In this paper we describe variations of the basic algorithm aimed at improving the speed and robustness of search. These include subsampling and using image residuals to drive the shape rather than full appearance model. We show examples of search and give the results of experiments comparing the performance of the different algorithms. 1 Introduction Model based methods are now widely used ...

We present a method for locating salient object features. Salient features are those which have a low probability of being mis-classified with any other feature, and are therefore more easily found in a similar image containing an example of the object. The local image structure can be described by vectors extracted using a standard `feature extractor' at a range of scales. We train statistical models for each feature, using vectors taken from a number of training examples. The feature models can then be used to find the probability of misclassifying a feature with all other features. Low probabilities indicate a salient feature. Results are presented showing that salient features can be relocated more reliably than features chosen using previous methods, including hand picked features.

Face images are difficult to interpret because they are highly variable. Sources of variability include individual appearance, 3D pose, facial expression, and lighting. We describe a compact parametrized model of facial appearance which takes into account all these sources of variability. The model represents both shape and gray-level appearance, and is created by performing a statistical analysis over a training set of face images. A robust multiresolution search algorithm is used to fit the model to faces in new images. This allows the main facial features to be located, and a set of shape, and gray-level appearance parameters to be recovered. A good approximation to a given face can be reconstructed using less than 100 of these parameters. This representation can be used for tasks such as image coding, person identification, 3D pose recovery, gender recognition, and expression recognition. Experimental results are presented for a database of 690 face images obtained under widely varying conditions of 3D pose, lighting, and facial expression. The system performs well on all the tasks listed above.

Active Shape Models (ASMs) provide an efficient means of locating objects in images. By statistically modelling the shape variations in a class of objects they can rapidly and robustly fit to new examples. However, if an ASM does not represent all the shape variation exhibited by the object, the model may not be able to locate new examples accurately. This paper describes two complementary approaches to allowing additional freedom to the points which compromise the model, enabling them to fit to the image data more accurately. We present results for synthetic and real images and discuss how the methods can be used in an interactive 'bootstrap ' training scheme where problems with over-constrained models are particularly important.

We describe a method of locating hypotheses for the positions of faces in an image. We use statistical feature detectors to locate candidates for features, then use a statistical model of the shape and orientation of the features to test combinations of such features to find the most plausible. The best sets can be used as the initial position of an Active Shape Model, which can then accurately locate the full face.

Model based approaches to the interpretation of face images have proved very successful. We have previously described statistically based models of face shape and grey-level appearance and shown how they can be used to perform various coding and interpretation tasks (Lanitis et al., 1995). In the paper we describe improved methods of modelling, which couple shape and grey-level information more directly than our existing methods, isolate the changes in appearance due to different sources of variability (person, expression, pose, lighting) and deal with nonlinear shape variation. We show that the new methods are better suited to interpretation and tracking tasks.

The absolute orientation problem involves finding the Euclidean transformation which minimises the sum of the squared errors between two pointsets. In the standard form of the problem a confidence may be attached to each of the errors via a set of positive scalar weights. In this paper we consider a generalisation of the standard problem in which the components of the error vectors are coupled via a set of weight matrices. We show how problems of this type arise and derive two distinct forms of the problem. We present a closed-form solution to the first form of the 3-D problem and iterative solutions to the second form of the 2-D problem and both forms of the 3-D problem.

Objects of the same class often exhibit variation in shape. This shape variation has previously been modelled by means of point distribution models (PDMs) in which there is a linear relationship between a set of shape parameters and the positions of points on the shape. Here we present a new form of PDM, which uses a multilayer perceptron (MLP) to carry out nonlinear principal component analysis. We demonstrate that MLP-PDMs can model the shape variability in classes of object for which the linear model fails. We describe the use of MLP-PDMs in image search and present quantitative results for a practical application (face recognition), demonstrating the ability to locate image structures accurately starting from a very poor initial approximation to their pose and shape.

Hand gesture recognition from video images is of considerable interest as a means of providing simple and intuitive man-machine interfaces. Possible applications range from replacing the mouse as a pointing device to virtual reality and communication with the deaf. We describe an approach to tracking a hand in an image sequence and recognising, in each video frame, which of five gestures it has adopted. A statistically based Point Distribution Model (PDM) is used to provide a compact parametrised description of the shape of the hand for any of the gestures or the transitions between them. The values of the resulting shape parameters are used in a statistical classifier to identify gestures. The model can be used as a deformable template to track a hand through a video sequence but this proves unreliable. We describe how a set of models, one for each of the five gestures, can be used for tracking with the appropriate model selected automatically. We show that this results in reliable tracking and gesture recognition for two 'unseen' video sequences in which all the gestures are used.

We describe how to build statistically-based flexible models of the 3D structure of variable objects, given a training set of uncalibrated images. We assume that for each example object there are two labelled images taken from different viewpoints. From each image pair a 3D structure can be reconstructed, up to either an affine or projective transformation, depending on which camera model is used. The reconstructions are aligned by choosing the transformations which minimise the distances between matched points across the training set. A statistical analysis results in an estimate of the mean structure of the training examples and a compact parameterised model of the variability in shape across the training set. Experiments have been performed using pinhole and affine camera models. Results are presented for both synthetic data and real images.

This paper presents a new method for modelling and locating objects in images for applications such as Printed Circuit Board (PCB) inspection. Objects of interest are assumed to exhibit little variation in size or shape from one example to the next, but may vary considerably in grey-level appearance. Simple correlation based approaches perform poorly on such examples. We demonstrate how a statistical model based approach combined with a multi-resolution search can accurately locate objects and reliably distinguish between good and bad components. We describe a 'bootstrap' approach to training and a method of automatically refining the final model to improve its performance. We demonstrate the method on PCB inspection, showing the approach is robust enough for use in a real production environment.

Objects of the same class sometimes exhibit variation in shape. This shape variation has previously been modelled by means of point distribution models (PDMs) in which there is a linear relationship between a set of shape parameters and the positions of points on the shape. A polynomial regression generalization of PDMs, which succeeds in capturing certain forms of non-linear shape variability, has also been described. Here we present a new form of PDM, which uses a multi-layer perceptron to carry out non-linear principal component analysis. We compare the performance of the new model with that of the existing models on two classes of variable shape: one exhibits bending, and the other exhibits complete rotation. The linear PDM fails on both classes of shape; the polynomial regression model succeeds for the first class of shapes but fails for the second; the new multi-layer perceptron model performs well for both classes of shape. The new model is the most general formulation for PDMs which has been proposed to date. 1.

Face images are difficult to interpret because they are highly variable. Sources of variability include individual appearance, 3D pose, facial expression and lighting. We describe a compact parametrised model of facial appearance which takes into account all these sources of variability. The model represents both shape and grey-level appearance and is created by performing a statistical analysis over a training set of face images. A robust multi-resolution search algorithm is used to fit the model to faces in new images. This allows the main facial features to be located and a set of shape and grey-level appearance parameters to be recovered. A good approximation to a given face can be reconstructed using less than 100 of these parameters. This representation can be used for tasks such as image coding, person identification, pose recovery, gender recognition and expression recognition. The system performs well on all the tasks listed above.>

This paper describes a method of combining two approaches to modelling flexible objects. Modal Analysis using Finite Element Methods (FEMs) generates a set of vibrational modes for a single shape. Point Distribution Models (PDMs) generate a statistical model of shape and shape variation from a set of example shapes. A new approach is described which generates vibrational modes when few example shapes are available and changes smoothly to using more statistical modes of variation when a large data set is presented. Results are given for both synthetic and real examples. Experiments using the models for image search show that the combined version performs better than either the PDM or FEM models alone. 1

We describe a new approach to modelling the appearance of structures in grey-level images. We assume that both the shape and grey-levels of the structures can vary from one image to another, and that a number of example images are available for training. A 2-D image can be thought of as a surface in 3 dimensions, with the third dimension being the grey-level intensity at each image point. We can represent the shape of this surface by planting landmark points across it. By examining the way such collections of points vary across different examples we can build a statistical model of the shape, which can be used to generate new examples, and to locate examples of the modelled structure in new images. We show examples of these composite appearance models and demonstrate their use in image interpretation.

We describe a multi-resolution technique for locating for variable structures in images. This is an extension of work on Active Shape Models (ASMs)- statistical models which iteratively deform to match image data. An ASM consists of a shape model controlling a set of landmark points, together with a statistical model of the grey-levels expected around each landmark. Both the shape model and the grey-level models are trained on sets of labelled example images. In order to apply a coarse-to-fine search strategy it is necessary to train a set of grey-level models for each landmark, one for every level of a multi-resolution image pyramid. During image search the model is started on the coarsest resolution image. As the search progresses it moves to finer and finer resolutions until no further improvement can be made. We describe an automatic technique for deciding when to:ss has converged. We demonstrate the ntitative experiments which show a sigi speed and quality of fit compared to previous methods.

Methods for automatic image interpretation based on the use of deformable template models have proved very successful. Whatever deformable template scheme is used, one of the basic requirements is a method for assessing the likelihood that a particular model instance is the correct interpretation of a given image. We describe a Bayesian `fitness' measure which combines the likelihood of the model shape with the evidential support in a principled way. Image search is carried out by minimising the fitness measure using multi-scale quasi-Newtonian optimisation. We have previously compared the perform# ance of different fitness measures. Here we give results for the new method and show that, by making optimal use of the image evidence, it achieves more accurate interpreta# tion than the best of the methods we have previously tested. Introduction Flexible template models have been used successfully for many applications of automatic image interpretation[1,2,3,4]. The template embodies ÁÂÄ...

We describe the use of flexible models for representing the shape and grey-level appearance of human faces. These models are controlled by a small number of parameters which can be used to code the overall appearance of a face for image compression and classification purposes. The model parameters control both inter-class and within-class variation. Discriminant analysis techniques are employed to enhance the effect of those parameters affecting inter-class variation, which are useful for classification. We have performed experiments on face coding and reconstruction and automatic face identification. Good recognition rates are obtained even when significant variation in lighting, expression and 3D viewpoint, is allowed. Human faces display significant variation in appearance due to changes in expression, 3D orientation, lighting conditions, hairstyles and so on. A successful automatic face identification system should be capable of suppressing the effect of these factors allowing any face image to be rendered expression-free with standardised 3D orientation and lighting. We describe how the variations in shape and grey-level appearance in face images can be modelled, and present results for a fully automatic face identification system which tolerates changes in expression, viewpoint and lighting.

We have previously described how to model shape variability by means of point distribution models (TDMs,) in which there is a linear relationship between a set of shape parameters and the positions of points on the shape. This linear formulation can fail for shapes which articulate or bend. ' we show examples of such failure for both real and synthetic classes of shape. A new, more general formulation for PDMs, based on polynomial regression, is presented. The resulting Polynomial Regression PDMs (PRPDMsj perform well on the data for which the linear method failed. 1.

We describe methods for using flexible models to locate structures in images. We have previously described statistical models of shape and shape variability which can be used for this purpose (active shape models). In this paper we show how statistical models of grey-level appearance can be incorporated, leading to improved reliability and accuracy. We describe experiments designed to: 1) test how well an active shape model can locate an object in a new image; 2) to assess the effects on performance of varying the model parameters; and 3) to compare the results using grey-level models with those using a search for strongest edges. The results demonstrate that the addition of grey-level models leads to considerable improvement over earlier schemes.

We describe a multiresolution approach to image search using flexible shape models. This is an extension of work on active shape models (ASMs)-statistical models which iteratively deform to match image data. An ASM consists of a shape model controlling a set of landmark points, together with a statistical model of the grey-levels expected around each landmark. Both the shape model and the grey-level models are trained on sets of labelled example images. In order to apply a coarse-to-fine search strategy it is necessary to train a set of grey-level models for each landmark, one for every level of a multiresolution image pyramid. We demonstrate the approach and give results of quantitative experiments which show a significant increase in both speed, robustness and quality of fit compared to previous methods.

We describe methods for locating known structures in images. We have previously described statistical models of shape and shape variability which can be used for this purpose (Active Shape Models). In this paper we show how statistical models of grey-level appearance can be incorporated, leading to improved reliability and accuracy. We describe experiments designed to (i) test how well an ASM can locate an object in a new image, (ii) to assess the effects on performance of varying the model parameters, and (iii) to compare the results using grey-level models with those using a search for strongest edges. The results demonstrate that the addition of grey-level models leads to considerable improvement over earlier schemes.

A technique for classifying variable objects using flexible template models is described.: is recognised as the input, plant seeds, handprinted characters and human faces; quantitative results are presented. 1

We address the problem of finding a consistent interpretation of an image when a number of object features may be detected independently, but unreliably, and their relative positions are known to be constrained. Our method treats feature detection and the application of spatial constraints as co-operating processes. We show that a Point Distribution Model can be used to model constraints on the configuration of features and that the model parameters define a convenient configuration space in which a region representing the set of currently feasible configurations can be maintained. We also introduce the idea of dealing with spatially compact groups of feature hypotheses rather than single hypotheses. We describe two reasoning strategies for dealing with hypothesis groups and feasible configuration regions. These lead to an efficient and exact solution to combinatorially explosive image interpretation problems. We demonstrate the feasibility of the approach by showing results for a system designed to interpret lateral skull radiographs.

The authors describe a technique for building compact models of the shape and appearance of flexible objects seen in 2-D images. The models are derived from the statistics of sets of labeled images of example objects. Each model consists of a flexible shape template, describing how important points of the object can vary, and a statistical model of the expected grey levels in regions around each model point. Such models have proved useful in a wide variety of applications. A description is given on how the models can be used in local image search, and examples of their application are included.>