3D Morphable Face Models (3DMM) have been used in pattern recognition for some time now. They have been applied as a basis for 3D face recognition, as well as in an assistive role for 2D face recognition to perform geometric and photometric normalisation of the input image, or in 2D face recognition system training. The statistical distribution underlying 3DMM is Gaussian. However, the single-Gaussian model seems at odds with reality when we consider different cohorts of data, e.g. Black and Chinese faces. Their means are clearly different. This paper introduces the Gaussian Mixture 3DMM (GM-3DMM) which models the global population as a mixture of Gaussian subpopulations, each with its own mean. The proposed GM-3DMM extends the traditional 3DMM naturally, by adopting a shared covariance structure to mitigate small sample estimation problems associated with data in high dimensional spaces. We construct a GM-3DMM, the training of which involves a multiple cohort dataset, SURREY-JNU, comprising 942 3D face scans of people with mixed backgrounds. Experiments in fitting the GM-3DMM to 2D face images to facilitate their geometric and photometric normalisation for pose and illumination invariant face recognition demonstrate the merits of the proposed mixture of Gaussians 3D face model.

We present a new Cascaded Shape Regression (CSR) architecture, namely Dynamic Attention-Controlled CSR (DAC-CSR), for robust facial landmark detection on unconstrained faces. Our DAC-CSR divides facial landmark detection into three cascaded sub-tasks: face bounding box refinement, general CSR and attention-controlled CSR. The first two stages refine initial face bounding boxes and output intermediate facial landmarks. Then, an online dynamic model selection method is used to choose appropriate domain-specific CSRs for further landmark refinement. The key innovation of our DAC-CSR is the fault-tolerant mechanism, using fuzzy set sample weighting, for attention-controlled domain-specific model training. Moreover, we advocate data augmentation with a simple but effective 2D profile face generator, and context-aware feature extraction for better facial feature representation. Experimental results obtained on challenging datasets demonstrate the merits of our DAC-CSR over the state-of-the-art methods.

We present a fully automatic approach to real-time 3D face reconstruction from monocular in-the-wild videos. With the use of a cascaded-regressor-based face tracking and a 3D morphable face model shape fitting, we obtain a semidense 3D face shape. We further use the texture information from multiple frames to build a holistic 3D face representation from the video footage. Our system is able to capture facial expressions and does not require any person-specific training. We demonstrate the robustness of our approach on the challenging 300 Videos in the Wild (300-VW) dataset. Our real-time fitting framework is available as an open-source library at http://4dface.org.

In this paper, we propose a novel fitting method that uses local image features to fit a 3D Morphable Face Model to 2D images. To overcome the obstacle of optimising a cost function that contains a non-differentiable feature extraction operator, we use a learning-based cascaded regression method that learns the gradient direction from data. The method allows to simultaneously solve for shape and pose parameters. Our method is thoroughly evaluated on Morphable Model generated data and first results on real data are presented. Compared to traditional fitting methods, which use simple raw features like pixel colour or edge maps, local features have been shown to be much more robust against variations in imaging conditions. Our approach is unique in that we are the first to use local features to fit a 3D Morphable Model. Because of the speed of our method, it is applicable for realtime applications. Our cascaded regression framework is available as an open source library at github.com/patrikhuber/superviseddescent.

In this letter, we present a random cascaded-regression copse (R-CR-C) for robust facial landmark detection. Its key innovations include a new parallel cascade structure design, and an adaptive scheme for scale-invariant shape update and local feature extraction. Evaluation on two challenging benchmarks shows the superiority of the proposed algorithm to state-of-the-art methods.

Face recognition has been the focus of attention for the past couple of decades and, as a result, a significant progress has been made in this area. However, the problem of spoofing attacks can challenge face biometric systems in practical applications. In this paper, an effective countermeasure against face spoofing attacks based on a kernel discriminant analysis approach is presented. Its success derives from different innovations. First, it is shown that the recently proposed multiscale dynamic texture descriptor based on binarized statistical image features on three orthogonal planes (MBSIF-TOP) is effective in detecting spoofing attacks, showing promising performance compared with existing alternatives. Next, by combining MBSIF-TOP with a blur-tolerant descriptor, namely, the dynamic multiscale local phase quantization (MLPQ-TOP) representation, the robustness of the spoofing attack detector can be further improved. The fusion of the information provided by MBSIF-TOP and MLPQ-TOP is realized via a kernel fusion approach based on a fast kernel discriminant analysis (KDA) technique. It avoids the costly eigen-analysis computations by solving the KDA problem via spectral regression. The experimental evaluation of the proposed system on different databases demonstrates its advantages in detecting spoofing attacks in various imaging conditions, compared with the existing methods.

A large amount of training data is usually crucial for successful supervised learning. However, the task of providing training samples is often time-consuming, involving a considerable amount of tedious manual work. In addition, the amount of training data available is often limited. As an alternative, in this paper, we discuss how best to augment the available data for the application of automatic facial landmark detection. We propose the use of a 3D morphable face model to generate synthesized faces for a regression-based detector training. Benefiting from the large synthetic training data, the learned detector is shown to exhibit a better capability to detect the landmarks of a face with pose variations. Furthermore, the synthesized training data set provides accurate and consistent landmarks automatically as compared to the landmarks annotated manually, especially for occluded facial parts. The synthetic data and real data are from different domains; hence the detector trained using only synthesized faces does not generalize well to real faces. To deal with this problem, we propose a cascaded collaborative regression algorithm, which generates a cascaded shape updater that has the ability to overcome the difficulties caused by pose variations, as well as achieving better accuracy when applied to real faces. The training is based on a mix of synthetic and real image data with the mixing controlled by a dynamic mixture weighting schedule. Initially, the training uses heavily the synthetic data, as this can model the gross variations between the various poses. As the training proceeds, progressively more of the natural images are incorporated, as these can model finer detail. To improve the performance of the proposed algorithm further, we designed a dynamic multi-scale local feature extraction method, which captures more informative local features for detector training. An extensive evaluation on both controlled and uncontrolled face data sets demonstrates the merit of the proposed algorithm.

Large pose and illumination variations are very challenging for face recognition. The 3D Morphable Model (3DMM) approach is one of the effective methods for pose and illumination invariant face recognition. However, it is very difficult for the 3DMM to recover the illumination of the 2D input image because the ratio of the albedo and illumination contributions in a pixel intensity is ambiguous. Unlike the traditional idea of separating the albedo and illumination contributions using a 3DMM, we propose a novel Albedo Based 3D Morphable Model (AB3DMM), which removes the illumination component from the images using illumination normalisation in a preprocessing step. A comparative study of different illumination normalisation methods for this step is conducted on PIE and Multi-PIE databases. The results show that overall performance of our method outperforms state-of-the-art methods.

We address the problem of anomaly detection in machine perception. The concept of domain anomaly is introduced as distinct from the conventional notion of anomaly used in the literature. We propose a unified framework for anomaly detection which exposes the multifaceted nature of anomalies and suggest effective mechanisms for identifying and distinguishing each facet as instruments for domain anomaly detection. The framework draws on the Bayesian probabilistic reasoning apparatus which clearly defines concepts such as outlier, noise, distribution drift, novelty detection (object, object primitive), rare events, and unexpected events. Based on these concepts we provide a taxonomy of domain anomaly events. One of the mechanisms helping to pinpoint the nature of anomaly is based on detecting incongruence between contextual and noncontextual sensor(y) data interpretation. The proposed methodology has wide applicability. It underpins in a unified way the anomaly detection applications found in the literature. To illustrate some of its distinguishing features, in here the domain anomaly detection methodology is applied to the problem of anomaly detection for a video annotation system.

The 3D Morphable Model (3DMM) is currently receiving considerable attention for \nhuman face analysis. Most existing work focuses on fitting a 3DMM to high resolution \nimages. However, in many applications, fitting a 3DMM to low-resolution images \nis also important. In this paper, we propose a Resolution-Aware 3DMM (RA- \n3DMM), which consists of 3 different resolution 3DMMs: High-Resolution 3DMM \n(HR- 3DMM), Medium-Resolution 3DMM (MR-3DMM) and Low-Resolution 3DMM \n(LR-3DMM). RA-3DMM can automatically select the best model to fit the input images \nof different resolutions. The multi-resolution model was evaluated in experiments \nconducted on PIE and XM2VTS databases. The experimental results verified that HR- \n3DMM achieves the best performance for input image of high resolution, and MR- \n3DMM and LR-3DMM worked best for medium and low resolution input images, respectively. \nA model selection strategy incorporated in the RA-3DMM is proposed based \non these results. The RA-3DMM model has been applied to pose correction of face images \nranging from high to low resolution. The face verification results obtained with \nthe pose-corrected images show considerable performance improvement over the result \nwithout pose correction in all resolutions

Lip region deformation during speech contains biometric information and is termed visual speech. This biometric information can be interpreted as being genetic or behavioral depending on whether static or dynamic features are extracted. In this paper, we use a texture descriptor called local ordinal contrast pattern (LOCP) with a dynamic texture representation called three orthogonal planes to represent both the appearance and dynamics features observed in visual speech. This feature representation, when used in standard speaker verification engines, is shown to improve the performance of the lip-biometric trait compared to the state-of-the-art. The best baseline state-of-the-art performance was a half total error rate (HTER) of 13.35% for the XM2VTS database. We obtained HTER of less than 1%. The resilience of the LOCP texture descriptor to random image noise is also investigated. Finally, the effect of the amount of video information on speaker verification performance suggests that with the proposed approach, speaker identity can be verified with a much shorter biometric trait record than the length normally required for voice-based biometrics. In summary, the performance obtained is remarkable and suggests that there is enough discriminative information in the mouth-region to enable its use as a primary biometric trait.

The lip-region can be interpreted as either a genetic or behavioural biometric trait depending on whether static or dynamic information is used. In this paper, we use a texture descriptor called Local Ordinal Contrast Pattern (LOCP) in conjunction with a novel spatiotemporal sampling method called Windowed Three Orthogonal Planes (WTOP) to represent both appearance and dynamics features ob served in visual speech. This representation, with standard speaker verification engines, is shown to improve the performance of the lip biometric trait compared to the state-of-the-art. The improvement obtained suggests that there is enough discriminative information in the mouth-region to enable its use as a primary biometric as opposed to a "soft" biometric trait.

Bags-of-visual-Words (BoW) and Spatio-Temporal Shapes (STS) are two very popular approaches for action recognition from video. The former (BoW) is an un-structured global representation of videos which is built using a large set of local features. The latter (STS) uses a single feature located on a region of interest (where the actor is) in the video. Despite the popularity of these methods, no comparison between them has been done. Also, given that BoW and STS differ intrinsically in terms of context inclusion and globality/locality of operation, an appropriate evaluation framework has to be designed carefully. This paper compares these two approaches using four different datasets with varied degree of space-time specificity of the actions and varied relevance of the contextual background. We use the same local feature extraction method and the same classifier for both approaches. Further to BoW and STS, we also evaluated novel variations of BoW constrained in time or space. We observe that the STS approach leads to better results in all datasets whose background is of little relevance to action classification.

A key prerequisite of automatic video indexing and summarisation is the description of events and actions. In the context of many sports, the motion of the ball and agents plays an essential role in describing events. However, the only existing solution for the tennis event recognition problem in the literature is the work in which relies on a set of heuristic rules such as proximity between ball and players or court lines to classify ball event candidates. We present hidden Markov models (HMMs) paradigm to automatically learn to identify events from ball trajectories and demonstrate that its ability to capture the dynamics of the ball movement lead to a much higher performance.

Anomaly detection has received much attention within the literature as a means of determining, in an unsupervised manner, whether a learning domain has changed in a fundamental way. This may require continuous adaptive learning to be abandoned and a new learning process initiated in the new domain. A related problem is that of anomaly rectification; the adaptation of the existing learning mechanism to the change of domain. As a concrete instantiation of this notion, the current paper investigates a novel lattice-based HMM induction strategy for arbitrary court-game environments. We test (in real and simulated domains) the ability of the method to adapt to a change of rule structures going from tennis singles to tennis doubles. Our long term aim is to build a generic system for transferring game-rule inferences.

In this paper we propose a new framework for super-resolving facial images under arbitrary pose. While example-based super-resolution methods have demonstrated impressive results for face super-resolution under given pose and imaging conditions, they have limitations dealing with different poses and illuminations. Due to these limitations their application to face super-resolution in generalized situations is either impractical or sub-optimal. The proposed framework utilizes a 3D morphable face model in order to address the problem of face super-resolution under arbitrary pose. This framework does not assume any pre-defined pose for the subject thus it can be readily applied to any pose. The main contribution of this work is defining a framework in which a 3D morphable model can be used in conjunction with any of the example-based super-resolution methods. Experimental results prove the potential power of this method in face superresolution and its application to face recognition.

In this paper, we propose a multilayered data association scheme with graph-theoretic formulation for tracking multiple objects that undergo switching dynamics in clutter. The proposed scheme takes as input object candidates detected in each frame. At the object candidate level, "tracklets'' are "grown'' from sets of candidates that have high probabilities of containing only true positives. At the tracklet level, a directed and weighted graph is constructed, where each node is a tracklet, and the edge weight between two nodes is defined according to the "compatibility'' of the two tracklets. The association problem is then formulated as an all-pairs shortest path (APSP) problem in this graph. Finally, at the path level, by analyzing the APSPs, all object trajectories are identified, and track initiation and track termination are automatically dealt with. By exploiting a special topological property of the graph, we have also developed a more efficient APSP algorithm than the general-purpose ones. The proposed data association scheme is applied to tennis sequences to track tennis balls. Experiments show that it works well on sequences where other data association methods perform poorly or fail completely.

In previous work, we developed a novel data association algorithm with graph-theoretic formulation, and used it to track a tennis ball in broadcast tennis video. However, the track initiation/termination was not automatic, and it could not deal with situations in which more than one ball appeared in the scene. In this paper, we extend our previous work to track multiple tennis balls fully automatically. The algorithm presented in this paper requires the set of all-pairs shortest paths in a directed and edge-weighted graph. We also propose an efficient All-Pairs Shortest Path algorithm by exploiting a special topological property of the graph. Comparative experiments show that the proposed data association algorithm performs well both in terms of efficiency and tracking accuracy. 1

It is well recognised that data association is critically important for object tracking. However, in the presence of successive misdetections, a large number of false candidates and an unknown number of abrupt model switchings that happen unpredictably, the data association problem can be very difficult. We tackle these difficulties by using a layered data association scheme. At the object level, trajectories are "grown" from sets of object candidates that have high probabilities of containing only true positives; by this means the otherwise combinatorial complexity is significantly reduced. Dijkstra’s shortest path algorithm is then used to perform data association at the trajectory level. The algorithm is applied to low-quality tennis video sequences to track a tennis ball. Experiments show that the algorithm is robust to abrupt model switchings, and performs well in heavily cluttered environments.

Several tennis ball tracking algorithms have been reported in the literature. However, most of them use high quality video and multiple cameras, and the emphasis has been on coordinating the cameras, or visualising the tracking results. In this paper, we propose a tennis ball tracking algorithm for low quality off-air video recorded with a single camera. Multiple visual cues are exploited for tennis candidate detection. A particle filter with improved sampling efficiency is used to track the tennis candidates. Experimental results show that our algorithm is robust and has a tracking accuracy that is sufficiently high for automatic annotation of tennis matches. 1

A new, fast, statistically robust, exhaustive, translational image-matching technique is presented: fast robust correlation. Existing methods are either slow or non-robust, or rely on optimization. Fast robust correlation works by expressing a robust matching surface as a series of correlations. Speed is obtained by computing correlations in the frequency domain. Computational cost is analyzed and the method is shown to be fast. Speed is comparable to conventional correlation and, for large images, thousands of times faster than direct robust matching. Three experiments demonstrate the advantage of the technique over standard correlation.

A new method of translational image registration is presented: 'orientation correlation'. The method is fast, exhaustive, statistically robust, and illumination invariant. No existing method has all of these properties. A modification that is particularly well suited to matching images of differing modalities, 'squared orientation correlation', is also given. Orientation correlation works by correlating 'orientation images'. Each pixel in a orientation image is a complex number that represents the orientation of intensity gradient. This representation is invariant to illumination change. Angles of gradient orientation are matched. Andrews robust kernel function is applied to angle differences. Through the use of correlation the method is exhaustive. The method is fast as the correlation can be computed using Fast Fourier Transforms.

The use of video and audio features for automated annotation of audio-visual data is becoming widespread. A major limitation of many of the current methods is that the stored indexing features are too low-level-they relate directly to properties of the data. We apply a further stage of processing that associates the feature measurements with real-world objects or events. The outputs, which we call "cues", denote the probability of the object being present in the scene. An additional advantage of this approach is that the cues from different types of features are presented in a homogeneous way.

The performance of shot detection methods in video sequences can be improved by the use of a threshold that adapts itself to the sequence statistics. In this paper we present some new techniques for adapting the threshold. We then compare the new techniques with an existing one, leading to an improved shot detection method.

We introduce an approach for texture-based annotation and retrieval. Given the outputs of 12 Gabor filters, we derive a texture feature space where the sensitivity of the features to illumination changes is attenuated by a suitable normalisation. We then annotate images by defining and selecting codes representing the quantised levels of the texture features appearing in each image. The annotations are stored in a hash table for retrieval efficiency. Ranking schemes are proposed to order the images retrieved at query time. In particular, we use results from psychological studies on the human perception of similarity to formulate a similarity measure. The choice of quantisation of the texture feature space can influence the accuracy of the retrieval. We compared several quantisation schemes in retrieval experiments involving texture images. We found that a uniform quantisation and a quantisation heuristically taking the variance of the texture features into account lead to the best retrieval performance.

The accuracy of gradient-based optical flow algorithms depends on the ability to measure intensity gradients accurately. We show how the temporal gradient can be compromised by temporal aliasing arising from motion and how appropriate post-sampling spatial filtering improves the situation. We also demonstrate a benefit of using higher-order gradient estimators.

When using a gradient-based method to determine the optical flow field for an image sequence, it is generally appreciated that some spatial pre-filtering of the images is usually needed, particularly for large motion values. However the characteristics of the filter are not generally given. In this paper we analyse the motion measurement from the point of view of sampling theory. We show how an aliasing problem can arise due to undersampling in the temporal domain, and how this problem can be alleviated by appropriate post-sampling spatial filtering. We also demonstrate a connection between this filtering and the methods used to generate the spatial and temporal image intensity gradients. 1 Introduction In all forms of optic flow measurement, if accurate measurements are required, we find that some form of initial estimate of the likely range of motion values is needed; thus there is an inherent bootstrap problem. For block-based methods, either optimisation techniques are u...

In previous work, a probabilistic method for matching of 2-D geometric features has been described. For this method, a noise model is required for the compatibility coefficients, which has hitherto been specified heuristically. In this paper, the authors show how an improved noise model can be derived, using as a basis the noise model of the feature measurements. The new noise model is shown to provide better performance than the heuristic one. Using a non-iterative version of the matching algorithm, a good solution was found using the new model, which was not the case for the old one. When an iterative version was used, both models generated good solutions, but the new model resulted in faster convergence. Furthermore, no training is needed to generate covariances for the new model. A means of explicitly modelling small scaling errors is also given. 1 Introduction Many authors have described methods for feature labelling based on probabilistic considerations, in which features extrac...

In this paper, we develop the theory of probabilistic relaxation for matching features extracted from 2D images, derive as limiting cases the various heuristic formulae used by researchers in matching problems, and state the conditions under which they apply, We successfully apply our theory to the problem of matching and recognizing aerial road network images based on road network models and to the problem of edge matching in a stereo pair. For this purpose, each line network is represented by an attributed relational graph where each node is a straight line segment characterized by certain attributes and related with every other node via a set of binary relations.>

The authors present the theory of probabilistic relaxation for matching symbolic structures, derive as limiting cases the various heuristic formulas used by researchers in matching problems, and state the conditions under which they apply. They successfully apply the theory to the problem of matching and recognizing aerial road network images based on road network models and to the problem of edge matching in a stereo pair. For this purpose, each line network is represented by an attributed relational graph where each node is a straight line segment characterized by certain attributes and related with every other node via a set of binary relations.>