Faces in motion contain rich information about a myriad of social cues, including identity. However, is still unknown how much information about a person’s identity is conveyed by different kinds of facial movements and the precise role of facial motion as a cue for identity is still under debate. Open questions include how much information about identity is embedded in facial movements, and how efficiently a perceptual system extracts those cues toward identity. We address these questions by exploring the potential of Neuromorphic Vision Sensors to perform identity recognition through facial dynamics derived from speech and simultaneously validate how contributory facial motion is for human face identity categorization. As far as we know, this is the first work that evaluates the aptitude of the technology offered by neuromorphic cameras for completing the task of facial identity recognition. A one-stream inflated 3D convnet was used to evaluate the ability of different frame-based events encoding strategies in classifying identity through Short-Time Facial Motion Tokens (ST-FAMOTO). A consistent set of evaluations was conducted using two small datasets of individuals engaged in speech tasks, specifically text reading and speech-driven visual passcode. Results reported consistently consolidate the contribution of facial dynamics in the process of identity recognition and the usefulness of neuromorphic sensor devices in acquiring identity discriminatory facial micro-movements. Additionally, it is also proposed the first Neuromorphic Vision Dataset for Speech Induced Facial Dynamics (NVSFD Dataset), created in the scope of this work and available to facilitate future third-party investigation on the topic.

This paper targets deformable face model matching in images using cascaded regression techniques. Recently, the cascaded regression strategies have became rather popular solutions to solve nonlinear objective functions by learning a pipeline sequence of linear regressors. However, despite their success, the standard formulation do not enforce shape consistency through the cascade (mostly because the individual regressors are learnt independently). In this paper we revisit the cascaded regression framework and propose a number of improvements. First we explore the simplicity and compactness of using a linear shape model for such tasks, effectively solving the previous drawback. Then we propose to extend the linear regression module into a nonlinear version, by means of recent Convolutional Neutral Networks (CNNs) techniques, modified to include a weighted shape aware loss function. Moreover, since CNNs often require massive amounts of data to perform well, we took advantage of the shape model probabilistic framework to efficiently bootstrap new data. Our nonlinear cascade regression method is evaluated in several databases (LFPW, LFW, HELEN and 300W), where the results demonstrate the effectiveness of the proposed method.

This paper addresses the problem of localizing facial landmarks with deformable face models using cascaded regression strategies. Recently, these methods have become quite popular, standing out as simple and efficient approaches to optimize nonlinear objective functions. In this paper, we target the well-known Lucas and Kanade (LK) image alignment formulation and introduce the Simultaneous Cascaded Regression (SCR) technique, which can be considered as a cascaded regression extension of the Simultaneous Forwards Additive / Inverse Composition approaches. In contrast to previous LK techniques (Newton based optimizations) which require to recompute Jacobian and Hessians matrices at each iteration, our approach learns (offline) a sequence of descent directions, effectively behaving as averaged steepest descent matrices. Under this revised technique, we propose a part-based generative model (with a linear warp function), that accounts with the underlying shape and appearance structure embedded into regression process itself. Our method is validated on a number of experiments on several datasets (LFPW, LFW, HELEN, 300W), demonstrating a noticeable gain in accuracy/fitting performance when compared with other face alignment solutions.

This paper proposes an extended Constrained Local Model (CLM) formulation for aligning faces using depth information. The CLMs are popular methods that were initially designed to locate facial features in regular intensity images. Briefly, they combine a set of local detectors, one for each landmark, whose locations are regularized by a linear shape model. Fitting a CLM is usually framed as a two step approach: locally search, using the detectors, producing response maps (likelihood maps) followed by a global optimization strategy that jointly maximize all detection scores while enforcing an appropriate shape. Including depth data could be simply posed as adding additional likelihood sources to the main formulation. The paper discusses several likelihood fusion techniques and propose to jointly learn a multi-dimensional correlation filter as a more reliable solution. Moreover, we propose to learn the local detectors, in the Fourier domain, effectively augmenting the training set with virtual samples. Besides improving the detections reliability, this approach is particular important when applied to depth data, as no additional processing is required (such as fill missing information). The performance evaluation shows that our extended approach further increases the fitting performance (accuracy) effectively proving the benefit of using depth data in facial alignment tasks.

Manifold-based domain adaptation algorithms are receiving increasing attention in computer vision to model distribution shifts between source and target domain. In contrast to early works, that mainly explore intermediate subspaces along geodesics, in this work we propose to interpolate subspaces through C1-smooth curves on the Grassmann manifold. The new methodis based on the geometric Casteljau algorithm that is used to generate smooth interpolating polynomial curves on non-euclidean spaces and can be extended to generate polynomial splines that interpolate a given set of data on the Grassmann manifold. To evaluate the usefulness of the proposed interpolating curves on vision related problems, several experiments were conducted. We show the advantage of using smooth subspaces interpolation in multi-source unsupervised domain adaptation problems and in object recognition problems across datasets.

This paper presents a novel Bayesian formulation for aligning faces in unseen images. Our approach revisits the Constrained Local Models (CLM) formulation where an ensemble of local feature detectors are constrained to lie within the subspace spanned by a Point Distribution Model (PDM). Fitting such a model to an image typically involves two main steps: a local search using a detector, obtaining response maps for each landmark (likelihood term) and a global optimization that finds the PDM parameters that jointly maximize all the detections at once. The so-called global optimization can be posed as a Bayesian inference problem, where the posterior distribution of the shape (and pose) parameters can be inferred in a maximum a posteriori (MAP) sense. This work introduces an extended Bayesian global optimization strategy that includes two novel additions: (1) to perform second order updates of the PDM parameters (accounting for their covariance) and (2) to model the underlying dynamics of the shape variations, encoded in the prior term, by using recursive Bayesian estimation. Extensive evaluations were performed against state-of-the-art methods on several standard datasets (IMM, BioID, XM2VTS, LFW and FGNET Talking Face). Results show that the proposed approach significantly increases the fitting performance.

Recently, a particular paradigm [18] in the domain adaptation field has received considerable attention by introducing novel and important insights to the problem. In this case, the source and target domains are represented in the form of subspaces, which are treated as points on the Grassmann manifold. The geodesic curve between them is sampled to obtain intermediate points. Then a classifier is learnt using the projections of the data onto these subspaces. Despite its relevance and popularity, this paradigm [18] contains some limitations. Firstly, in real-world applications, that simple curve (i.e. shortest path) does not provide the necessary flexibility to model the domain shift between the training and testing data sets. Secondly, by using the geodesic curve, we are restricted to only one source domain, which does not allow to take fully advantage of the multiple datasets that are available nowadays. It is then, natural to ask whether this popular concept could be extended to deal with more complex curves and to integrate multi-sources domains. This is a hard problem considering the Riemannian structure of the space, but we propose a mathematically well-founded idea that enables us to solve it. We exploit the geometric insight of rolling maps [30] to compute a spline curve on the Grassmann manifold. The benefits of the proposed idea are demonstrated through several empirical studies on standard datasets. This novel concept allows to explicitly integrate multi-source domains while the previous one [18] uses the mean of all sources. This enables to model better the domain shift and take fully advantage of the training datasets.

The core component of most modern trackers is a discriminative classifier, tasked with distinguishing between the target and the surrounding environment. To cope with natural image changes, this classifier is typically trained with translated and scaled sample patches. Such sets of samples are riddled with redundancies-any overlapping pixels are constrained to be the same. Based on this simple observation, we propose an analytic model for datasets of thousands of translated patches. By showing that the resulting data matrix is circulant, we can diagonalize it with the discrete Fourier transform, reducing both storage and computation by several orders of magnitude. Interestingly, for linear regression our formulation is equivalent to a correlation filter, used by some of the fastest competitive trackers. For kernel regression, however, we derive a new kernelized correlation filter (KCF), that unlike other kernel algorithms has the exact same complexity as its linear counterpart. Building on it, we also propose a fast multi-channel extension of linear correlation filters, via a linear kernel, which we call dual correlation filter (DCF). Both KCF and DCF outperform top-ranking trackers such as Struck or TLD on a 50 videos benchmark, despite running at hundreds of frames-per-second, and being implemented in a few lines of code (Algorithm 1). To encourage further developments, our tracking framework was made open-source.

This work presents a novel non-parametric Bayesian formulation for aligning faces in unseen images. Popular approaches, such as the Constrained Local Models (CLM) or the Active Shape Models (ASM), perform facial alignment through a local search, combining an ensemble of detectors with a global optimization strategy that constraints the facial feature points to be within the subspace spanned by a Point Distribution Model (PDM). The global optimization can be posed as a Bayesian inference problem, looking to maximize the posterior distribution of the PDM parameters in a maximum a posteriori (MAP) sense. Previous approaches rely exclusively on Gaussian inference techniques, i.e. both the likelihood (detectors responses) and the prior (PDM) are Gaussians, resulting in a posterior which is also Gaussian, whereas in this work the posterior distribution is modeled as being non-parametric by a Kernel Density Estimator (KDE). We show that this posterior distribution can be efficiently inferred using Sequential Monte Carlo methods, in particular using a Regularized Particle Filter (RPF). The technique is evaluated in detail on several standard datasets (IMM, BioID, XM2VTS, LFW and FGNET Talking Face) and compared against state-of-the-art CLM methods. We demonstrate that inferring the PDM parameters non-parametrically significantly increase the face alignment performance.

This paper addresses to the problem of aligning images in unseen faces. The Constrained Local Models (CLM) are popular methods that combine a set of local landmark detectors whose locations are constrained to lie in a subspace spanned by a linear shape model. The CLM fitting is usually based on a two step approach: locally search, using the detectors, producing response maps (likelihood) followed by a global optimization strategy that jointly maximize all detections at once. In this paper, we mainly focus on the first stage: improving the detectors reliability. Usually the local landmarks detectors are far from perfect. Most often are designed to be fast, having a small support region and are learnt from limited data. As consequence, they will suffer from detection ambiguities. Here we propose to improve the detectors performance by considering multiple detection per landmark. In particular, we propose a joint learning of the detectors by clustering of their training data. Afterwards, the multiple likelihoods are combined using a nonlinear fusion approach. The performance evaluation shows that our (extended) approach further increases the fitting performance of the CLM formulation, when compared with recent state-of-the-art methods.

In the past few years there has been a growing interest on geometric frameworks to learn supervised classification models on Riemannian manifolds [32, 28]. A popular framework, valid over any Riemannian manifold, was proposed in [32] for binary classification. Once moving from binary to multi-class classification this paradigm is not valid anymore, due to the spread of multiple positive classes on the manifold [28]. It is then natural to ask whether the multi-class paradigm could be extended to operate on a large class of Riemannian manifolds. We propose a mathematically well-founded classification paradigm that allows to extend the work in [32] to multi-class models, taking into account the structure of the space. The idea is to project all the data from the manifold onto an affine tangent space at a particular point. To mitigate the distortion induced by local diffeomorphisms, we introduce for the first time in the computer vision community a well-founded mathematical concept, so-called Rolling map [22, 17]. The novelty in this alternate school of thought is that the manifold will be firstly rolled (without slipping or twisting) as a rigid body, then the given data is unwrapped onto the affine tangent space, where the classification is performed.

This work presents a novel Bayesian formulation for aligning faces in unseen images. Our approach is closely related to Constrained Local Models (CLM) [2] and Active Shape Models (ASM) [6], where an ensemble of local feature detectors are constrained to lie within the subspace spanned by a Point Distribution Model (PDM). Fitting a model to an image typically involves two steps: a local search using a detector, obtaining response maps for each landmark (likelihood term) and a global optimization that finds the PDM parameters that jointly maximize all the detections. The global optimization can be seen as a Bayesian inference problem, where the posterior distribution of the PDM parameters (and pose) can be inferred in a maximum a posteriori (MAP) sense. We present a novel Bayesian global optimization strategy, where the prior is used to encode the dynamic transitions of the PDM parameters. Using recursive Bayesian estimation we model the prior distribution of the data as being Gaussian. The mean and covariance were assumed to be unknown and treated as random variables. The Shape Model: The shape of a PDM is represented by the 2D locations of a mesh s = (x1,y1, . . . ,xv,yv) (v landmarks). Applying PCA on training examples, results in the parametric model s = s0 + Φb + Ψq, where s0 is the mean shape, Φ is the shape subspace matrix (n eigenvectors), b is a vector of shape parameters, q the pose parameters vector and Ψ holds four special eigenvectors that linearly model the 2D pose [4]. Goal: Given a 2v vector of observed positions y, the goal is to find the optimal set of parameters b that maximizes the posterior probability of being aligned. Using an Bayesian approach, the shape parameters are

Background modelling on tensor field has recently been proposed for foreground detection tasks. Taking into account the Riemannian structure of the tensor manifold, recent research has focused on developing parametric methods on the tensor domain e.g. gaussians mixtures (GMM) [7]. However, in some scenarios, simple parametric models do not accurately explain the physical processes. Kernel density estimators (KDE) have been successful to model, on Euclidean sample spaces the nonparametric nature of complex, time varying, and non-static backgrounds [8]. Founded on the mathematically rigorous KDE paradigm on general Riemannian manifolds [15], we define a KDE specifically to operate on the tensor manifold. We present a mathematically-sound framework for nonparametric modeling on tensor field to foreground segmentation. We endow the tensor manifold with two well-founded Riemannian metrics, i.e. Affine-Invariant and Log-Euclidean. Theoretical aspects are defined and the metrics are compared experimentally. Theoretic analysis and experimental results demonstrate the promise/effectiveness of the framework.

The paper proposes a new method to perform foreground detection by means of background modeling using the tensor concept. Sometimes, statistical modelling directly on image values is not enough to achieve a good discrimination. Thus the image may be converted into a more information rich form, such as a tensor field, to yield latent discriminating features. Taking into account the theoretically well-founded differential geometrical properties of the Riemannian manifold where tensors lie, we propose a new approach for foreground detection on tensor field based on data modeling by means of Gaussians mixtures directly on tensor domain. We introduced a online Kmeans approximation of the Expectation Maximization algorithm to estimate the parameters based on an Affine-Invariant metric. This metric has excellent theoretical properties but essentially due to the space curvature the computational burden is high. We propose a novel Kmeans algorithm based on a new family of metrics, called Log-Euclidean, in order to speed up the process, while conserving the same theoretical properties. Contrary to the affine case, we obtain a space with a null curvature. Hence, classical statistical tools usually reserved to vectors are efficiently generalized to tensors in the Log-Euclidean framework. Theoretical aspects are presented and the Affine-Invariant and Log-Euclidean frameworks are compared experimentally. From a practical point of view, results are similar to those of the Affine-Invariant framework but are obtained much faster. Theoretic analysis and experimental results demonstrate the promise and effectiveness of the proposed framework.

This work addresses the fitting of 3D deformable face models from a single view through 2.5D Active Appearance Models (AAM). The main contribution of this paper is the use of 2.5D AAM that combines a 3D metric Point Distribution Model (PDM) and a 2D appearance model whose control points are defined by full perspective projections of the PDM. The advantage is that, assuming a calibrated camera, 3D metric shapes can be retrieved from single view images. Two algorithms and computationally efficient approximations are proposed, the Simultaneous Forwards Additive (SFA) and the Normalization Forwards Additive (NFA), both based on the Lucas Kanade framework. The SFA algorithm is computationally expensive but more accurate. It searches for shape and appearance parameters simultaneously whereas the NFA projects out the appearance from the error image and searches only for the shape parameters. Expanded solutions for the SFA and NFA are also proposed in order to take into account head self occlusions. An extensive performance evaluation is presented. The frequency of convergence for the SFA, NFA and their efficient approximation is evaluated, showing that the 2.5D model can outperform 2D based methods. The Robust extensions to occlusion were tested on a synthetic sequence showing that the model can deal robustly with large head rotation.

A solution for Discriminative Active Appearance Models is proposed. The model consists in a set of descriptors which are covariances of multiple features evaluated over the neighborhood of the landmarks whose locations are governed by a Point Distribution Model (PDM). The covariance matrices are a special set of tensors that lie on a Riemannian manifold, which make it possible to measure the dissimilarity and to update them, imposing the temporal appearance consistency. The discriminative fitting method produce patch response maps found by convolution around the current landmark position. Since the minimum of the responce map isn't always the correct solution due to detection ambiguities, our method finds candidates to solutions based on a mean-shift algorithm, followed by an unsupervised clustering technique used to locate and group the candidates. A mahalanobis based metric is used to select the best solution that is consistent with the PDM. Finally the global PDM optimization step is performed using a weighted least-squares warp update, based on the Lucas Kanade framework. The weights were extracted from a landmark matching score statistics. The effectiveness of the proposed approach was evaluated on unseen data on the challenging Talking Face video sequence, demonstrating the improvement in performance.

A solution for identity and facial expression recognition is proposed using a two stage classifier approach using low dimensional representation of the geometry of the face. Face geometry is extracted from input images using Active Appearance Models (AAM) and low dimensional manifolds were then derived using Laplacian Eigen-Maps (LE) resulting in two types of manifolds, one for model identity and the other for person-specific facial expression. The first stage uses a multiclass Support Vector Machines (SVM) to establish identity across expression changes. The second stage deals with person-specific expression recognition, and is composed by a network of seven Hidden Markov Models (HMM) displaced in parallel, each one specialized on the several facial emotions analysed. The decision was made by the sequence that yielded the highest probability. For evaluation proposes a database was build consisting on 6770 images captured from 4 people exhibiting 7 different emotions. The identity overall recognition rate was 96.8%. Facial expression results are identity dependent, and the most expressive individual achieves 81.2% of overall recognition rate.

A framework for automatic human head pose estimation from single view images is proposed. The 6DOF head pose was estimated using pose from orthography and scaling with iterations (POSIT) where a statistical anthropometric 3D rigid model is used as an approximation of the human head, combined with active appearance models (AAM) for facial features extraction and tracking. The overall performance of the proposed solution was evaluated comparing the results with a ground-truth data obtained by a pose planar approach. The results show that orientations and head location were, on average, found within 2deg or 1 cm error standard deviations respectively.

This work addresses the problem of human head pose estimation from single view images. 3D rigid head pose is estimated combining Active Appearance Models(AAM) with Pose from Orthography and Scaling with ITerations(POSIT). AAM shape landmarks are tracked over time and used in POSIT for pose estimation. A statistical anthropometric 3D model is used as reference. Several experiences were performed comparing our approach with a planar ground truth. It was achieved average standard deviations about 2 degrees in orientation and 1 centimeter in distance.