This paper presents a novel Bayesian formulation for aligning faces in unseen images. Our approach revisits the Constrained Local Models (CLM) formulation where an ensemble of local feature detectors are constrained to lie within the subspace spanned by a Point Distribution Model (PDM). Fitting such a model to an image typically involves two main steps: a local search using a detector, obtaining response maps for each landmark (likelihood term) and a global optimization that finds the PDM parameters that jointly maximize all the detections at once. The so-called global optimization can be posed as a Bayesian inference problem, where the posterior distribution of the shape (and pose) parameters can be inferred in a maximum a posteriori (MAP) sense. This work introduces an extended Bayesian global optimization strategy that includes two novel additions: (1) to perform second order updates of the PDM parameters (accounting for their covariance) and (2) to model the underlying dynamics of the shape variations, encoded in the prior term, by using recursive Bayesian estimation. Extensive evaluations were performed against state-of-the-art methods on several standard datasets (IMM, BioID, XM2VTS, LFW and FGNET Talking Face). Results show that the proposed approach significantly increases the fitting performance.

Recently, a particular paradigm [18] in the domain adaptation field has received considerable attention by introducing novel and important insights to the problem. In this case, the source and target domains are represented in the form of subspaces, which are treated as points on the Grassmann manifold. The geodesic curve between them is sampled to obtain intermediate points. Then a classifier is learnt using the projections of the data onto these subspaces. Despite its relevance and popularity, this paradigm [18] contains some limitations. Firstly, in real-world applications, that simple curve (i.e. shortest path) does not provide the necessary flexibility to model the domain shift between the training and testing data sets. Secondly, by using the geodesic curve, we are restricted to only one source domain, which does not allow to take fully advantage of the multiple datasets that are available nowadays. It is then, natural to ask whether this popular concept could be extended to deal with more complex curves and to integrate multi-sources domains. This is a hard problem considering the Riemannian structure of the space, but we propose a mathematically well-founded idea that enables us to solve it. We exploit the geometric insight of rolling maps [30] to compute a spline curve on the Grassmann manifold. The benefits of the proposed idea are demonstrated through several empirical studies on standard datasets. This novel concept allows to explicitly integrate multi-source domains while the previous one [18] uses the mean of all sources. This enables to model better the domain shift and take fully advantage of the training datasets.

Semantic segmentation and object detection are nowadays dominated by methods operating on regions obtained as a result of a bottom-up grouping process (segmentation) but use feature extractors developed for recognition on fixed-form (e.g. rectangular) patches, with full images as a special case. This is most likely suboptimal. In this paper we focus on feature extraction and description over free-form regions and study the relationship with their fixed-form counterparts. Our main contributions are novel pooling techniques that capture the second-order statistics of local descriptors inside such free-form regions. We introduce second-order generalizations of average and max-pooling that together with appropriate non-linearities, derived from the mathematical structure of their embedding space, lead to state-of-the-art recognition performance in semantic segmentation experiments without any type of local feature coding. In contrast, we show that codebook-based local feature coding is more important when feature extraction is constrained to operate over regions that include both foreground and large portions of the background, as typical in image classification settings, whereas for high-accuracy localization setups, second-order pooling over free-form regions produces results superior to those of the winning systems in the contemporary semantic segmentation challenges, with models that are much faster in both training and testing.

The core component of most modern trackers is a discriminative classifier, tasked with distinguishing between the target and the surrounding environment. To cope with natural image changes, this classifier is typically trained with translated and scaled sample patches. Such sets of samples are riddled with redundancies-any overlapping pixels are constrained to be the same. Based on this simple observation, we propose an analytic model for datasets of thousands of translated patches. By showing that the resulting data matrix is circulant, we can diagonalize it with the discrete Fourier transform, reducing both storage and computation by several orders of magnitude. Interestingly, for linear regression our formulation is equivalent to a correlation filter, used by some of the fastest competitive trackers. For kernel regression, however, we derive a new kernelized correlation filter (KCF), that unlike other kernel algorithms has the exact same complexity as its linear counterpart. Building on it, we also propose a fast multi-channel extension of linear correlation filters, via a linear kernel, which we call dual correlation filter (DCF). Both KCF and DCF outperform top-ranking trackers such as Struck or TLD on a 50 videos benchmark, despite running at hundreds of frames-per-second, and being implemented in a few lines of code (Algorithm 1). To encourage further developments, our tracking framework was made open-source.

This work presents a novel non-parametric Bayesian formulation for aligning faces in unseen images. Popular approaches, such as the Constrained Local Models (CLM) or the Active Shape Models (ASM), perform facial alignment through a local search, combining an ensemble of detectors with a global optimization strategy that constraints the facial feature points to be within the subspace spanned by a Point Distribution Model (PDM). The global optimization can be posed as a Bayesian inference problem, looking to maximize the posterior distribution of the PDM parameters in a maximum a posteriori (MAP) sense. Previous approaches rely exclusively on Gaussian inference techniques, i.e. both the likelihood (detectors responses) and the prior (PDM) are Gaussians, resulting in a posterior which is also Gaussian, whereas in this work the posterior distribution is modeled as being non-parametric by a Kernel Density Estimator (KDE). We show that this posterior distribution can be efficiently inferred using Sequential Monte Carlo methods, in particular using a Regularized Particle Filter (RPF). The technique is evaluated in detail on several standard datasets (IMM, BioID, XM2VTS, LFW and FGNET Talking Face) and compared against state-of-the-art CLM methods. We demonstrate that inferring the PDM parameters non-parametrically significantly increase the face alignment performance.

This paper addresses to the problem of aligning images in unseen faces. The Constrained Local Models (CLM) are popular methods that combine a set of local landmark detectors whose locations are constrained to lie in a subspace spanned by a linear shape model. The CLM fitting is usually based on a two step approach: locally search, using the detectors, producing response maps (likelihood) followed by a global optimization strategy that jointly maximize all detections at once. In this paper, we mainly focus on the first stage: improving the detectors reliability. Usually the local landmarks detectors are far from perfect. Most often are designed to be fast, having a small support region and are learnt from limited data. As consequence, they will suffer from detection ambiguities. Here we propose to improve the detectors performance by considering multiple detection per landmark. In particular, we propose a joint learning of the detectors by clustering of their training data. Afterwards, the multiple likelihoods are combined using a nonlinear fusion approach. The performance evaluation shows that our (extended) approach further increases the fitting performance of the CLM formulation, when compared with recent state-of-the-art methods.

In the past few years there has been a growing interest on geometric frameworks to learn supervised classification models on Riemannian manifolds [32, 28]. A popular framework, valid over any Riemannian manifold, was proposed in [32] for binary classification. Once moving from binary to multi-class classification this paradigm is not valid anymore, due to the spread of multiple positive classes on the manifold [28]. It is then natural to ask whether the multi-class paradigm could be extended to operate on a large class of Riemannian manifolds. We propose a mathematically well-founded classification paradigm that allows to extend the work in [32] to multi-class models, taking into account the structure of the space. The idea is to project all the data from the manifold onto an affine tangent space at a particular point. To mitigate the distortion induced by local diffeomorphisms, we introduce for the first time in the computer vision community a well-founded mathematical concept, so-called Rolling map [22, 17]. The novelty in this alternate school of thought is that the manifold will be firstly rolled (without slipping or twisting) as a rigid body, then the given data is unwrapped onto the affine tangent space, where the classification is performed.

Competitive sliding window detectors require vast training sets. Since a pool of natural images provides a nearly endless supply of negative samples, in the form of patches at different scales and locations, training with all the available data is considered impractical. A staple of current approaches is hard negative mining, a method of selecting relevant samples, which is nevertheless expensive. Given that samples at slightly different locations have overlapping support, there seems to be an enormous amount of duplicated work. It is natural, then, to ask whether these redundancies can be eliminated. In this paper, we show that the Gram matrix describing such data is block-circulant. We derive a transformation based on the Fourier transform that block-diagonalizes the Gram matrix, at once eliminating redundancies and partitioning the learning problem. This decomposition is valid for any dense features and several learning algorithms, and takes full advantage of modern parallel architectures. Surprisingly, it allows training with all the potential samples in sets of thousands of images. By considering the full set, we generate in a single shot the optimal solution, which is usually obtained only after several rounds of hard negative mining. We report speed gains on Caltech Pedestrians and INRIA Pedestrians of over an order of magnitude, allowing training on a desktop computer in a couple of minutes.

This work presents a novel Bayesian formulation for aligning faces in unseen images. Our approach is closely related to Constrained Local Models (CLM) [2] and Active Shape Models (ASM) [6], where an ensemble of local feature detectors are constrained to lie within the subspace spanned by a Point Distribution Model (PDM). Fitting a model to an image typically involves two steps: a local search using a detector, obtaining response maps for each landmark (likelihood term) and a global optimization that finds the PDM parameters that jointly maximize all the detections. The global optimization can be seen as a Bayesian inference problem, where the posterior distribution of the PDM parameters (and pose) can be inferred in a maximum a posteriori (MAP) sense. We present a novel Bayesian global optimization strategy, where the prior is used to encode the dynamic transitions of the PDM parameters. Using recursive Bayesian estimation we model the prior distribution of the data as being Gaussian. The mean and covariance were assumed to be unknown and treated as random variables. The Shape Model: The shape of a PDM is represented by the 2D locations of a mesh s = (x1,y1, . . . ,xv,yv) (v landmarks). Applying PCA on training examples, results in the parametric model s = s0 + Φb + Ψq, where s0 is the mean shape, Φ is the shape subspace matrix (n eigenvectors), b is a vector of shape parameters, q the pose parameters vector and Ψ holds four special eigenvectors that linearly model the 2D pose [4]. Goal: Given a 2v vector of observed positions y, the goal is to find the optimal set of parameters b that maximizes the posterior probability of being aligned. Using an Bayesian approach, the shape parameters are

Background modelling on tensor field has recently been proposed for foreground detection tasks. Taking into account the Riemannian structure of the tensor manifold, recent research has focused on developing parametric methods on the tensor domain e.g. gaussians mixtures (GMM) [7]. However, in some scenarios, simple parametric models do not accurately explain the physical processes. Kernel density estimators (KDE) have been successful to model, on Euclidean sample spaces the nonparametric nature of complex, time varying, and non-static backgrounds [8]. Founded on the mathematically rigorous KDE paradigm on general Riemannian manifolds [15], we define a KDE specifically to operate on the tensor manifold. We present a mathematically-sound framework for nonparametric modeling on tensor field to foreground segmentation. We endow the tensor manifold with two well-founded Riemannian metrics, i.e. Affine-Invariant and Log-Euclidean. Theoretical aspects are defined and the metrics are compared experimentally. Theoretic analysis and experimental results demonstrate the promise/effectiveness of the framework.

Multiple object tracking has been formulated recently as a global optimization problem, and solved efficiently with optimal methods such as the Hungarian Algorithm. A severe limitation is the inability to model multiple objects that are merged into a single measurement, and track them as a group, while retaining optimality. This work presents a new graph structure that encodes these multiple-match events as standard one-to-one matches, allowing computation of the solution in polynomial time. Since identities are lost when objects merge, an efficient method to identify groups is also presented, as a flow circulation problem. The problem of tracking individual objects across groups is then posed as a standard optimal assignment. Experiments show increased performance on the PETS 2006 and 2009 datasets compared to state-of-the-art algorithms.

The paper proposes a new method to perform foreground detection by means of background modeling using the tensor concept. Sometimes, statistical modelling directly on image values is not enough to achieve a good discrimination. Thus the image may be converted into a more information rich form, such as a tensor field, to yield latent discriminating features. Taking into account the theoretically well-founded differential geometrical properties of the Riemannian manifold where tensors lie, we propose a new approach for foreground detection on tensor field based on data modeling by means of Gaussians mixtures directly on tensor domain. We introduced a online Kmeans approximation of the Expectation Maximization algorithm to estimate the parameters based on an Affine-Invariant metric. This metric has excellent theoretical properties but essentially due to the space curvature the computational burden is high. We propose a novel Kmeans algorithm based on a new family of metrics, called Log-Euclidean, in order to speed up the process, while conserving the same theoretical properties. Contrary to the affine case, we obtain a space with a null curvature. Hence, classical statistical tools usually reserved to vectors are efficiently generalized to tensors in the Log-Euclidean framework. Theoretical aspects are presented and the Affine-Invariant and Log-Euclidean frameworks are compared experimentally. From a practical point of view, results are similar to those of the Affine-Invariant framework but are obtained much faster. Theoretic analysis and experimental results demonstrate the promise and effectiveness of the proposed framework.

This work addresses the fitting of 3D deformable face models from a single view through 2.5D Active Appearance Models (AAM). The main contribution of this paper is the use of 2.5D AAM that combines a 3D metric Point Distribution Model (PDM) and a 2D appearance model whose control points are defined by full perspective projections of the PDM. The advantage is that, assuming a calibrated camera, 3D metric shapes can be retrieved from single view images. Two algorithms and computationally efficient approximations are proposed, the Simultaneous Forwards Additive (SFA) and the Normalization Forwards Additive (NFA), both based on the Lucas Kanade framework. The SFA algorithm is computationally expensive but more accurate. It searches for shape and appearance parameters simultaneously whereas the NFA projects out the appearance from the error image and searches only for the shape parameters. Expanded solutions for the SFA and NFA are also proposed in order to take into account head self occlusions. An extensive performance evaluation is presented. The frequency of convergence for the SFA, NFA and their efficient approximation is evaluated, showing that the 2.5D model can outperform 2D based methods. The Robust extensions to occlusion were tested on a synthetic sequence showing that the model can deal robustly with large head rotation.

In this paper is proposed a novel statistical learning approach, to identify cast shadows, and model their generation. We exploit the theoretically well-founded directional statistics field, in order to formulate the generation of cast shadows as a Mixture of Von Mises-Fisher distributions (MovMF) on the unit sphere. This formulation is based on a bi-illuminant physical model of cast shadows, where no prior assumptions of the spectral power distribution (SPD) of the direct light sources and ambient illumination in the scene are made. Founded on a rigorous directional statistics approach, this parametric framework is capable of modelling the shaded surface behavior in complex illumination scenes and meet real time requirements. This better model discriminating cast shadows provides a more compact representation, and achieve better accuracy, with less data and much less computation time, compared with non-parametric models previously proposed. Theoretic analysis and experimental evaluations demonstrate the effectiveness of the proposed framework.

Matching methods such as the Hungarian algorithm have recently made an appearance as an alternative to classical tracking algorithms in computer vision, since they are able to find the set of tracks that optimizes well-defined criteria over a given video sequence. However, despite being globally optimal, they carry a cost: since they require complete knowledge of the sequence, such methods cannot work with continuous video streams, a crucial requirement of realistic video surveillance applications. We were able to use the recently proposed Dynamic Hungarian Algorithm in an innovative way, adapting it to the well-known sliding window methodology. The algorithm is able to run in real-time, while retaining its optimality. We tested our implementation on several datasets, tracking humans and vehicles, and obtained reliable results using the same set of parameters on all sequences.

Statistical modeling in color space is a widely used approach for background modeling to foreground segmentation. Nevertheless, sometimes computing such statistics directly on image values is not enough to achieve a good discrimination. Thus the image may be converted into a more information rich form, such as a tensor field, in which can be encoded color and gradients. In this paper, we exploit the theoretically well-founded differential geometrical properties of the Riemannian manifold where tensors lie. We propose a novel and efficient approach for foreground segmentation on tensor field based on data modeling by means of Gaussians mixtures (GMM) directly in the tensor domain. We introduced a Expectation Maximization (EM) algorithm to estimate the mixture parameters, and are proposed two algorithms based on an online K-means approximation of EM, in order to speed up the process. Theoretic analysis and experimental evaluations demonstrate the promise and effectiveness of the proposed framework.

This paper presents a general approach for the simultaneous tracking of multiple moving targets using a generic active stereo setup. The problem is formulated on the plane, where cameras are modeled as “line scan cameras,” and targets are described as points with unconstrained motion. We propose to control the active system parameters in such a manner that the images of the targets in the two views are related by a homography. This homography is specified during the design stage and, thus, can be used to implicitly encode the desired tracking behavior. Such formulation leads to an elegant geometric framework that enables a systematic and thorough analysis of the problem at hand. The benefits of the approach are illustrated by applying the framework to two distinct stereo configurations. In the first case, we assume two pan-tilt-zoom cameras, with rotation and zoom control, which are arbitrarily placed in the working environment. It is proved that such a stereo setup can track up toN= 3 free-moving targets, while assuring that the image location of each target is the same for both views. The second example considers a robot head with neck pan motion and independent eye rotation. For this case, it is shown that it is not possible to track more thanN= 2 targets because of the lack of zoom. The theoretical framework is used to derive the control equations, and the implementation of the tracking behavior is described in detail. The correctness of the results is confirmed through simulations and real tracking experiments.

This article presents a general approach for the active stereo tracking of multiple moving targets. The problem is formulated on the plane, where cameras are modeled as “line scan cameras” and targets are described as points with unconstrained motion. We propose to control the active system parameters in such a manner that the images of the targets in the two views are related by an homography. This homography is specified during the design stage and implicitly encodes the tracking behavior. It is shown that this formulation leads to an elegant geometric framework that enables to decide about the feasibility of a particular active tracking task. We apply it to prove that two cameras with rotation and zoom control, can track up to three moving targets, while assuring that the image location of each target is the same for both views. In addition, the framework is also useful for devising tracking strategies and deriving the required control equations. This feature is illustrated through a real experiment on tracking two independent targets using a binocular stereo head.