Predicting human body movements paves the way for many modern applications, such as pedestrian path control in autonomous driving, adaption of human-robot cooperation, and many more. Yet, it is not just about knowing what comes next, it’s about anticipating, guiding and evaluating motion in real-time, which can be particularly interesting for the automated analysis of physical exercises. Therefore, we propose a dual-stream model that is sensitive to the underlying temporal patterns present in physical exercise actions. The model leverages both motion history and its accumulation over time to improve the prediction accuracy of the 3D poses. The training procedure is built on the premise that learning the necessary displacements from the last pose seen to the future ones is more effective than direct pose regression. Moreover, the supervision loss is designed to follow certain principles of human kinematics. Namely, the human joints should not have the same weight during the optimization process as they tend display different variability in motion characteristics. The overall results show that our proposal improves the long-term prediction of human movements in the target Fit3D dataset. Moreover, our model follows the recent success of MLP-based approaches that emphasize that simplicity is the key to success and contains only ≈ 0.28 M trainable parameters.

• Proposes a self-supervised pre-train framework for vehicle-centric visual tasks. • Extends CGD-MAE with richer data analysis and an enhanced pre-training design. • Unifies masked-contrastive and CLIP-guided semantic objectives via feature fusion. • Ablation and qualitative results validate the proposed design. • Improves state-of-the-art vehicle-centric benchmarks in fine-tuning and linear-probe. In this work, we present VC-SCMAE, a Vehicle-Centric Semantic Contrastive-Guided Masked Autoencoder framework that distills knowledge from multimodal foundational models. Our approach extends MAE pre-training with contrastive guidance, combining masked image modeling with instance-level discrimination to produce more robust and transferable representations. On top of this discriminative backbone, we apply CLIP-style semantic distillation, leveraging a large-scale vehicle dataset (Automobile1M) and a visually grounded unpaired text corpus . Unlike conventional vision–language models that rely on aligned image–text pairs, our method transfers semantic knowledge from a pre-trained CLIP model without requiring explicit alignment. We further introduce specialized distillation losses that enhance open-vocabulary logits during vision-language distillation, thereby strengthening semantic alignment across modalities. Experiments demonstrate that VC-SCMAEeffectively transfers to vehicle-specific downstream tasks via both linear probing and fine-tuning, unifying structural, discriminative, and semantic understanding within a single pre-training framework.

This work introduces a CLIP-guided Masked Autoencoder (CGD-MAE) pre-training strategy designed to enhance the performance of existing ViT-based architectures for vehicle re-identification (V-ReID). We propose a simple yet effective ViT backbone pre-training approach that prioritizes data quality, quantity, and diversity, leveraging Automobile1M—a novel large-scale curated vehicle dataset derived from publicly available sources. Using hierarchical k-means clustering for automatic data curation, we select one million diverse samples from a large pool of vehicle images, addressing long-tailed distributions and improving backbone performance. Moreover, a modified global-context semantic distillation from large CLIP models further emphasizes the impact of dataset curation. Pre-training CGD-MAE on Automobile1M has proven to be beneficial in enhancing performance in state-of-the-art (SoTA) ViT-based model architectures for V-ReID and various downstream vehicle-centric applications. These results highlight its strong potential as a universal vehicle-specific pre-training strategy, enhancing feature learning and adaptability across a wide range of vehicle-related tasks. Code is available at https://github.com/AlexMaks02/CGD-MAE.

Abstract Probabilistic models with tractable marginalization are those in which evidence queries involving marginalization of variables are guaranteed to be computable in polynomial time in the size of the model. The tractability of marginalization of current classes of tractable probabilistic models is achieved through structural properties imposed over the functions they decompose to and it is an open question whether there are more general classes of probabilistic models with tractable marginalization than the current ones. That question is settled in this paper by the usage of boolean circuits to express them. On one hand, bounds on the number of simple and local operations required by a circuit to solve a problem are used as measures of complexity and on the other the circuit value problem is guaranteed to have complexity bounded by a polynomial function on the number of its gates. It is shown that choices of the definitions of the variables to use have an impact on the circuit description length as they are paramount to definitions of simplicity and locality. The framework that is put forward builds on a construction of a circuit representation of a dataset and on the definition of problems that take that circuit as an input such that the answers of the problems correspond to values of interest. Based on that, a set of circuit value problems are defined so that given inputs that identify queries of interest, the corresponding output values are obtained as a result of evaluation of those circuits. This approach is more general than current ones, in that, for any algorithm that runs in polynomial time over a model there is a circuit with number of gates bounded by a polynomial function of the time that the algorithm takes to be executed.

Temporal analysis of workout routines comes with a major difficulty: the observed actions or processes are naturally sequences of variable length. Besides being highly dependent on the exercise itself, the faster or slower repetition of the pattern’s temporal dynamics is also determined by the individual’s performance. In this paper, we present a Transformer-based Deep Neural Network to perform classification over 19 phases of 5 exercises, common to CrossFit routines. From this baseline, we aim to perform workout video segmentation, while creating a descriptor that enables repetition counting or feedback on the exercise execution. More specifically, a 2D Human Pose Network creates heatmap-based features that capture the human body pose, which are fed into the Transformer Encoder with 4 attention heads. A final Multilayer Perceptron with 3 Dense layers performs the phase classification task. To this end, we have trained our model using a previously acquired dataset that is naturally imbalanced, e.g. 6 classes have less than 8k samples and 9 classes have more than 24k. Finally, the obtained results show that we are able to divide videos in a temporally consistent manner, outperforming a state-of-the-art model that counts repetitive actions, specifically for 4 out of the 5 exercises.

Faces in motion contain rich information about a myriad of social cues, including identity. However, is still unknown how much information about a person’s identity is conveyed by different kinds of facial movements and the precise role of facial motion as a cue for identity is still under debate. Open questions include how much information about identity is embedded in facial movements, and how efficiently a perceptual system extracts those cues toward identity. We address these questions by exploring the potential of Neuromorphic Vision Sensors to perform identity recognition through facial dynamics derived from speech and simultaneously validate how contributory facial motion is for human face identity categorization. As far as we know, this is the first work that evaluates the aptitude of the technology offered by neuromorphic cameras for completing the task of facial identity recognition. A one-stream inflated 3D convnet was used to evaluate the ability of different frame-based events encoding strategies in classifying identity through Short-Time Facial Motion Tokens (ST-FAMOTO). A consistent set of evaluations was conducted using two small datasets of individuals engaged in speech tasks, specifically text reading and speech-driven visual passcode. Results reported consistently consolidate the contribution of facial dynamics in the process of identity recognition and the usefulness of neuromorphic sensor devices in acquiring identity discriminatory facial micro-movements. Additionally, it is also proposed the first Neuromorphic Vision Dataset for Speech Induced Facial Dynamics (NVSFD Dataset), created in the scope of this work and available to facilitate future third-party investigation on the topic.

Small untethered soft robots have potential for diverse applications, particularly in constrained spaces where the use of a tethered device would be infeasible. Examples include biomedical applications such as brachytherapy, fine-needle biospy and micro-needle drug delivery. To advance soft robots towards these applications, there is a need to establish methods for tracking and control using clinically-relevant methods. This study demonstrates motion planning and magnetic control of a soft untethered robot, using ultrasound images as feedback. The closed-loop control of the Millipede soft robot is first validated using a camera-based tracker, where the deviation between the planned path and the trajectory of the robot is 1.71 mm. Afterwards, two methods for ultrasound-based tracking capable of estimating the pose of the robot are proposed, a geometric approach and a convolutional neural network (CNN), and their performance is compared using a video camera as ground truth. Following this, the CNN method replaces the camera tracker to estimate the position and orientation of the robot. The closed-loop system using ultrasound images guides the robot through the workspace while avoiding virtual obstacles, and achieves an average tracking error of 1.59 mm and an angle error of 2.24°.

This paper targets deformable face model matching in images using cascaded regression techniques. Recently, the cascaded regression strategies have became rather popular solutions to solve nonlinear objective functions by learning a pipeline sequence of linear regressors. However, despite their success, the standard formulation do not enforce shape consistency through the cascade (mostly because the individual regressors are learnt independently). In this paper we revisit the cascaded regression framework and propose a number of improvements. First we explore the simplicity and compactness of using a linear shape model for such tasks, effectively solving the previous drawback. Then we propose to extend the linear regression module into a nonlinear version, by means of recent Convolutional Neutral Networks (CNNs) techniques, modified to include a weighted shape aware loss function. Moreover, since CNNs often require massive amounts of data to perform well, we took advantage of the shape model probabilistic framework to efficiently bootstrap new data. Our nonlinear cascade regression method is evaluated in several databases (LFPW, LFW, HELEN and 300W), where the results demonstrate the effectiveness of the proposed method.

This paper addresses the problem of localizing facial landmarks with deformable face models using cascaded regression strategies. Recently, these methods have become quite popular, standing out as simple and efficient approaches to optimize nonlinear objective functions. In this paper, we target the well-known Lucas and Kanade (LK) image alignment formulation and introduce the Simultaneous Cascaded Regression (SCR) technique, which can be considered as a cascaded regression extension of the Simultaneous Forwards Additive / Inverse Composition approaches. In contrast to previous LK techniques (Newton based optimizations) which require to recompute Jacobian and Hessians matrices at each iteration, our approach learns (offline) a sequence of descent directions, effectively behaving as averaged steepest descent matrices. Under this revised technique, we propose a part-based generative model (with a linear warp function), that accounts with the underlying shape and appearance structure embedded into regression process itself. Our method is validated on a number of experiments on several datasets (LFPW, LFW, HELEN, 300W), demonstrating a noticeable gain in accuracy/fitting performance when compared with other face alignment solutions.

This paper proposes an extended Constrained Local Model (CLM) formulation for aligning faces using depth information. The CLMs are popular methods that were initially designed to locate facial features in regular intensity images. Briefly, they combine a set of local detectors, one for each landmark, whose locations are regularized by a linear shape model. Fitting a CLM is usually framed as a two step approach: locally search, using the detectors, producing response maps (likelihood maps) followed by a global optimization strategy that jointly maximize all detection scores while enforcing an appropriate shape. Including depth data could be simply posed as adding additional likelihood sources to the main formulation. The paper discusses several likelihood fusion techniques and propose to jointly learn a multi-dimensional correlation filter as a more reliable solution. Moreover, we propose to learn the local detectors, in the Fourier domain, effectively augmenting the training set with virtual samples. Besides improving the detections reliability, this approach is particular important when applied to depth data, as no additional processing is required (such as fill missing information). The performance evaluation shows that our extended approach further increases the fitting performance (accuracy) effectively proving the benefit of using depth data in facial alignment tasks.

Manifold-based domain adaptation algorithms are receiving increasing attention in computer vision to model distribution shifts between source and target domain. In contrast to early works, that mainly explore intermediate subspaces along geodesics, in this work we propose to interpolate subspaces through C1-smooth curves on the Grassmann manifold. The new methodis based on the geometric Casteljau algorithm that is used to generate smooth interpolating polynomial curves on non-euclidean spaces and can be extended to generate polynomial splines that interpolate a given set of data on the Grassmann manifold. To evaluate the usefulness of the proposed interpolating curves on vision related problems, several experiments were conducted. We show the advantage of using smooth subspaces interpolation in multi-source unsupervised domain adaptation problems and in object recognition problems across datasets.

This paper presents a novel Bayesian formulation for aligning faces in unseen images. Our approach revisits the Constrained Local Models (CLM) formulation where an ensemble of local feature detectors are constrained to lie within the subspace spanned by a Point Distribution Model (PDM). Fitting such a model to an image typically involves two main steps: a local search using a detector, obtaining response maps for each landmark (likelihood term) and a global optimization that finds the PDM parameters that jointly maximize all the detections at once. The so-called global optimization can be posed as a Bayesian inference problem, where the posterior distribution of the shape (and pose) parameters can be inferred in a maximum a posteriori (MAP) sense. This work introduces an extended Bayesian global optimization strategy that includes two novel additions: (1) to perform second order updates of the PDM parameters (accounting for their covariance) and (2) to model the underlying dynamics of the shape variations, encoded in the prior term, by using recursive Bayesian estimation. Extensive evaluations were performed against state-of-the-art methods on several standard datasets (IMM, BioID, XM2VTS, LFW and FGNET Talking Face). Results show that the proposed approach significantly increases the fitting performance.

While data has certainly taken the center stage in computer vision in recent years, it can still be difficult to obtain in certain scenarios. In particular, acquiring ground truth 3D shapes of objects pictured in 2D images remains a challenging feat and this has hampered progress in recognition-based object reconstruction from a single image. Here we propose to bypass previous solutions such as 3D scanning or manual design, that scale poorly, and instead populate object category detection datasets semi-automatically with dense, per-object 3D reconstructions, bootstrapped from:(i) class labels, (ii) ground truth figure-ground segmentations and (iii) a small set of keypoint annotations. Our proposed algorithm first estimates camera viewpoint using rigid structure-from-motion and then reconstructs object shapes by optimizing over visual hull proposals guided by loose within-class shape similarity assumptions. The visual hull sampling process attempts to intersect an object's projection cone with the cones of minimal subsets of other similar objects among those pictured from certain vantage points. We show that our method is able to produce convincing per-object 3D reconstructions and to accurately estimate cameras viewpoints on one of the most challenging existing object-category detection datasets, PASCAL VOC. We hope that our results will re-stimulate interest on joint object recognition and 3D reconstruction from a single image.

Recently, a particular paradigm [18] in the domain adaptation field has received considerable attention by introducing novel and important insights to the problem. In this case, the source and target domains are represented in the form of subspaces, which are treated as points on the Grassmann manifold. The geodesic curve between them is sampled to obtain intermediate points. Then a classifier is learnt using the projections of the data onto these subspaces. Despite its relevance and popularity, this paradigm [18] contains some limitations. Firstly, in real-world applications, that simple curve (i.e. shortest path) does not provide the necessary flexibility to model the domain shift between the training and testing data sets. Secondly, by using the geodesic curve, we are restricted to only one source domain, which does not allow to take fully advantage of the multiple datasets that are available nowadays. It is then, natural to ask whether this popular concept could be extended to deal with more complex curves and to integrate multi-sources domains. This is a hard problem considering the Riemannian structure of the space, but we propose a mathematically well-founded idea that enables us to solve it. We exploit the geometric insight of rolling maps [30] to compute a spline curve on the Grassmann manifold. The benefits of the proposed idea are demonstrated through several empirical studies on standard datasets. This novel concept allows to explicitly integrate multi-source domains while the previous one [18] uses the mean of all sources. This enables to model better the domain shift and take fully advantage of the training datasets.

Semantic segmentation and object detection are nowadays dominated by methods operating on regions obtained as a result of a bottom-up grouping process (segmentation) but use feature extractors developed for recognition on fixed-form (e.g. rectangular) patches, with full images as a special case. This is most likely suboptimal. In this paper we focus on feature extraction and description over free-form regions and study the relationship with their fixed-form counterparts. Our main contributions are novel pooling techniques that capture the second-order statistics of local descriptors inside such free-form regions. We introduce second-order generalizations of average and max-pooling that together with appropriate non-linearities, derived from the mathematical structure of their embedding space, lead to state-of-the-art recognition performance in semantic segmentation experiments without any type of local feature coding. In contrast, we show that codebook-based local feature coding is more important when feature extraction is constrained to operate over regions that include both foreground and large portions of the background, as typical in image classification settings, whereas for high-accuracy localization setups, second-order pooling over free-form regions produces results superior to those of the winning systems in the contemporary semantic segmentation challenges, with models that are much faster in both training and testing.

The core component of most modern trackers is a discriminative classifier, tasked with distinguishing between the target and the surrounding environment. To cope with natural image changes, this classifier is typically trained with translated and scaled sample patches. Such sets of samples are riddled with redundancies-any overlapping pixels are constrained to be the same. Based on this simple observation, we propose an analytic model for datasets of thousands of translated patches. By showing that the resulting data matrix is circulant, we can diagonalize it with the discrete Fourier transform, reducing both storage and computation by several orders of magnitude. Interestingly, for linear regression our formulation is equivalent to a correlation filter, used by some of the fastest competitive trackers. For kernel regression, however, we derive a new kernelized correlation filter (KCF), that unlike other kernel algorithms has the exact same complexity as its linear counterpart. Building on it, we also propose a fast multi-channel extension of linear correlation filters, via a linear kernel, which we call dual correlation filter (DCF). Both KCF and DCF outperform top-ranking trackers such as Struck or TLD on a 50 videos benchmark, despite running at hundreds of frames-per-second, and being implemented in a few lines of code (Algorithm 1). To encourage further developments, our tracking framework was made open-source.

This work presents a novel non-parametric Bayesian formulation for aligning faces in unseen images. Popular approaches, such as the Constrained Local Models (CLM) or the Active Shape Models (ASM), perform facial alignment through a local search, combining an ensemble of detectors with a global optimization strategy that constraints the facial feature points to be within the subspace spanned by a Point Distribution Model (PDM). The global optimization can be posed as a Bayesian inference problem, looking to maximize the posterior distribution of the PDM parameters in a maximum a posteriori (MAP) sense. Previous approaches rely exclusively on Gaussian inference techniques, i.e. both the likelihood (detectors responses) and the prior (PDM) are Gaussians, resulting in a posterior which is also Gaussian, whereas in this work the posterior distribution is modeled as being non-parametric by a Kernel Density Estimator (KDE). We show that this posterior distribution can be efficiently inferred using Sequential Monte Carlo methods, in particular using a Regularized Particle Filter (RPF). The technique is evaluated in detail on several standard datasets (IMM, BioID, XM2VTS, LFW and FGNET Talking Face) and compared against state-of-the-art CLM methods. We demonstrate that inferring the PDM parameters non-parametrically significantly increase the face alignment performance.

We address the problem of populating object category detection datasets with dense, per-object 3D reconstructions, bootstrapped from class labels, ground truth figure-ground segmentations and a small set of keypoint annotations. Our proposed algorithm first estimates camera viewpoint using rigid structure-from-motion, then reconstructs object shapes by optimizing over visual hull proposals guided by loose within-class shape similarity assumptions. The visual hull sampling process attempts to intersect an object's projection cone with the cones of minimal subsets of other similar objects among those pictured from certain vantage points. We show that our method is able to produce convincing per-object 3D reconstructions on one of the most challenging existing object-category detection datasets, PASCAL VOC. Our results may re-stimulate once popular geometry-oriented model-based recognition approaches.

This paper addresses to the problem of aligning images in unseen faces. The Constrained Local Models (CLM) are popular methods that combine a set of local landmark detectors whose locations are constrained to lie in a subspace spanned by a linear shape model. The CLM fitting is usually based on a two step approach: locally search, using the detectors, producing response maps (likelihood) followed by a global optimization strategy that jointly maximize all detections at once. In this paper, we mainly focus on the first stage: improving the detectors reliability. Usually the local landmarks detectors are far from perfect. Most often are designed to be fast, having a small support region and are learnt from limited data. As consequence, they will suffer from detection ambiguities. Here we propose to improve the detectors performance by considering multiple detection per landmark. In particular, we propose a joint learning of the detectors by clustering of their training data. Afterwards, the multiple likelihoods are combined using a nonlinear fusion approach. The performance evaluation shows that our (extended) approach further increases the fitting performance of the CLM formulation, when compared with recent state-of-the-art methods.

Why is pedestrian detection still very challenging in realistic scenes? How much would a successful solution to monocular depth inference aid pedestrian detection? In order to answer these questions we trained a state-of-the-art deformable parts detector using different configurations of optical images and their associated 3D point clouds, in conjunction and independently, leveraging upon the recently released KITTI dataset. We propose novel strategies for depth upsampling and contextual fusion that together lead to detection performance which exceeds that of the RGB-only systems. Our results suggest depth cues as a very promising mid-level target for future pedestrian detection approaches.

In the past few years there has been a growing interest on geometric frameworks to learn supervised classification models on Riemannian manifolds [32, 28]. A popular framework, valid over any Riemannian manifold, was proposed in [32] for binary classification. Once moving from binary to multi-class classification this paradigm is not valid anymore, due to the spread of multiple positive classes on the manifold [28]. It is then natural to ask whether the multi-class paradigm could be extended to operate on a large class of Riemannian manifolds. We propose a mathematically well-founded classification paradigm that allows to extend the work in [32] to multi-class models, taking into account the structure of the space. The idea is to project all the data from the manifold onto an affine tangent space at a particular point. To mitigate the distortion induced by local diffeomorphisms, we introduce for the first time in the computer vision community a well-founded mathematical concept, so-called Rolling map [22, 17]. The novelty in this alternate school of thought is that the manifold will be firstly rolled (without slipping or twisting) as a rigid body, then the given data is unwrapped onto the affine tangent space, where the classification is performed.

Competitive sliding window detectors require vast training sets. Since a pool of natural images provides a nearly endless supply of negative samples, in the form of patches at different scales and locations, training with all the available data is considered impractical. A staple of current approaches is hard negative mining, a method of selecting relevant samples, which is nevertheless expensive. Given that samples at slightly different locations have overlapping support, there seems to be an enormous amount of duplicated work. It is natural, then, to ask whether these redundancies can be eliminated. In this paper, we show that the Gram matrix describing such data is block-circulant. We derive a transformation based on the Fourier transform that block-diagonalizes the Gram matrix, at once eliminating redundancies and partitioning the learning problem. This decomposition is valid for any dense features and several learning algorithms, and takes full advantage of modern parallel architectures. Surprisingly, it allows training with all the potential samples in sets of thousands of images. By considering the full set, we generate in a single shot the optimal solution, which is usually obtained only after several rounds of hard negative mining. We report speed gains on Caltech Pedestrians and INRIA Pedestrians of over an order of magnitude, allowing training on a desktop computer in a couple of minutes.

This work presents a novel Bayesian formulation for aligning faces in unseen images. Our approach is closely related to Constrained Local Models (CLM) [2] and Active Shape Models (ASM) [6], where an ensemble of local feature detectors are constrained to lie within the subspace spanned by a Point Distribution Model (PDM). Fitting a model to an image typically involves two steps: a local search using a detector, obtaining response maps for each landmark (likelihood term) and a global optimization that finds the PDM parameters that jointly maximize all the detections. The global optimization can be seen as a Bayesian inference problem, where the posterior distribution of the PDM parameters (and pose) can be inferred in a maximum a posteriori (MAP) sense. We present a novel Bayesian global optimization strategy, where the prior is used to encode the dynamic transitions of the PDM parameters. Using recursive Bayesian estimation we model the prior distribution of the data as being Gaussian. The mean and covariance were assumed to be unknown and treated as random variables. The Shape Model: The shape of a PDM is represented by the 2D locations of a mesh s = (x1,y1, . . . ,xv,yv) (v landmarks). Applying PCA on training examples, results in the parametric model s = s0 + Φb + Ψq, where s0 is the mean shape, Φ is the shape subspace matrix (n eigenvectors), b is a vector of shape parameters, q the pose parameters vector and Ψ holds four special eigenvectors that linearly model the 2D pose [4]. Goal: Given a 2v vector of observed positions y, the goal is to find the optimal set of parameters b that maximizes the posterior probability of being aligned. Using an Bayesian approach, the shape parameters are

The paper describes a kinematic model-based solution to estimate simultaneously the calibration parameters of the vision system and the full-motion of an object using a sequence of noisy images captured by a set of stereo affine cameras. Assuming a smooth motion, an Iterated Extended Kalman Filter (IEKF) is used to recursively estimate the cameras projection matrices and the object's full-motion over time. The estimator was developed having in mind the structure health monitoring of large structures of civil engineering domain, observed at long distance, in particular, of long deck suspension bridges. Results related to the performance evaluation, obtained by numerical simulation and with real experiments, are reported.

Background modelling on tensor field has recently been proposed for foreground detection tasks. Taking into account the Riemannian structure of the tensor manifold, recent research has focused on developing parametric methods on the tensor domain e.g. gaussians mixtures (GMM) [7]. However, in some scenarios, simple parametric models do not accurately explain the physical processes. Kernel density estimators (KDE) have been successful to model, on Euclidean sample spaces the nonparametric nature of complex, time varying, and non-static backgrounds [8]. Founded on the mathematically rigorous KDE paradigm on general Riemannian manifolds [15], we define a KDE specifically to operate on the tensor manifold. We present a mathematically-sound framework for nonparametric modeling on tensor field to foreground segmentation. We endow the tensor manifold with two well-founded Riemannian metrics, i.e. Affine-Invariant and Log-Euclidean. Theoretical aspects are defined and the metrics are compared experimentally. Theoretic analysis and experimental results demonstrate the promise/effectiveness of the framework.

Multiple object tracking has been formulated recently as a global optimization problem, and solved efficiently with optimal methods such as the Hungarian Algorithm. A severe limitation is the inability to model multiple objects that are merged into a single measurement, and track them as a group, while retaining optimality. This work presents a new graph structure that encodes these multiple-match events as standard one-to-one matches, allowing computation of the solution in polynomial time. Since identities are lost when objects merge, an efficient method to identify groups is also presented, as a flow circulation problem. The problem of tracking individual objects across groups is then posed as a standard optimal assignment. Experiments show increased performance on the PETS 2006 and 2009 datasets compared to state-of-the-art algorithms.

The paper proposes a new method to perform foreground detection by means of background modeling using the tensor concept. Sometimes, statistical modelling directly on image values is not enough to achieve a good discrimination. Thus the image may be converted into a more information rich form, such as a tensor field, to yield latent discriminating features. Taking into account the theoretically well-founded differential geometrical properties of the Riemannian manifold where tensors lie, we propose a new approach for foreground detection on tensor field based on data modeling by means of Gaussians mixtures directly on tensor domain. We introduced a online Kmeans approximation of the Expectation Maximization algorithm to estimate the parameters based on an Affine-Invariant metric. This metric has excellent theoretical properties but essentially due to the space curvature the computational burden is high. We propose a novel Kmeans algorithm based on a new family of metrics, called Log-Euclidean, in order to speed up the process, while conserving the same theoretical properties. Contrary to the affine case, we obtain a space with a null curvature. Hence, classical statistical tools usually reserved to vectors are efficiently generalized to tensors in the Log-Euclidean framework. Theoretical aspects are presented and the Affine-Invariant and Log-Euclidean frameworks are compared experimentally. From a practical point of view, results are similar to those of the Affine-Invariant framework but are obtained much faster. Theoretic analysis and experimental results demonstrate the promise and effectiveness of the proposed framework.

This work addresses the fitting of 3D deformable face models from a single view through 2.5D Active Appearance Models (AAM). The main contribution of this paper is the use of 2.5D AAM that combines a 3D metric Point Distribution Model (PDM) and a 2D appearance model whose control points are defined by full perspective projections of the PDM. The advantage is that, assuming a calibrated camera, 3D metric shapes can be retrieved from single view images. Two algorithms and computationally efficient approximations are proposed, the Simultaneous Forwards Additive (SFA) and the Normalization Forwards Additive (NFA), both based on the Lucas Kanade framework. The SFA algorithm is computationally expensive but more accurate. It searches for shape and appearance parameters simultaneously whereas the NFA projects out the appearance from the error image and searches only for the shape parameters. Expanded solutions for the SFA and NFA are also proposed in order to take into account head self occlusions. An extensive performance evaluation is presented. The frequency of convergence for the SFA, NFA and their efficient approximation is evaluated, showing that the 2.5D model can outperform 2D based methods. The Robust extensions to occlusion were tested on a synthetic sequence showing that the model can deal robustly with large head rotation.

In this paper is proposed a novel statistical learning approach, to identify cast shadows, and model their generation. We exploit the theoretically well-founded directional statistics field, in order to formulate the generation of cast shadows as a Mixture of Von Mises-Fisher distributions (MovMF) on the unit sphere. This formulation is based on a bi-illuminant physical model of cast shadows, where no prior assumptions of the spectral power distribution (SPD) of the direct light sources and ambient illumination in the scene are made. Founded on a rigorous directional statistics approach, this parametric framework is capable of modelling the shaded surface behavior in complex illumination scenes and meet real time requirements. This better model discriminating cast shadows provides a more compact representation, and achieve better accuracy, with less data and much less computation time, compared with non-parametric models previously proposed. Theoretic analysis and experimental evaluations demonstrate the effectiveness of the proposed framework.

Matching methods such as the Hungarian algorithm have recently made an appearance as an alternative to classical tracking algorithms in computer vision, since they are able to find the set of tracks that optimizes well-defined criteria over a given video sequence. However, despite being globally optimal, they carry a cost: since they require complete knowledge of the sequence, such methods cannot work with continuous video streams, a crucial requirement of realistic video surveillance applications. We were able to use the recently proposed Dynamic Hungarian Algorithm in an innovative way, adapting it to the well-known sliding window methodology. The algorithm is able to run in real-time, while retaining its optimality. We tested our implementation on several datasets, tracking humans and vehicles, and obtained reliable results using the same set of parameters on all sequences.

A solution for Discriminative Active Appearance Models is proposed. The model consists in a set of descriptors which are covariances of multiple features evaluated over the neighborhood of the landmarks whose locations are governed by a Point Distribution Model (PDM). The covariance matrices are a special set of tensors that lie on a Riemannian manifold, which make it possible to measure the dissimilarity and to update them, imposing the temporal appearance consistency. The discriminative fitting method produce patch response maps found by convolution around the current landmark position. Since the minimum of the responce map isn't always the correct solution due to detection ambiguities, our method finds candidates to solutions based on a mean-shift algorithm, followed by an unsupervised clustering technique used to locate and group the candidates. A mahalanobis based metric is used to select the best solution that is consistent with the PDM. Finally the global PDM optimization step is performed using a weighted least-squares warp update, based on the Lucas Kanade framework. The weights were extracted from a landmark matching score statistics. The effectiveness of the proposed approach was evaluated on unseen data on the challenging Talking Face video sequence, demonstrating the improvement in performance.

Statistical modeling in color space is a widely used approach for background modeling to foreground segmentation. Nevertheless, sometimes computing such statistics directly on image values is not enough to achieve a good discrimination. Thus the image may be converted into a more information rich form, such as a tensor field, in which can be encoded color and gradients. In this paper, we exploit the theoretically well-founded differential geometrical properties of the Riemannian manifold where tensors lie. We propose a novel and efficient approach for foreground segmentation on tensor field based on data modeling by means of Gaussians mixtures (GMM) directly in the tensor domain. We introduced a Expectation Maximization (EM) algorithm to estimate the mixture parameters, and are proposed two algorithms based on an online K-means approximation of EM, in order to speed up the process. Theoretic analysis and experimental evaluations demonstrate the promise and effectiveness of the proposed framework.

In this paper a technique is presented to estimate the traffic intensity for each lane. This method does not require background estimation or even the identification and tracking of individual vehicles. It requires only the identification of each lane and the estimation of a bird eye view of the highway using a rectification method. To each rectified lane, an intensity profile is computed along the traffic flow direction for each frame, obtaining an image that represents the displacement as a time function. The main idea is to search for the best matching profile for different times and spaces on each lane and consequently obtain the velocity profile.

A solution for identity and facial expression recognition is proposed using a two stage classifier approach using low dimensional representation of the geometry of the face. Face geometry is extracted from input images using Active Appearance Models (AAM) and low dimensional manifolds were then derived using Laplacian Eigen-Maps (LE) resulting in two types of manifolds, one for model identity and the other for person-specific facial expression. The first stage uses a multiclass Support Vector Machines (SVM) to establish identity across expression changes. The second stage deals with person-specific expression recognition, and is composed by a network of seven Hidden Markov Models (HMM) displaced in parallel, each one specialized on the several facial emotions analysed. The decision was made by the sequence that yielded the highest probability. For evaluation proposes a database was build consisting on 6770 images captured from 4 people exhibiting 7 different emotions. The identity overall recognition rate was 96.8%. Facial expression results are identity dependent, and the most expressive individual achieves 81.2% of overall recognition rate.

This paper presents an approach to label and track multiple objects through both temporally and spatially significant occlusions. To this end, tracking is performed at both the region level and the object level. At the region level, a kernel based particle filter method is used to search for optimal region tracks which limits the scope of object trajectories. At the object level, each object is located based on adaptive appearance models, spatial distributions and inter-occlusion relationships. Region covariance matrices are used to model objects appearance. We analyzed the advantages of using Gabor functions as features and embedded them in the RCMs to get a more accurate descriptor. The proposed architecture is capable of tracking multiple objects even in the presence of periods of full occlusions. Results from experiments with real video data show the effectiveness of the approach hereby proposed.

This paper presents an integrated solution for vehicle's velocity estimation and vehicle counting. The proposed restores the scene geometric properties, building a ground plane rectified image. Moreover, multiple vehicles tracking is performed embedding the concept of region covariance descriptors in a particle filter framework. The results show the effectiveness of the approach here proposed in very clutter scenes.

A framework for automatic human head pose estimation from single view images is proposed. The 6DOF head pose was estimated using pose from orthography and scaling with iterations (POSIT) where a statistical anthropometric 3D rigid model is used as an approximation of the human head, combined with active appearance models (AAM) for facial features extraction and tracking. The overall performance of the proposed solution was evaluated comparing the results with a ground-truth data obtained by a pose planar approach. The results show that orientations and head location were, on average, found within 2deg or 1 cm error standard deviations respectively.

This paper presents an automatic method to estimate vehicle velocity and determine the number of vehicles per lane using rectified images. This approach requires the knowledge of two lengths on the ground plane and can be applied to highway scenarios that possess fairly straight lanes in areas near the camera. A scale factor is determined, in order to ascertain vehicle velocity in the sought units. This scale factor associates distances traveled on the image plane with distances on the ground plane and can be obtained by identifying the period and location of the striped highway lanes on the rectified background image. Once the previously referred lines are located it is then possible to locate the highway lane boundaries and, therefore, determine the mean vehicle velocity of each lane.

This work addresses the problem of human head pose estimation from single view images. 3D rigid head pose is estimated combining Active Appearance Models(AAM) with Pose from Orthography and Scaling with ITerations(POSIT). AAM shape landmarks are tracked over time and used in POSIT for pose estimation. A statistical anthropometric 3D model is used as reference. Several experiences were performed comparing our approach with a planar ground truth. It was achieved average standard deviations about 2 degrees in orientation and 1 centimeter in distance.

In this paper it is presented a robust segmentation process for detecting incidents on highways. This segmentation process is based on background subtraction and uses an efficient background model initialization and update to work 24/7. A cross- correlation based shadow detection is also used for minimizing ghosts. It is also proposed a stopped vehicle detection system based on the pixel history cache. This method has proved to be quite robust in terms of different weather conditions, lighting and image quality. Some experiments carried out on some highway scenarios demonstrate the robustness of the proposed solution.

We present an approach for detection, labelling and tracking multiple objects through both temporally and spatially significant occlusions. The proposed method builds on the idea of multiple objects scenario where grouping and occlusions are a reality. To this end, the objects are represented by covariance matrices and particle filters perform the object tracking. We propose a different measurement for the particles weights and a new update for the objects descriptor in a Riemannian framework. The results show the effectiveness of the approach hereby proposed in very clutter scenes.

In this paper it is described a solution to detect wrong way drivers on highways. The proposed solution is based on three main stages: learning, detection and validation. In the first stage, the orientation pattern of vehicles motion flow is learned and modelled by a mixture of Gaussians. The second stage (detection and temporal validation) makes use of the learned orientation model to detect objects moving on lane's opposite direction. The third and final stage uses an appearance based approach to ensure the detection of a vehicle before triggering an alarm. This methodology has proven to be quite robust in terms of different weather conditions, illumination and image quality. Some experiments carried out with several movies from traffic surveillance cameras on highways show the robustness of the proposed solution.

Deals with active tracking of 3D moving targets. Visual tracking is presented as a regulation control problem. The performance and robustness in visual control of motion depends both on the vision algorithms and the control structure. Delays and system latencies substantially affect the performance of visually guided systems. We discuss ways to cope with delays while improving system performance. Model predictive control strategies are proposed to compensate for the mechanical latency in visual control of motion.

Describes a tracking and accommodation system controlled by retinal motion flow. The tracking system decomposes two basic behaviors: smooth-pursuit and vergence. Both behaviors are optical flow based. The eye's accommodation (focusing) is based on a focusing-by-vergence approach combining a flow based vergence process with an off-line pre-calibration of the lens focusing odometry. The focusing and tracking system run in parallel and are both velocity controlled.

We describe an indoor surveillance system which enables the control of a binocular active vision system by a remote operator wearing a head mounted display. Visual feedback to the head mounted display is obtained using the images captured by the active vision system. To maximize the 3D perception of the operator the system is able to adjust the vergence angle of the active vision system in order to maintain fixation on the objects in the center of the image. A wide-angle lens static camera located on the surveillance area is also used to enable the detection and tracking of intruders. In case of an intrusion a map with the location of the detected targets is displayed on top of the images shown on the head mounted display. This information allows the remote operator to redirect his attention to where the target is.

We describe a focusing-by-vergence system combining a flow based vergence process with an off-line pre-calibration of the focusing odometry. The vergence process is controlled by the retinal optical flow disparity and the target depth velocity is directly obtained from this disparity. The relationship between the target depth (focused target distance) and the focusing odometry of the motorized lens is obtained by an off-line focusing calibration, using a bivariate polynomial to model the relationship between these parameters. The focusing system is velocity controlled and the focusing velocity is dependent on the target focusing depth velocity and the focal distance velocity. The influence of the focal distance velocity that arise due to the focusing process itself is compensated by the adjustment of the zoom parameter based on a constant magnification approach, or through the modeling of the focal distance as a function of the zoom and focus lens settings.

A system for tracking multiple targets in 3D is described. The system is made up of two pan-and-tilt units that are attached to the extremities of a rotating arm. This configuration has several advantages and can deal with several specific instances of tracking more than one target. A control strategy that guarantees equal target disparities in both images whenever targets are seen by both cameras is presented. This has advantages for segmentation and trajectory reconstruction. Target images are simultaneously visible in the cameras, enabling the recovery of the targets 3D trajectory. It is also shown that mutual occlusion occurs in a well-defined configuration and can therefore be dealt with.

Perspective camera calibration has been a research subject for a large group of researchers. However, only a small number of those methods base their approaches on the use of monoplane calibration points. We developed one of those methodologies using monoplane calibration points to perform an explicit three-dimensional (3-D) camera calibration. This methodology is based on an iterative approach. To avoid the singularities obtained with the calibration equations when monoplane calibration points are used, this method computes the calibration parameters in a multistep procedure and requires a first-guess solution for the intrinsic parameters. These intrinsic parameters are updated and their accuracy increased during the iterative procedure. All computations required are linear and in addition to the extrinsic parameters the proposed method also computes the first coefficient of the radial distortion (k/sub 1/) and the skew angle. A first-guess value for the focal length of the lens is required but its value is iteratively updated using the Gauss lens model. This methodology also includes the uncertainty horizontal image scale factor (S/sub x/) on the set of calibration parameters to be computed, which makes this approach independent of the accuracy of the horizontal scale factor. The proposed methodology has the advantage that it can be used with monoplane calibration data with no restrictions for the pose geometry of the camera. In this approach the pose estimation problem is treated separately, computing the pose orientation of the camera in a first step and using this information to compute the pose location. A model for the expected stability of the camera look angles from noisy image data and its stability analysis as a function of the pose geometry of the camera is presented. Several experimental and simulated analyzes were performed and are presented.

Perspective camera calibration has been in the last decades a research subject for a large group of researchers and as a result several camera calibration methodologies can be found in the literature. However only a small number of those methods base their approaches on the use of monoplane calibration points. This paper describes one of those methodologies that uses monoplane calibration points to realize an explicit 3D camera calibration. To avoid the singularity obtained with the calibration equations when monoplane calibration points are used, this method computes the calibration parameters in a multi-step procedure and requires a first-guess solution for the intrinsic parameters. These parameters are updated and their accuracy increased through an iterative procedure. A stability analysis as a function of the pose of the camera is presented. Camera pose view strategies for accurate camera orientation computation can be extracted from the pose view stability analysis.

We describe a surveillance system made up of a fixed camera and a binocular active vision system. A fixed camera is used to provide the system with peripheral motion detection capabilities. The binocular active system is used to track in 3D and reconstruct the trajectories of specific targets. Both are fully integrated and the redundancy is exploited to achieve a higher degree of robustness in detecting and tracking intruders.

The performance of a binocular active vision system depends mainly on two aspects: vision/image processing and control. We characterize the monocular performance of smooth pursuit. This system is used to track binocularly targets in a surveillance environment. One of the aspects of this characterization was the inclusion of the vision processing. To characterize the performance from the control point of view four standard types of inputs were used: step, ramp, parabola and sinusoid. The responses can be used to identify which subsystems can be optimized. We show that prediction and a velocity estimate are essential for a good tracking performance.

This article describes one solution for the problem of pursuit of objects moving on a plane by using a mobile robot and an active vision system. The solution deals with the interaction of different control systems using visual feedback and it is accomplished by the implementation of a visual gaze holding process interacting cooperatively with the control of the trajectory of a mobile robot. These two systems are integrated to follow a moving object at constant distance and orientation with respect to the mobile robot. The orientation and the position of the active vision system running a gaze holding process give the feedback signals to the control used to pursuit the target in real-time. The paper addresses the problems of visual fixation, visual smooth pursuit, navigation using visual feedback and compensation for system's movements. The algorithms for visual processing and control are described in the article. The mechanisms of cooperation between the different control and visual algorithms are also described. The final solution is a system able to operate at approximately human walking rates as the experimental results show in the paper.

We describe the real-time implementation of several visual behaviors in an active vision system. Issues related to the real-time implementation are discussed, concerning the modeling of the measurements made in the image. We also describe a methodology for calibrating the camera head, taking advantage of its degrees of freedom. These calibration parameters are used to evaluate the performance of the system. Another important issue of the operation of active vision binocular heads is their integration into more complex robotic systems. We claim that higher levels of autonomy and integration can be obtained by designing the system architecture based on the concept of purposive behavior. We show how this architecture can be used to implement a pursuit behavior using optical flow. Simultaneously vergence control can also be performed using the same visual processes.

In this paper we describe the implementation of real-time binocular gaze control. This implementation is performance by using a complex binocular active vision system. In addition real-time vergence is also implemented. Vergence and gaze control are implemented based on binocular optical flow. Vergence control is implemented based on optical flow disparity, using the horopter as a reference. A Kalman filter is used to filter out the signals and compensate for the delays of the system.

An active vision system must be able to perform reactive visual processes in real time. In this paper we discuss a number of issues related to the implementation of a real-time control architecture and describe the architecture we use with camera heads. Another important issue of the operation of active vision binocular heads is their integration into more complex robotic systems. Higher levels of autonomy and integration can be obtained by designing the control system architecture based on the concept of purposive behavior. At the lower levels we consider the vision as a sensor and integrate it in the control systems (both feedforward and servo loops); and several visual processes are implemented in parallel, computing relevant measures for the control process. At higher levels the architecture is modelled as a state transition system. Finally, we show how this architecture can be used to implement a pursuit behavior using optical flow. Simultaneously vergence control can also be performed using the same visual processes.

The main purpose of this paper is not to present a new camera calibration algorithm or a new pose estimation method, but just to analyse the stability of the camera pose estimation using the Haralick observation which is based on the fact that there is sufficient information on the 2D perspective projection of a rectangle of unknown size in 3D space to determine the three camera look angles. A brief description of the camera look angles computation is presented and a model for the expected stability of the camera look angles computation front noisy image data is described. A stability analysis as a function of the pose of the camera is also described, and the influence of the camera pose is confirmed with simulated data. As a result, camera pose view strategies for accurate camera orientation computation can be extracted from this pose view stability analysis.

This article describes one solution for the problem of pursuit by using a mobile robot and an active vision system. The solution deals with the interaction of different control systems using visual feedback and it is accomplished by the implementation of a visual gaze holding process interacting cooperatively with the control of the trajectory of a mobile robot. These two systems are integrated to follow a moving object at constant distance and orientation with respect to the mobile robot. The orientation and the position of the active vision system running a gaze holding process give the feedback signals to the control used to pursuit the target in real-time. The mechanisms of cooperation between the different control and visual algorithms are described. The final solution is a system able to operate at approximately human walking rates.

Addresses the geometric modelling aspects related with the use of a pair of video cameras mounted on a six degrees-of-freedom manipulator in an hand-eye configuration. The information obtained by these sensors shall be used to generate the depth map of the environment where the manipulator works. The procedure to determining the image formation parameters is known by camera calibration and a method to optimize these calibration parameters is presented. After presenting the calibration technique used, the authors develop a calibration matrix invariant to movement. The optimization process is applied to this matrix to obtain a better performance in the numerical data.>