We propose a framework for general multiple target tracking, where the input is a set of candidate regions in each frame, as obtained from a state of the art background learning, and the goal is to recover trajectories of targets over time from noisy observations. Due to occlusions by targets and static objects, noisy segmentation and false alarms, one foreground region may not correspond to one target faithfully. Therefore the one-to-one assumption used in most data association algorithm is not always satisfied. Our method overcomes the one-to-one assumption by formulating the visual tracking problem in terms of finding the best spatial and temporal association of observations, which maximizes the consistency of both motion and appearance of trajectories. To avoid enumerating all possible solutions, we take a data driven Markov chain Monte Carlo (DD-MCMC) approach to sample the solution space efficiently. The sampling is driven by an informed proposal scheme controlled by a joint probability model combining motion and appearance. To make sure the Markov chain to converge to a desired distribution, we propose an automatic approach to determine the parameters in the target distribution. Comparative experiments with quantitative evaluations are provided.

We present a method for detecting motion regions in video sequences observed by a moving camera, in the presence of strong parallax due to static 3D structures. The proposed method classifies each image pixel into planar background, parallax or motion regions by sequentially applying 2D planar homographies, the epipolar constraint and a novel geometric constraint, called "structure consistency constraint". The structure consistency constraint, as the main contribution of this paper, is derived from the relative camera poses among three frames and implemented within the "Plane+Parallax" framework. Unlike previous planar-parallax constraints, the proposed constraint does not require the reference plane to be constant across multiple views. It directly measures the inconsistency between the projective structures from the same point under camera motion and reference plane change. The structure consistency constraint is capable of detecting moving objects followed by a moving camera in the same direction, a so called degenerate configuration where the epipolar constraint fails. We demonstrate the effectiveness and robustness of our method with experimental results on real-world video sequences.

We present a method for 3-D shape reconstruction of retinal fundus from fluorescein images. Our method extracts the location of vessels’ bifurcation as a reliable feature for estimating the epipolar geometry using a plane-and-parallax approach. The proposed solution robustly estimates the fundamental matrix for nearly planar surfaces, such as the retinal fundus. We propose the use of mutual information criteria for accurate estimation of the disparity maps, where the matched Y-features are used for automatically estimating the bounds of the disparities range space. Our experimental results validate the proposed method on sets of difficult fluorescein image pairs.

Estimating human body poses in static images is important for many image understanding applications including semantic content extraction and image database query and retrieval. This problem is challenging due to the presence of clutter in the image, ambiguities in image observation, unknown human image boundary, and high-dimensional state space due to the complex articulated structure of the human body. Human pose estimation can be made more robust by integrating the detection of body components such as face and limbs, with the highly constrained structure of the articulated body. In this paper, a data-driven approach based on Markov chain Monte Carlo (DD-MCMC) is used, where component detection results generate state proposals for 3D pose estimation. To translate these observations into pose hypotheses, we introduce the use of "proposal maps," an efficient way of consolidating the evidence and generating 3D pose candidates during the MCMC search. Experimental results on a set of test images show that the method is able to estimate the human pose in static images of real scenes.

Our long-term objective is to create smart room technologies that are aware of the users presence and their behavior and can become an active, but not an intrusive, part of the interaction. In this work, we present a multimodal approach for estimating and tracking the location and identity of the participants including the active speaker. Our smart room design contains three user-monitoring systems: four CCD cameras, an omnidirectional camera and a 16 channel microphone array. The various sensory modalities are processed both individually and jointly and it is shown that the multimodal approach results in significantly improved performance in spatial localization, identification and speech activity detection of the participants.

In this study, we present a Y-feature extraction method for registering color and fluorescein angiograms of the retina. The registration of multimodal fluorescein imagery requires the identification of strong geometric features in the retinal images that are invariant across modalities and to temporal grey level variations due to the propagation of the dye in the vessels. The most informative features, invariant across the considered modalities, are the locations of vessels' ramification: the so-called Y-features. We propose a Y-feature extraction method based on the local classification of image gradient information and an articulated model. An appropriate cost function is proposed for fitting the model using a gradient based approach. The fitted Y-features are subsequently matched across the images for registering the color and fluorescein images. Experimental results obtained on a large database validate the proposed method.

We present a novel approach to detect and track independently moving regions in a 3D scene observed by a moving camera in the presence of strong parallax. Detected moving pixels are classified into independently moving regions or parallax regions by analyzing two geometric constraints: the commonly used epipolar constraint, and the structure consistency constraint. The second constraint is implemented within a "plane+parallax" framework and represented by a bilinear relationship which relates the image points to their relative depths. This newly derived relationship is related to trilinear tensor, but can be enforced into more than three frames. It does not assume a constant reference plane in the scene and therefore eliminates the need for manual selection of reference plane. Then, a robust parallax filtering scheme is proposed to accumulate the geometric constraint errors within a sliding window and estimate a likelihood map for pixel classification. The likelihood map is integrated into our tracking framework based on the spatio-temporal joint probability data association filter (JPDAF). This tracking approach infers the trajectory and bounding box of the moving objects by searching the optimal path with maximum joint probability within a fixed size of buffer. We demonstrate the performance of the proposed approach on real video sequences where parallax effects are significant.

In this paper, we introduce an interdisciplinary project, involving researchers from the fields of Physical Therapy, Computer Science, Psychology, Communication and Cell Neurobiology, to develop an integrated virtual reality, haptics and modern sensing technique system for post-stroke rehabilitation. The methodology to develop the system includes identification of movement pattern, development of simulated task and diagnostics. Each part of the methodology can be achieved through several sub-steps that are described in detail in this paper. The system is designed from Physical Therapy perspective that can address the motor rehabilitation needs of stroke patients. The system is implemented through stereoscopic displays, force feedback devices and modern sensing techniques that have game-like features and can capture accurate data for further analysis. Diagnostics and evaluation can be made through an Artificial Intelligence based model using collected data and clinical tests have been conducted.

This paper focuses on extracting 2D parametric motion regions from uncalibrated images. Our approach simultaneously infers and detects multiple image regions characterized by 2D motions, affine or homography transformations, from noisy initial matches. This approach is based on: (1) a parametric method to detect and extract 2D affine or homography motion regions; (2) the representation of the matching points in decoupled joint image spaces; (3) the characterization of the property associated with affine transformation in the defined spaces; (4) a non-iterative process to extract multiple 2D motions simultaneously based on tensor-voting; (5) local affine to global homography estimation; (6) region refinement based on a hybrid property: motion and color homogeneity. The robustness of the approach is demonstrated with several results.

We describe a system that extracts layers from a video sequence based on a method estimating and stabilizing the dominant motion. Our approach performs pair-wise registration of images parameterized by 2D affine or projective transformation as the first step. This pair-wise registration is based on: (1) hierarchical parameter estimation and refinement; (2) feature-matching; (3) FFT (fast Fourier transformation)-based global matching; and (4) RANSAC-based parameter estimation. A global registration is then performed among overlapping multi-frames. The global registration is based on inferring the topology (i.e. spatio-temporal relation among frames) and the characterization of the likelihood within overlapping areas. As the last step, we extract the background layer that consists of static components of the video stream after motion estimation. Our background extraction is based on deriving a pixel-wise color distribution in time and provides a basis for a compact description of a video. The presented approach is illustrated by a set of challenging examples.

This paper presents a new approach for continuous tracking of moving objects observed by multiple, heterogeneous cameras. Our approach simultaneously processes video streams from stationary and pan-tilt-zoom cameras. The detection of moving objects from moving camera streams is performed by defining an adaptive background model that takes into account the camera motion approximated by an affine transformation. We address the tracking problem by separately modeling motion and appearance of the moving objects using two probabilistic models. For the appearance model, multiple color distribution components are proposed for ensuring a more detailed description of the object being tracked. The motion model is obtained using a Kalman filter (KF) process, which predicts the position of the moving object. The tracking is performed by the maximization of a joint probability model. The novelty of our approach consists in modeling the multiple trajectories observed by the moving and stationary cameras in the same KF framework. It allows deriving a more accurate motion measurement for objects simultaneously viewed by the two cameras and an automatic handling of occlusions, errors in the detection and camera handoff. We demonstrate the performances of the system on several video surveillance sequences.

We present a system which takes as input a video stream obtained from an airborne moving platform and produces an analysis of the behavior of the moving objects in the scene. To achieve this functionality, our system relies on two modular blocks. The first one detects and tracks moving regions in the sequence. It uses a set of features at multiple scales to stabilize the image sequence, that is, to compensate for the motion of the observer, then extracts regions with residual motion and uses an attribute graph representation to infer their trajectories. The second module takes as input these trajectories, together with user-provided information in the form of geospatial context and goal context to instantiate likely scenarios. We present details of the system, together with results on a number of real video sequences and also provide a quantitative analysis of the results.

The general problem of surface matching is considered in this study. The process described in this work hinges on a geodesic distance equation for a family of surfaces embedded in the graph of a cost function. The cost function represents the geometrical matching criterion between the two 3D surfaces. This graph is a hypersurface in 4-dimensional space, and the theory presented herein is a generalization of the geodesic curve evolution method introduced by R. Kimmel et al. (1995). It also generalizes the 2D matching process developed in Cohen and Herlin (1998). An Eulerian level-set formulation of the geodesic surface evolution is also used, leading to a numerical scheme for solving partial differential equations originating from hyperbolic conservation laws, which has proven to be very robust and stable. The method is applied on examples showing both small and large deformations, and arbitrary topological changes.

Tackles the general problem of surface tracking. The authors present a new method based on the computation of geodesic distance maps between a source and a destination surface. The method used in this case is a generalization to the case of surfaces of a 2D matching process introduced by Cohen and Herlin (1998). It relies on a level-set formulation, thus providing stable and robust resolution schemes. The obtained matching method accommodates both small and large deformations. Actually, the matching constraints are explicitly expressed by the definition of a hypersurface. The connecting paths between the two surfaces lie on this hypersurface. The authors construct a hypersurface that allows the matching process to preserve geometric similarities in the case of small deformations, and loosen these geometric constraints when large deformations occur. The applicative framework is the study of the Earth's crustal evolution to characterize the deformation of digital elevation models.

We describe a graph-based global registration method for creating 2D mosaic images. When multi-frames overlap in space, global registration is necessary to minimize the accumulated registration errors. We use a graph to represent the temporal and spatial connectivity and show that global registration can be obtained through the search for an optimal path in the constructed graph. The definition of an adequate objective function characterizing the global registration provides a direct manipulation of the graph. The framework presented here allows the automatic construction of the graph, and the construction of a consistent mosaic from a collection of frames using projective transformations.

We address the problem of detection and tracking of moving objects in a video stream obtained from a moving airborne platform. The proposed method relies on a graph representation of moving objects which allows to derive and maintain a dynamic template of each moving object by enforcing their temporal coherence. This inferred template along with the graph representation used in our approach allows us to characterize objects trajectories as an optimal path in a graph. The proposed tracker allows to deal with partial occlusions, stop and go motion in very challenging situations. We demonstrate results on a number of different real sequences. We then define an evaluation methodology to quantify our results and show how tracking overcome detection errors.

We present a method for matching curves which accommodates large and small deformation. The method preserves geometric similarities in the case of small deformation, and loosens these geometric constraints when large deformations occur. The approach is based on the computation of a set of geodesic paths connecting the curves. These two curves are defined as a source area and a destination area which can have an arbitrary number of connected components and different topologies. The applicative framework of the presented method is the study of the crustal deformation from a set of iso-elevation curves. An experiment with real curves demonstrates that the approach can be successfully applied to characterize deformation of Digital Elevation Models.

This paper is concerned with the problem of tracking clouds structures like vortices in meteorological images. For this purpose we characterize the deformation between two successive occurrences, by matching their two boundary curves. Our approach is based on the computation of the set of paths connecting the two curves to be matched. It minimizes a cost function which measures the local similarity of the two curves. These matching paths are obtained as geodesic curves on this cost surface. Moreover our method allows to consider complex curves of arbitrary topology since these curves are represented through an implicit function rather than through a parameterization. Experimental results are given to illustrate the properties of the method in processing synthetic and then meteorologic remotely-sensed data.

To analyze SAR interferometric data, an unwrapping process must be first performed. Most so far proposed solutions use either local or global methods. In this paper we propose a mixed method to solve this problem, based on a 3-step iterative process: a local unwrapping is first performed and then improved through a Markovian segmentation, false unwrapped residues are finally detected and corrected. A deterministic algorithm is used for the relaxation process; wrong unwrapped areas are corrected by a backtracking mechanism. Some results obtained by this approach, which presents a lower computational cost compared to complex stochastic unwrapping methods, are also presented.

Motion study in computer vision highly depends on the type of imagery and on the kind of the studied objects. A popular approach of non-rigid motion supposes that the objects are continuously and locally deformed. Local features, such as curvature extrema, are then often used. However, these features can not always be accurately computed, and motion may involve large deformation of the object between two consecutive temporal occurrences. In these cases, a real need exists for an approach that does not rely on local features. That study of motion requires additional information. We introduce a geometrical evolution model that enables one to generate a surface interpolating successive contours of the object during its temporal evolution. This geometrical model may be viewed as a simplification of a true physical model of motion. This approach is particularly well suited to remote sensed data: structures of interest do not have a well defined shape, and the temporal resolution may be poor, involving large deformation. The model is successfully applied to vortex tracking on sea color and meteorologic images.

This paper is concerned with three main problems of image processing occuring on temporal sequences of satellite oceanographic images: approximate localization of the interesting structures on the images; segmentation or determination of the boundary of the structures; and temporal tracking of these boundaries to illustrate their evolution. In this paper, application is done on vortices and results are displayed all over the paper. For this purpose, we propose a new method for optical flow computation and interpretation and a geometric modelling of the structures. Oceanographic images obtained from environmental satellite platforms present a new challenge in computer science. The huge amount of data collected each day and the need for characterizing some specific structures on these images for oceanographic monitoring justify our approach for the detection and tracking of vortices on oceanographic images.

We describe a new method for computing displacement field for sequences of temporal images. It involves minimizing the energy defined on the space of correspondence functions. This energy is divided in two terms: one which matches the contour points and particularly high curvature points, and one which regularizes the field. We introduce an adaptive mesh the resolution of which depends on the presence of edges and/or points of high curvature, and we show how to use it to reduce the computational time for our method. We present experimental results on medical images which prove the validity of the approach and the accuracy of the computed displacement fields.

We present in this paper some improvements to the hyperquadric model allowing an efficient representation of the shape through a numerical parameterization and an energy minimization approach for data fitting. This last feature gives an accurate location of the image edge points independently of the implicit description of the shape. The advantage of our model is that it describes global shape properties through a unique implicit equation yielding a representation of the shape by means of its parameters, independently of the chosen numerical resolution.

The use of energy-minimizing curves, known as "snakes" to extract features of interest in images has been introduced by Kass, Witkin and Terzopoulos (1987). A balloon model was introduced by Cohen (1991) as a way to generalize and solve some of the problems encountered with the original method. A 3-D generalization of the balloon model as a 3-D deformable surface, which evolves in 3-D images, is presented. It is deformed under the action of internal and external forces attracting the surface toward detected edgels by means of an attraction potential. We also show properties of energy-minimizing surfaces concerning their relationship with 3-D edge points. To solve the minimization problem for a surface, two simplified approaches are shown first, defining a 3-D surface as a series of 2-D planar curves. Then, after comparing finite-element method and finite-difference method in the 2-D problem, we solve the 3-D model using the finite-element method yielding greater stability and faster convergence. This model is applied for segmenting magnetic resonance images.>

A 3-D generalization of the balloon model as a 3-D deformable surface, which evolves in 3-D images, is presented. It is deformed under the action of internal and external forces attracting the surface toward detected edge elements by means of an attraction potential. To solve the minimization problem for a surface, two simplified approaches are shown, defining a 3-D surface as a series of 2-D planar curves. Then the 3-D model is solved using the finite-element method, yielding greater stability and faster convergence. This model has been used to segment magnetic resonance images.>

A 3D deformable model is introduced which evolves in true 3D images, under the action of internal forces (describing some elasticity properties of the surface), and external forces attracting the surface toward some detected edges. The formalism leads to the minimization of an energy which is expressed as a functional. The authors use a variational approach and a finite-element method to express the surface in a discrete basis of continuous functions. This leads to a reduced computational complexity and a better numerical stability. The power of the approach to segment 3D images is demonstrated by a set of experimental results on various complex medical 3D images.>

The authors present a model of deformation which solves some of the problems encountered with the original method such as instability and initial data while reducing the computational complexity. This model makes the curve, behave like a balloon which is inflated by an additional force. The initial curve need no longer be close to the solution to converge. The external forces that push the curve to the edges are modified to give more stable results. The system is solved using a conform finite element method in the minimization process. The evolution to the equilibrium presents less oscillations, convergence is obtained faster, and the final results are more accurate. This model is applied for segmenting ultrasound and magnetic resonance images. The authors have also made a first stage to 3-D object reconstruction. by tracking the extracted contour on a series of successive cross sections.>