In this paper, we introduce a new method for background subtraction for freely moving cameras and arbitrary scene geometry. Instead of relying on frame-to-frame estimation, we simultaneously estimate the epipolar geometries induced by a moving camera in a temporally consistent manner for a number of frames using the temporal fundamental matrix (TFM). The TFM is robustly estimated from the track lets generated by dense point tracking and used to compute the probability of each track let belonging to the background. In order to ensure the color, spatial, and temporal consistencies of track let labeling, we minimize spatiotemporal labeling cost in locality of track lets. Extensive experiments with challenging videos show that the proposed method is comparable and, in most cases, outperforms the state-of-the-art.

We propose a novel approach to associate objects across multiple PTZ cameras that can be used to perform camera handoff in wide-area surveillance scenarios. While previous approaches relied on geometric, appearance, or correlation-based information for establishing correspondences between static cameras, they each have well-known limitations and are not extendable to wide-area settings with PTZ cameras. In our approach, the slave camera only passively follows the target (by loose registration with the master) and bootstraps itself from its own incoming imagery, thus effectively circumventing the problems faced by previous approaches and avoiding the need to perform any model transfer. Towards this goal, we also propose a novel Multiple Instance Learning (MIL) formulation for the problem based on the logistic softmax function of covariance-based region features within a MAP estimation framework. We demonstrate our approach with multiple PTZ camera sequences in typical outdoor surveillance settings and show a comparison with state-of-the-art approaches.

The success of any graph-based clustering algorithm depends heavily on the quality of the similarity matrix being clustered, which is itself highly dependent on point-wise scaling parameters. We propose a novel technique for finding point-wise scaling parameters based on Ripley's K-function which enables data clustering at different density scales within the same dataset. Additionally, we provide a method for enhancing the spatial similarity matrix by including a density metric between neighborhoods. We show how our proposed methods for building similarity matrices can improve the results attained by traditional approaches for several well known clustering algorithms on a variety of datasets.

We present a novel approach for discovering directed intention-driven pedestrian activities across large urban areas. The proposed approach is based on a mutual information co-clustering technique that simultaneously clusters trajectory start locations in the scene which have similar distributions across stop locations and vice-versa. The clustering assignments are obtained by minimizing the loss of mutual information between a trajectory start-stop association matrix and a compressed co-clustered matrix, after which the scene activities are inferred from the compressed matrix. We demonstrate our approach using a dataset of long duration trajectories from multiple PTZ cameras covering a large area and show improved results over two other popular trajectory clustering and entry-exit learning approaches.

We present a novel framework for extracting "pathlets" from tracking data. A pathlet is defined as a motion region that contains tracks having the same origin and destination in the scene and that are temporally correlated. The proposed method requires only weak tracking data (multiple fragmented tracks per target). We employ a probabilistic state space representation to construct a Markovian transition model and estimate the scene entry/exit locations. The resulting model is treated as a set of vertices in a graph and a similarity matrix is built which describes broader nonlocal relationships between states. A Spectral Clustering approach is then used to automatically extract the pathlets of the scene. We present experimental results from scenes of varying difficulty and compare against other approaches.

We propose a novel registration framework to map the field-of-coverage of pan-tilt cameras to a GIS (Geographic Information System) planar coordinate system. The camera's field-of-coverage is obtained by building a spherical panorama using an efficient active camera model. The pan-tilt orientation rays from the panoramic image are projected onto a GIS orthophoto ground plane and registered using a transformation matrix. The parameters of the transformation are learned in linear time using least squares. The proposed model is experimentally evaluated by registering panoramas from multiple cameras with an orthophoto and then overlaying them with GIS metadata ground truth to validate the accuracy of registration. We also demonstrate the applicability of such a GIS-based framework to a multi-camera, master-slave active tracking system.

We present a practical framework for detecting and modeling 3D static occlusions for wide-baseline, multi-camera scenarios where the number of cameras is small. The framework consists of an iterative learning procedure where at each frame the occlusion model is used to solve the voxel occupancy problem, and this solution is then used to update the occlusion model. Along with this iterative procedure, there are two contributions of the proposed work: (1) a novel energy function (which can be minimized via graph cuts) specifically designed for use in this procedure, and (2) an application that incorporates our probabilistic occlusion model into a 3D tracking system. Both qualitative and quantitative results of the proposed algorithm and its incorporation with a 3D tracker are presented for support.

We present an online, recursive filtering technique to model linear dynamical systems that operate on the state space of symmetric positive definite matrices (tensors) that lie on a Riemannian manifold. The proposed approach describes a predict-and-update computational paradigm, similar to a vector Kalman filter, to estimate the optimal tensor state. We adapt the original Kalman filtering algorithm to appropriately propagate the state over time and assimilate observations, while conforming to the geometry of the manifold. We validate our algorithm with synthetic data experiments and demonstrate its application to visual object tracking using covariance features.

We propose an efficient active camera model to map image coordinates to the camera's pan-tilt orientations in constant time. The model is based on the elliptical locus of the projections of a fixed point on the original image plane of a moving camera. The parametric location of this point along the ellipse defines the change in camera orientation. This model does not require any knowledge of camera parameters other than the focal length. Using synthetic and real data, we show the accuracy of the model by generating seamless spherical panoramas from a set of images and demonstrate the applicability of the model with a real-time active tracking application.

We present a method for the simultaneous detection and segmentation of people from static images. The proposed technique requires no manual segmentation during training, and exploits top-down and bottom-up processing within a single framework for both object localization and 2D shape estimation. First, the coarse shape of the object is learned from a simple training phase utilizing low-level edge features. Motivated by the observation that most object categories have regular shapes and closed boundaries, relations between these features are then exploited to derive mid-level cues, such as continuity and closure. A novel Markov random field defined on the edge features is presented that integrates the coarse shape information with our expectation that objects are likely to have boundaries that are regular and closed. The algorithm is evaluated on pedestrian datasets of varying difficulty, including a wide range of camera viewpoints, and person orientations. Quantitative results are presented for person detection and segmentation, demonstrating the effectiveness of the proposed technique to simultaneously address both these tasks.

We present a computer vision system for robust object tracking in 3D by combining evidence from multiple calibrated cameras. This kernel-based 3D tracker is automatically bootstrapped by constructing 3D point clouds. These points clouds are then clustered and used to initialize the trackers and validate their performance. The framework describes a complete tracking system that fuses appearance features from all available camera sensors and is capable of automatic initialization and drift detection. Its elegance resides in its inherent ability to handle problems encountered by various 2D trackers, including scale selection, occlusion, view-dependence, and correspondence across views. Tracking results for an indoor smart room and a multi-camera outdoor surveillance scenario are presented. We demonstrate the effectiveness of this unified approach by comparing its performance to a baseline 3D tracker that fuses results of independent 2D trackers, as well as comparing the re-initialization results to known ground truth.

We present a unified method for simultaneously acquiring both the location and the silhouette shape of people in outdoor scenes. The proposed algorithm integrates top-down and bottom-up processes in a balanced manner, employing both appearance and motion cues at different perceptual levels. Without requiring manually segmented training data, the algorithm employs a simple top-down procedure to capture the high-level cue of object familiarity. Motivated by regularities in the shape and motion characteristics of humans, interactions among low-level contour features are exploited to extract mid-level perceptual cues such as smooth continuation, common fate, and closure. A Markov random field formulation is presented that effectively combines the various cues from the top-down and bottom-up processes. The algorithm is extensively evaluated on static and moving pedestrian datasets for both detection and segmentation.

We address the limited automatic scanning functionality of standard PTZ camera systems. We present an adaptive, scene-specific model using standard PTZ camera hardware. The adaptive model is constructed automatically by detecting human activity in motion history images (MHIs) using an iterative candidacy-classification-reduction process. The target motion is quantified and employed in the construction of a global activity map, which in turn is used to direct or navigate the camera

We present a method for the simultaneous detection and segmentation of objects from static images. We employ low-level contour features that enable us to learn the coarse object shape using a simple training phase requiring no manual segmentation. Based on the observation that most interesting objects (e.g., people) have regular and closed boundaries, we exploit relations between these features to extract midlevel cues, such as continuity and closure. For segmentation, we employ a Markov random field that combines these cues with information learned from training. The algorithm is evaluated for extracting person silhouettes from surveillance images, and quantitative results are presented

We present an action recognition method based on the concept of reliable inference. Our approach is formulated in a probabilistic framework using posterior class ratios to verify the saliency of an input before committing to any action classification. The framework is evaluated in the context of walking, running, and standing at multiple viewpoints and compared to ML and MAP approaches. Results examining individual silhouette images with the framework demonstrate that these actions can be reliably discriminated while discounting confusing images.

We present a computational framework capable of labeling the effort of an action corresponding to the perceived level of exertion by the performer (low - high). The approach initially factorizes examples (at different efforts) of an action into its three-mode principal components to reduce the dimensionality. Then a learning phase is introduced to compute expressive-feature weights to adjust the model's estimation of effort to conform to given perceptual labels for the examples. Experiments are demonstrated recognizing the efforts of a person carrying bags of different weight and for multiple people walking at different paces.

A view-based approach to the representation and recognition of human movement is presented. The basis of the representation is a temporal template-a static vector-image where the vector value at each point is a function of the motion properties at the corresponding spatial location in an image sequence. Using aerobics exercises as a test domain, we explore the representational power of a simple, two component version of the templates: The first value is a binary value indicating the presence of motion and the second value is a function of the recency of motion in a sequence. We then develop a recognition method matching temporal templates against stored instances of views of known actions. The method automatically performs temporal segmentation, is invariant to linear changes in speed, and runs in real-time on standard platforms.

Many communicative behaviors in the animal kingdom consist of performing and recognizing specialized patterns of oscillatory motion. Here we present an approach to the representation and recognition of these oscillatory motions based on the categorical organization of a simple sinusoidal model having very specific and limited parameter values. This characterization is used to specify the types and layout of computation for recognizing the patterns. Results of the method are demonstrated with real oscillatory motions showing the viability of a structured categorical framework.

This paper uses a simple method for representing motion in successively layered silhouettes that directly encode system time termed the timed Motion History Image (tMHI). This representation can be used to both (a) determine the current pose of the object and to (b) segment and measure the motions induced by the object in a video scene. These segmented regions are not "motion blobs", but instead motion regions naturally connected to the moving parts of the object of interest. This method may be used as a very general gesture recognition "toolbox". We use it to recognize waving and overhead clapping motions to control a music synthesis program.

This paper presents a glove-free method for tracking hand movements using a set of 3-D models. In this approach, the hand is represented by five cylindrical models which are fit to the third phalangeal segments of the fingers. Six 3-D motion parameters for each model are calculated that correspond to the movement of the fingertips in the image plane. Trajectories of the moving models are then established to show the 3-D nature of the hand motion.

A new view-based approach to the representation and recognition of action is presented. The basis of the representation is a temporal template-a static vector-image where the vector value at each point is a function of the motion properties at the corresponding spatial location in an image sequence. Using 18 aerobics exercises as a test domain, we explore the representational power of a simple, two component version of the templates: the first value is a binary value indicating the presence of motion, and the second value is a function of the recency of motion in a sequence. We then develop a recognition method which matches these temporal templates against stored instances of views of known actions. The method automatically performs temporal segmentation, is invariant to linear changes in speed, and runs in real-time on a standard platform. We recently incorporated this technique into the KIDSROOM: an interactive, narrative play-space for children.

A real-time tracking algorithm that uses contextual information is described. The method is capable of simultaneously tracking multiple, non-rigid objects when erratic movement and object collisions are common. A closed-world assumption is used to adaptively select and weight image features used for correspondence. Results of algorithm testing and the limitations of the method are discussed. The algorithm has been used to track children in an interactive, narrative playspace.

A new view-based approach to the representation of action is presented. Our underlying representations are view-based descriptions of the coarse image motion associated with viewing given actions from particular directions. Using these descriptions, we propose an appearance-based action-recognition strategy comprised of two stages: 1) a motion energy image (MEI) is computed that grossly describes the spatial distribution of motion energy for a given view of a given action, and the input MEI is matched against stored models which span the range of views of known actions; 2) any models that plausibly match the input are tested for a coarse, categorical agreement between a stored motion model of the action and a parametrization of the input motion. Using a "sitting" action as an example, and using a manually placed stick model, we develop a representation and verification technique that collapses the temporal variations of the motion parameters into a single, low-order vector.

A new view based approach to the representation and recognition of action is presented. The basis of the representation is a motion history image (MHI)-a static image where intensity is a function of the recency of motion in a sequence. We develop a recognition method which uses both binary and scalar valued versions of the MHI as temporal templates to match against stored instances of actions. The method automatically performs temporal segmentation, as invariant to linear changes in speed, and runs in real time on a standard platform. The applications we have begun to develop include simple room monitoring and an interactive game.