We present an end-to-end system for detecting and clustering faces by identity in full-length movies. Unlike works that start with a predefined set of detected faces, we consider the end-to-end problem of detection and clustering together. We make three separate contributions. First, we combine a state-of-the-art face detector with a generic tracker to extract high quality face tracklets. We then introduce a novel clustering method, motivated by the classic graph theory results of Erdös and Rényi. It is based on the observations that large clusters can be fully connected by joining just a small fraction of their point pairs, while just a single connection between two different people can lead to poor clustering results. This suggests clustering using a verification system with very few false positives but perhaps moderate recall. We introduce a novel verification method, rank-1 counts verification, that has this property, and use it in a link-based clustering scheme. Finally, we define a novel end-to-end detection and clustering evaluation metric allowing us to assess the accuracy of the entire end-to-end system. We present state-of-the-art results on multiple video data sets and also on standard face databases.

Geotagged imagery, from satellite, aerial, and ground-level cameras, provides a rich record of how the appearance of scenes and objects differ across the globe. Modern web- based mapping software makes it easy to see how different places around the world look, both from satellite and ground-level views. Unfortunately, interfaces for exploring how the appearance of objects depend on geographic location are quite limited. In this work, we focus on a particularly common object, the human face, and propose learning generative models that relate facial appearance and geographic location. We train these models using a novel dataset of geotagged face imagery we constructed for this task. We present qualitative and quantitative results that demonstrate that these models capture meaningful trends in appearance. We also describe a framework for constructing a web-based visualization that captures the geospatial distribution of human facial appearance.

Live wide-area persistent surveillance (WAPS) systems must provide effective multi-target tracking on downlinked video streams in real-time. This paper presents the first published aerial tracking system that is documented to process over 100 megapixels per second. The implementation addresses the challenges with the mosaicked, low-resolution, grayscale NITF imagery provided by most currently fielded WAPS platforms and the flexible computation architecture required to provide real-time performance. This paper also provides ground-truth for repeatable evaluation of wide-area persistent surveillance on a 2009 dataset collected by AFRL [1] that is available to the public as well as a quantitative analysis of this real-time implementation. To our knowledge, this is the only publication that (1) provides details of a real-time implementation for detection and tracking in (2) mosaicked, composed imagery from a fielded WAPS sensor, and (3) provides annotation data and quantitative analysis for repeatable WAPS tracking experimentation in the computer vision community.

Computer vision is the most promising technology for automated, passive tracking of multiple objects over large areas. Effective inter-camera and intra-camera visual tracking can enable information about a vehicle or a pedestrian to be integrated from various sources. Unfortunately, tracking objects across multiple non-overlapping cameras requires reliable comparison of the objects ’ appearance under widely-varying view angles and resolutions. Fortunately, in most cases, an object of a particular type entering a scene at a particular position and direction will tend to be in a very similar pose. This paper introduces Translation Templates (TTs). TTs exploit this regularity to learn a color-based matching metric for images from a pair of tracking source and sink points, without prior knowledge of object type or object pose. This model benefits from histogram-based aggregation while still preserving spatial relationships between the two images. The model can be learned directly from data and used to compare arbitrary types of objects observed from extremely different viewpoints, as long as the relationship between the viewpoints is preserved. This paper describes TTs, describes a method for efficient computation and for visualization of TTs, and presents experimental results from both an indoor pedestrian data set and an outdoor vehicle data set. 1

This paper introduces a method for robustly estimating a planar tracking correspondence model (TCM) for a large camera network directly from tracking data and for employing said model to reliably track objects through multiple cameras. By exploiting the unique characteristics of tracking data, our method can reliably estimate a planar TCM in large environments covered by many cameras. It is robust to scenes with multiple simultaneously moving objects and limited visual overlap between the cameras. Our method introduces the capability of automatic calibration of large camera networks in which the topology of camera overlap is unknown and in which all cameras do not necessarily overlap. Quantitative results are shown for a five camera network in which the topology is not specified.

We discuss building complex classifiers from a single labeled example and vast number of unlabeled observation sets, each derived from observation of a single process or object. When data can be measured by observation, it is often plentiful and it is often possible to make more than one observation of the state of a process or object. We discuss how to exploit the variability across such sets of observations of the same object to estimate class labels for unlabeled examples given a minimal number of labeled examples. In contrast to similar semisupervised classification procedures that define the likelihood that two observations share a label as a function of the embedded distance between the two observations, this method uses the Naive Bayes estimate of how often the two observations did result from the same observed process. Exploiting this additional source of information in an iterative estimation procedure can generalize complex classification models from single labeled observations. Some examples involving classification of tracked objects in a low-dimensional feature space given thousands of unlabeled observation sets are used to illustrate the effectiveness of this method.

This paper investigates applications of a new representation for images, the similarity template. A similarity template is a probabilistic representation of the similarity of pixels in an image patch. It has application to detection of a class of objects, because it is reasonably invariant to the color of a particular object. Further, it enables the decomposition of a class of objects into component parts over which robust statistics of color can be approximated. These regions can be used to create a factored color model that is useful for recognition. Detection results are shown on a system that learns to detect a class of objects (pedestrians) in static scenes based on examples of the object provided automatically by a tracking system. Applications of the factored color model to image indexing and anomaly detection are pursued on a database of images of pedestrians.

Recent work has shown impressive transform-invariant modeling and clustering for sets of images of objects with similar appearance. We seek to expand these capabilities to sets of images of an object class that show considerable variation across individual instances (e.g. pedestrian images) using a representation based on pixel-wise similarities, similarity templates. Because of its invariance to the colors of particular components of an object, this representation en- ables detection of instances of an object class and enables alignment of those instances. Further, this model implicitly represents the re- gions of color regularity in the class-speci(cid:12)c image set enabling a decomposition of that object class into component regions.

Our goal is to develop a visual monitoring system that passively observes moving objects in a site and learns patterns of activity from those observations. For extended sites, the system will require multiple cameras. Thus, key elements of the system are motion tracking, camera coordination, activity classification, and event detection. In this paper, we focus on motion tracking and show how one can use observed motion to learn patterns of activity in a site. Motion segmentation is based on an adaptive background subtraction method that models each pixel as a mixture of Gaussians and uses an online approximation to update the model. The Gaussian distributions are then evaluated to determine which are most likely to result from a background process. This yields a stable, real-time outdoor tracker that reliably deals with lighting changes, repetitive motions from clutter, and long-term scene changes. While a tracking system is unaware of the identity of any object it tracks, the identity remains the same for the entire tracking sequence. Our system leverages this information by accumulating joint co-occurrences of the representations within a sequence. These joint co-occurrence statistics are then used to create a hierarchical binary-tree classification of the representations. This method is useful for classifying sequences, as well as individual instances of activities in a site.

While a tracking system is unaware of the identity of any object it tracks, the identity remains the same for the entire tracking sequence. Our system leverages this information by using accumulated joint cooccurrences of the representations within the sequence to create a hierarchical binary-tree classifier of the representations. This classifier is useful to classify sequences as well as individual instances. We illustrate the use of this method on two separate representations the tracked object's position, movement, and size; and the tracked object's binary motion silhouettes.

A common method for real-time segmentation of moving regions in image sequences involves "background subtraction", or thresholding the error between an estimate of the image without moving objects and the current image. The numerous approaches to this problem differ in the type of background model used and the procedure used to update the model. This paper discusses modeling each pixel as a mixture of Gaussians and using an on-line approximation to update the model. The Gaussian, distributions of the adaptive mixture model are then evaluated to determine which are most likely to result from a background process. Each pixel is classified based on whether the Gaussian distribution which represents it most effectively is considered part of the background model. This results in a stable, real-time outdoor tracker which reliably deals with lighting changes, repetitive motions from clutter, and long-term scene changes. This system has been run almost continuously for 16 months, 24 hours a day, through rain and snow.

We describe a vision system that monitors activity in a site over extended periods of time. The system uses a distributed set of sensors to cover the site, and an adaptive tracker detects multiple moving objects in the sensors. Our hypothesis is that motion tracking is sufficient to support a range of computations about site activities. We demonstrate using the tracked motion data to calibrate the distributed sensors, to construct rough site models, to classify detected objects, to learn common patterns of activity for different object classes, and to detect unusual activities.