We present a method for automatically aligning words to image regions that integrates specific object classifiers (e.g., "car" detectors) with weak models based on appearance features. Previous strategies have largely focused on the latter, and thus have not exploited progress on object category recognition. Hence, we augment region labeling with object detection, which simplifies the problem by reliably identifying a subset of the labels, and thereby reducing correspondence ambiguity overall. Comprehensive testing on the SAIAPR TC dataset shows that principled integration of object detection improves the region labeling task.

We address the problem of interactive search for a target of interest in surveillance imagery. Our solution consists of iteratively learning a distance metric for retrieval, based on user feedback. The approach employs (retrieval) rank based constraints and convex optimization to efficiently learn the distance metric. The algorithm uses both user labeled and unlabeled examples in the learning process. The method is fast enough for a new metric to be learned interactively for each target query. In order to reduce the burden on the user, a model-independent active learning method is used to select key examples, for response solicitation. This leads to a significant reduction in the number of user-interactions required for retrieving the target of interest. The proposed method is evaluated on challenging pedestrian and vehicle data sets, and compares favorably to the state of the art in target re-acquisition algorithms.

This paper proposes a new solution to geo-register the nearly feature-less maritime video feeds. We detect the horizon using sizable or uniformly moving vessels, and estimate the vertical apex using water reflections of the street lamps. The computed horizon and apex provide a metric rectification that removes the affine distortions and reduces the searching space for geo-registration. Geo-registration is obtained by searching the best orientation where the estimated water masks on satellite images and camera views are matched. The proposed solution has the following contributions: first, water and coastlines are used as features for registration between horizontally looking maritime views and satellite images. Second, water reflections are proposed to estimate the vertical vanishing point. Third, we give algorithms for the detection of water areas in both satellite images and camera views. Experimental results and applications on cross camera tracking are demonstrated. We also discuss several observations, as well as limitations of the proposed approach.

A review of commercial and research applications of automated video analysis in the Americas, EMEA and Asia a) offers a snapshot of the state-of-the-art, b) reveals some of the forces that determine future solutions, and c) suggests a price tag for this new technology. Despite a rapid increase in (video) data and the fact that this deluge cannot be managed with human resources alone people are expensive and ineffective customers are reluctant to accept help from a technology that occasionally makes really dumb mistakes. Will this reluctance vanish over time without radical improvements in the technology? Will the technology improve and prevent these mistakes in the future? If so, how? Are there applications where the current mistakes are acceptable? What is the right price tag for the technology?

Recent advances in computer vision and artificial intelligence algorithms have allowed automatic extraction of metadata from video. This metadata can be represented by using the RDF/OWL ontology which can encode scene objects and their relationships in an unambiguous and well-formed manner. The encoded data can be queried using SPARQL. However, SPARQL has a steep learning curve and cannot be directly utilized by a general user for video content search. In this paper, we propose a method to bridge this gap by automatically translating user provided natural language query into an ontology-based SPARQL query for semantic video search. The proposed method consists of three major steps. First, semantically labeled training corpus of natural language query sentences is used for learning the Semantic Stochastic Context Free Grammar (SSCFG). Second, given a user provided natural language query sentence, we use the Earley-Stolcke parsing algorithm to determine the maximum likelihood semantic parsing of the query sentence. This parsing infers the semantic meaning for each word in the query sentence from which the SPARQL query is constructed. Third, the SPARQL query is executed to retrieve relevant video segments from the RDF-OWL video content database. The method is evaluated by running natural language queries on surveillance videos from maritime and land-based domains, though the framework itself is general and extensible to search videos from other domains.

This paper presents a multi-frame data association algorithm for tracking multiple targets in video sequences. Multi-frame data association involves finding the most probable correspondences between target tracks and measurements (collected over multiple time instances) as well as handling the common tracking problems such as, track initiations and terminations, occlusions, and noisy detections. The problem is known to be NP-Hard for more than two frames. A rank constrained continuous formulation of the problem is presented that can be efficiently solved using nonlinear optimization methods. It is shown that the global and local extrema of the continuous problem respectively coincide with the maximum and the maximal solutions of the discrete counterpart. A scanning window based tracking algorithm is developed using the formulation that performs well under noisy conditions with frequent occlusions and multiple track initiations and terminations. The above claims are supported by experiments and quantitative evaluations using both synthetic and real data under different operating conditions.

We propose an automatic method to geo-register maritime video feeds to satellite images. The method first detects horizon during the day time and apex during the night time for similarity rectification. It then finds water pixels on both satellite images and camera views. Finally, geo-registration is obtained by searching the best orientation where the water masks are matched. The current state-art-of-state video surveillance system benefits with this method by providing situational awareness on maps with physical data including object size, location, and velocity in the world coordinates. Experimental results and applications such as cross camera tracking are demonstrated.

We propose a new method for warping high-resolution images to efficiently track objects on the ground plane in real time. Recently, the emergence of high resolution video cameras (> 5 megapixels) has enabled surveillance over a much larger area using only a single camera. However, real-time processing of high resolution video for automatic detection and tracking of multiple targets is a challenge. When the surveillance camera covers greater depth of ground regions, due to perspective effect, the image size of a target varies significantly depending on the distance between the camera and the target. In this study, we propose a framework to transform high resolution images into warped images using a plane homography to make the target size uniform regardless of the position. The method not only reduces the number of pixels to be processed for speed-up, but also improves the tracking performance. We provide experimental results on object tracking in high-resolution maritime videos to demonstrate the validity of our method.

This paper presents an unsupervised data driven scheme to automatically estimate the relative topology of overlapping cameras in a large visual sensor network. The proposed method learns the camera topology by employing the statistics of co-occurring observations (of moving targets) in each sensor. Since target observation data is typically very noisy in realistic scenarios, an efficient two step method is used for robust estimation of the planar homography between camera views. In the first step, modes in the co-occurrence data are learned using meanshift. In the second step, a RANSAC based procedure is used to estimate the homography from weighted co-occurrence modes. Note that the first step not only lessens the effects of noise but also reduces the search space for efficient calculation. Unlike most existing algorithms for overlapping camera calibration, the proposed method uses an update mechanism to adapt online to the changes in network topology. The method does not assume prior knowledge about the scene, target, or network properties. It is also robust to noise, traffic intensity, and the amount of overlap between the fields of view. Experiments and quantitative evaluation using both synthetic and real data are presented to support the above claims.

This paper describes an algorithm called ComMode (competitive mode estimation) for building a statistical background model of a complete video shot in the presence of independent camera motion, and independently moving foreground objects. Furthermore, as a by-product of the algorithm, the foreground is automatically segmented as a first step towards detection and tracking of video objects. The algorithm works by treating the intensity value at each pixel location as a time series, and segmenting it into statistically coherent modes. Initially, the most likely background mode is selected by a heuristic. Then, pixels competitively propagate their background mode to neighbouring pixels. The result is that each pixel location ends up with the most likely estimate for its background statistical model.