In this paper, we demonstrated an application of video tracking to radiation detection, where a vision-based tracking system enables a traditional CZT (cadmium zinc telluride)-based radiation imaging device to detect radioactive targets that are in motion. An integrated real-time system consisting of multiple fixed cameras and radiation detectors was implemented and tested. The multi-camera tracking system combines multiple feature cues (such as silhouette, appearance, and geometry) from different viewing angles to ensure consistent target identities under challenging tracking conditions. Experimental results show that both the video tracking and the integrated systems perform accurately and persistently under various scenarios involving multiple vehicles, driving speeds, and driving patterns. The results also validate and reiterate the importance of video tracking as an enabling technology in the field of radiation imaging.

We present a comprehensive approach to track gaze by estimating location, body pose, and head pose direction of multiple individuals in unconstrained environments. The approach combines person detections from fixed cameras with directional face detections obtained from actively controlled pan tilt zoom (PTZ) cameras. The main contribution of this work is to estimate both body pose and head pose (gaze) direction independently from motion direction, using a combination of sequential Monte Carlo Filtering and MCMC sampling. There are numerous benefits in tracking body pose and gaze in surveillance. It allows to track people's focus of attention, can optimize the control of active cameras for biometric face capture, and can provide better interaction metrics between pairs of people. The availability of gaze and face detection information also improves localization and data association for tracking in crowded environments. The performance of the system will be demonstrated on data captured at a real-time surveillance site.

This paper addresses the challenge of recognizing behavior of groups of individuals in unconstraint surveillance environments. As opposed to approaches that rely on agglomerative or decisive hierarchical clustering techniques, we propose to recognize group interactions without making hard decisions about the underlying group structure. Instead we use a probabilistic grouping strategy evaluated from the pairwise spatial-temporal tracking information. A path-based grouping scheme determines a soft segmentation of groups and produces a weighted connection graph where its edges express the probability of individuals belonging to a group. Without further segmenting this graph, we show how a large number of low- and high-level behavior recognition tasks can be performed. Our work builds on a mature multi-camera multi-target person tracking system that operates in real-time. We derive probabilistic models to analyze individual track motion as well as group interactions. We show that the soft grouping can combine with motion analysis elegantly to robustly detect and predict group-level activities. Experimental results demonstrate the efficacy of our approach.

We present a real-time approach to estimating the gaze direction of multiple individuals using a network of far-field surveillance cameras. This work is part of a larger surveillance system that utilizes a network of fixed cameras as well as PTZ cameras to perform site-wide tracking of individuals. Based on the tracking information, one or more PTZ cameras are cooperatively controlled to obtain close-up facial images of individuals. Within these close-up shots, face detection and head pose estimation are performed and the results are provided back to the tracking system to track the individual gazes. A new cost metric based on location and gaze orientation is proposed to robustly associate head observations with tracker states. The tracking system can thus leverage the newly obtained gaze information for two purposes: (i) improve the localization of individuals in crowded settings, and (ii) aid high-level surveillance tasks such as understanding gesturing, interactions between individuals, and finding the object-of-interest that people are looking at. In security application, our system can detect if a subject is looking at the security cameras or guard posts.

Environments such as schools, public parks and prisons and others that contain a large number of people are typically characterized by frequent and complex social interactions. In order to identify activities and behaviors in such environments, it is necessary to understand the interactions that take place at a group level. To this end, this paper addresses the problem of detecting and predicting suspicious and in particular aggressive behaviors between groups of individuals such as gangs in prison yards. The work builds on a mature multi-camera multi-target person tracking system that operates in real-time and has the ability to handle crowded conditions. We consider two approaches for grouping individuals: (i) agglomerative clustering favored by the computer vision community, as well as (ii) decisive clustering based on the concept of modularity, which is favored by the social network analysis community. We show the utility of such grouping analysis towards the detection of group activities of interest. The presented algorithm is integrated with a system operating in real-time to successfully detect highly realistic aggressive behaviors enacted by correctional officers in a simulated prison environment. We present results from these enactments that demonstrate the efficacy of our approach.

In this work we propose a dynamic scene model to provide information about the presence of salient motion in the scene, and that could be used for focusing the attention of a pan/tilt/zoom camera, or for background modeling purposes. Rather than proposing a set of saliency detectors, we define what we mean by salient motion, and propose a precise model for it. Detecting salient motion becomes equivalent to detecting a model change. We derive optimal online procedures to solve this problem, which enable a very fast implementation. Promising results show that our model can effectively detect salient motion even in severely cluttered scenes, and while a camera is panning and tilting.

Intelligent video in urban settings can be challenging due the presence of crowds, clutter, poor camera placement and continuously changing light conditions. The surveillance of sports venues is particularly difficult, because thousands of people can enter or exit a venue in short periods of time. This paper presents a case study of successfully monitoring a sports venue using a multi-camera multi-target tracking system. The system performed site-wide tracking throughout a network of calibrated cameras and was able to accurately track thousands of people in real-time under challenging conditions. The extracted tracking information was used to detect a range of real-time events such as crowd formation, left luggage, and loitering. In addition all video,track and event information was indexed and stored to allow operators to perform playback and forensic search. This paper will present an overview of the deployed system and discuss the challenges that were encountered during the deployment.

This work addresses the important problem of the discovery and analysis of social networks from surveillance video. A computer vision approach to this problem is made possible by the proliferation of video data obtained from camera networks, particularly state-of-the-art Pan-Tilt-Zoom (PTZ) and tracking camera systems that have the capability to acquire high-resolution face images as well as tracks of people under challenging conditions. We perform “opportunistic” face recognition on captured images and compute motion similarities between tracks of people on the ground plane. To deal with the unknown correspondences between faces and tracks, we present a novel graph-cut based algorithm to solve this association problem. It enables the robust estimation of a social network that captures the interactions between individuals in spite of large amounts of noise in the datasets. We also introduce an algorithm that we call “modularity-cut”, which is an Eigen-analysis based approach for discovering community and leadership structure in the estimated social network. Our approach is illustrated with promising results from a fully integrated multi-camera system under challenging conditions over long period of time.

This paper provides comments to the paper "Visual Players Detection and Tracking in Soccer Matches". It was prepared as part of an experimental open review process for the 5th IEEE International Conference On Advanced Video and Signal Based Surveillance for which the author of this commentary paper acted as a reviewer. Due to the conference submission process, this comment was prepared for the review version of the paper and does not consider changes that the authors may have made for the print version of their work. The reviewed paper addresses the problems of automatically detecting, classifying and tracking players in soccer matches. It focuses on three separate sub-problems: estimation of a background model that represents the empty soccer field, classification of team players, and player tracking.

This paper presents a novel distributed framework for multi-target tracking with an efficient data association computation. A decentralized representation of trackers’ motion and association variables is adopted. Considering the interleaved nature of data association and tracker filtering, the multi-target tracking is formulated as a missing data problem, and the solution is found by the proposed variational EM algorithm. We analytically show that 1) the posteriori distributions of trackers’ motions (the real interests in terms of tracking applications) can be effectively computed in the E-step of the EM iterations, and 2) the solution of trackers’ association variables can be pursued under a derived graph-based discrete optimization formulation, thus efficiently estimated in the M-step by the recently emerging graph optimization algorithms. The proposed approach is very general such that sophisticated data association priori and likelihood function can be easily incorporated. This general framework is tested with both simulation data and real world surveillance video. The reported qualitative and quantitative studies verify the effectiveness and low computational cost of the algorithm.

Computer vision approaches for retail applications can provide value far beyond the common domain of loss prevention. Gaining insight into the movement and behaviors of shoppers is of high interest for marketing, merchandizing, store operations and data mining. Of particular interest is the process of purchase decision making. What catches a customers attention? What products go unnoticed? What does a customer look at before making a final decision? Towards this goal we presents a system that detects and tracks both the location and gaze of shoppers in retail environments. While networks of standard overhead store cameras are used for tracking the location of customers, small in-shelf cameras are used for estimating customer gaze. The presented system operates robustly in real-time and can be deployed in a variety of retail applications.

Using a detect and track paradigm, we present a surveillance framework where each camera uses local resources to perform real-time person detection. These detections are then processed by a distributed site-wide tracking system. The detectors themselves are based on boosted user-defined exemplars, which capture both appearance and shape information. The detectors take integral images of both intensity and Sobel responses as input. This data representation enables efficient processing without relying on background subtraction or other motion cues. View-specific person detectors are constructed by iteratively presenting the boosting algorithm with training data associated with each individual camera. These detections are then transmitted from a distributed set of tracking clients to a server, which maintains a set of site-wide target tracks. Automatic calibration methods allow for tracking to be performed in a ground plane representation, which enables effective camera hand-off. Factors such as network latencies and scalability will be discussed.

The ability to determine the identity of a skull found at a crime scene is of critical importance to the law enforcement community. Traditional clay-based methods attempt to reconstruct the face so as to enable identification of the deceased by members of the general public. However, these reconstructions lack consistency from practitioner to practitioner and it has been shown that the human recognition of these reconstructions against a photo gallery of potential victims is little better than chance. In this paper we propose the automation of the reconstruction process. For a given skull, a data-driven 3D generative model of the face is constructed using a database of CT head scans. The reconstruction can be constrained based on prior knowledge such as age and or weight. To determine whether or not these reconstructions have merit, geometric methods for comparing reconstructions against a gallery of facial images are proposed. First, active shape models are used to automatically detect a set of facial landmarks on each image. These landmarks are associated with 3D points on the reconstruction. Direct comparison of the reconstruction is problematic since in general the camera geometry used for image capture is unknown and there are uncertainties associated with the reconstruction and landmark detection processes. The first method of comparison uses constrained optimization to determine the optimal projection of the reconstruction on to the image. Residuals are then analyzed resulting in a ranking of the gallery. The second method uses boosting to learn which points are both reliable and discriminating. This results in a match/no-match classifier. Experimental evidence indicating that skull recognition from facial images can be achieved is presented.

It has been shown that under a small number of assumptions, observations of people can be used to obtain metric calibration information of a camera, which is particularly useful for surveillance applications. However, previous work had to exclude the common criticial configuration of the camera’s principal point falling on the horizon line and very long focal lengths, both of which occur commonly in practise. Due to noise, the quality of the calibration quickly degrades at and in the vicinity of these configurations. This paper provides a robust solution to this problem by incorporating information about the motion of people into the estimation process. It is shown that under the assumption that people walk with a constant velocity, calibration performance can be improved significantly. In addition to solving the above problem, the incorporation of temporal data also helps to take correlations between subsequent detections into consideration, which leads to an up-front reduction of the noise in the measurements and an overall improvement in auto-calibration performance. 1

A number of surveillance scenarios require the detection and tracking of people. Although person detection and counting systems are commercially available today, there is need for further research to address the challenges of real world scenarios. The focus of this work is the segmentation of groups of people into individuals. One relevant application of this algorithm is people counting. Experiments document that the presented approach leads to robust people counts.

The main focus of this work is the integration of feature grouping and model based segmentation into one consistent framework. The algorithm is based on partitioning a given set of image features using a likelihood function that is parameterized on the shape and location of potential individuals in the scene. Using a variant of the EM formulation, maximum likelihood estimates of both the model parameters and the grouping are obtained simultaneously. The resulting algorithm performs global optimization and generates accurate results even when decisions can not be made using local context alone. An important feature of the algorithm is that the number of people in the scene is not modeled explicitly. As a result no prior knowledge or assumed distributions are required. The approach is shown to be robust with respect to partial occlusion, shadows, clutter, and can operate over a large range of challenging view angles including those that are parallel to the ground plane. Comparisons with existing crowd segmentation systems are made and the utility of coupling crowd segmentation with a temporal tracking system is demonstrated.

In the context of visual surveillance of human activity knowledge about a camera's internal and external parameters is useful, as it allows for the establishment of a connection between image and world measurements. Unfortunately, calibration information is rarely available and difficult to obtain after a surveillance system has been installed. In this paper, a method for camera autocalibration based on information gathered by tracking people is developed. It brings two main contributions: first, we show how a foot-to-head plane homology can be used to obtain the calibration parameters and then we show an approach how to efficiently estimate initial parameter estimates from measurements; second, we present a Bayesian solution to the calibration problem that can elegantly handle measurement uncertainties, outliers, as well as prior information. It is shown how the full posterior distribution of calibration parameters given the measurements can be estimated, which allows making statements about the accuracy of both the calibration parameters and the measurements involving them

This paper presents a systematic approach to acquiring model appearance information online for monocular model-based tracking. The acquired information is used to drive a set of complementary imaging cues to obtain a highly discriminatory observation model. Appearance is modeled as a Markov random field of color distributions over the model surface. The online acquisition process estimates appearance-based on uncertain image measurements and is designed to greatly reduce the chance of mapping non-object image data onto the model. Confidences about the different appearance driven imaging cues are estimated in order to adaptively balance the contributions of the different cues. The discriminatory power of the resulting model is good enough to allow long-duration single-hypothesis model-based tracking with no prior appearance information. Careful evaluation based on real and semi-synthetic video sequences shows that the presented algorithm is able to robustly track a wide variety of targets under challenging conditions.

Emergency response requires strategic assessment of risks, decisions, and communications that are time critical while requiring teams of individuals to have fast access to large volumes of complex information and technologies that enable tightly coordinated work. The access to this information by crisis management teams in emergency operations centers can be facilitated through various human-computer interfaces. Unfortunately, these interfaces are hard to use, require extensive training, and often impede rather than support teamwork. Dialogue-enabled devices, based on natural, multimodal interfaces, have the potential of making a variety of information technology tools accessible during crisis management. This paper establishes the importance of multimodal interfaces in various aspects of crisis management and explores many issues in realizing successful speech-gesture driven, dialogue-enabled interfaces for crisis management. This paper is organized in five parts. The first part discusses the needs of crisis management that can be potentially met by the development of appropriate interfaces. The second part discusses the issues related to the design and development of multimodal interfaces in the context of crisis management. The third part discusses the state of the art in both the theories and practices involving these human-computer interfaces. In particular, it describes the evolution and implementation details of two representative systems, Crisis Management (XISM) and Dialog Assisted Visual Environment for Geoinformation (DAVE/spl I.bar/G). The fourth part speculates on the short-term and long-term research directions that will help addressing the outstanding challenges in interfaces that support dialogue and collaboration. Finally, the fifth part concludes the paper.

This paper presents a framework for designing a natural multimodal human computer interaction (HCI) system. The core of the proposed framework is a principled method for combining information derived from audio and visual cues. To achieve natural interaction, both audio and visual modalities are fused along with feedback through a large screen display. Careful design along with due considerations of possible aspects of a systems interaction cycle and integration has resulted in a successful system. The performance of the proposed framework has been validated through the development of several prototype systems as well as commercial applications for the retail and entertainment industry. To assess the impact of these multimodal systems (MMS), informal studies have been conducted. It was found that the system performed according to its specifications in 95% of the cases and that users showed ad-hoc proficiency, indicating natural acceptance of such systems.

This paper presents a multimodal crisis management system (XISM). It employs processing of natural gesture and speech commands elicited by a user to efficiently manage complex dynamic emergency scenarios on a large display. The developed prototype system demonstrates the means of incorporating unconstrained free-hand gestures and speech in a real-time interactive interface. This paper provides insights into the design aspects of the XISM system. In particular, it addresses the issues of extraction and fusion of gesture and speech modalities to allow more natural interactive behavior. Performance characteristics of the current prototype and considerations for future work are discussed. A series of studies indicated positive response with respect to ease of interacting with the current system.