This paper studies the problem of predicting the distribution over multiple possible future paths of people as they move through various visual scenes. We make two main contributions. The first contribution is a new dataset, created in a realistic 3D simulator, which is based on real world trajectory data, and then extrapolated by human annotators to achieve different latent goals. This provides the first benchmark for quantitative evaluation of the models to predict multi-future trajectories. The second contribution is a new model to generate multiple plausible future trajectories, which contains novel designs of using multi-scale location encodings and convolutional RNNs over graphs. We refer to our model as Multiverse. We show that our model achieves the best results on our dataset, as well as on the real-world VIRAT/ActEV dataset (which just contains one possible future).

We present a multimodal sensor system for wound assessment and pressure ulcer care. Multiple imaging modalities including RGB, three- dimensional (3-D) depth, thermal, multispectral, and chemical sensing are integrated into a portable hand-held probe for real-time wound assessment. Analytic and quantitative algorithms for various assessments including tissue composition, wound measurement in 3-D, temperature profiling, spectral, and chemical vapor analysis are developed. After each assessment scan, 3-D models of the wound are generated on the fly for geometric measurement, while multimodal observations are analyzed to estimate healing progress. Collaboration between developers and clinical practitioners was conducted at the Charlie Norwood VA Medical Center for in-field data collection and experimental evaluation. A total of 133 assessment sessions from 23 enrolled subjects were collected, on which the multimodal data were analyzed and validated with respect to clinical notes associated with each subject. The system can be operated by nontechnical caregivers on a regular basis to aid wound assessment and care. A web portal front-end was developed for clinical decision and telehealth support, where all historical patient data including wound measurements and analysis can be organized online.

Drift is the most difficult issue in object visual tracking based on framework of “tracking-by-detection”. Due to the self-taught learning, the mis-aligned samples are potentially to be incorporated in learning and degrade the discrimination of the tracker. This paper proposes a new tracking approach that resolves this problem by three multi-level collaborative components: a high-level global appearance tracker provides a basic prediction, upon which the structure preserved low-level local patches matching helps to guarantee precise tracking with minimized drift. Those local patches are deliberately deployed on the foreground object via foreground/background segmentation, which is realized by a simple and efficient classifier trained by super-pixel segments. Experimental results show that the three closely collaborated components enable our tracker runs in real time and performs favourably against state-of-the-art approaches on challenging benchmark sequences.

In this paper, our goal is to speed up a standard sliding window detector while maintaining detection accuracies. We do this by decomposing its weight template into multiple component detectors with no redundancy. Each component detector captures partial discriminative appearance, and they can be used in a cascade detection pipeline to aggressively reduce the size of search space. We formulate the decomposition problem as an orthogonal procrustes problem. We further approximate the component detector in the first cascade layer as separable 2D filters (i.e. a product between two vector filters). The running time of such component detector is thus reduced from quadratic to linear with the dimension lengths of the detector template. We conduct extensive experiments using our approach on two well-known object detection datasets: INRIA pedestrian detection dataset and PASCAL VOC 2007 detection dataset. We used HOG features and CNN features in our experiments. Our approach is almost "free lunch": without manipulating the feature representations, it makes the detection process several times faster.

We investigate the use of a vision-based system capable of estimating social states such as rapport and hostility. We study the correlation between interpretations automatically generated by our system and those reported by social scientists. Our multi-camera vision system collects visual cues including location (proximity), motion, pose, gaze, and facial expressions in real-time from multiple subjects moving freely in an unconstrained environment. We performed experiments on 80+ subjects. Preliminary regression analysis suggests high correlation between machine distilled time series signals and assessments made by human experts.

This paper presents a non-invasive human activity monitoring system with a low-cost Doppler sensor and a pair of radio frequency (RF) sensors. This radio-based system combines the strengths of two sensing modalities: fine granularity from a Doppler sensor and large sensing coverage from an RF link. The system is capable of detecting subtle human motion such as breathing, as well as walking in a large area. We deploy the system and perform experiments in a 5.5 m by 7.5 m room to classify four activities. Experimental results show that the average classification rate is 90%, 31% more accurate than a single Doppler system.

In this paper, we demonstrated an application of video tracking to radiation detection, where a vision-based tracking system enables a traditional CZT (cadmium zinc telluride)-based radiation imaging device to detect radioactive targets that are in motion. An integrated real-time system consisting of multiple fixed cameras and radiation detectors was implemented and tested. The multi-camera tracking system combines multiple feature cues (such as silhouette, appearance, and geometry) from different viewing angles to ensure consistent target identities under challenging tracking conditions. Experimental results show that both the video tracking and the integrated systems perform accurately and persistently under various scenarios involving multiple vehicles, driving speeds, and driving patterns. The results also validate and reiterate the importance of video tracking as an enabling technology in the field of radiation imaging.

Given the size and congestion associated with a standard convenience store, it is possible to achieve automated site-wide tracking for each customer via a network of in-store video cameras. For the purposes of reducing customer related shrinkage (theft), it may be advantageous to produce summary videos of the activities associated with each customer. As the customer approaches an exit or checkout station, the cashier could quickly review the associated summery video for the purposes of determining whether or not a theft has taken place. Such a summery video would be automatically generated from clips produced by a network of in-store cameras. The value of the summery videos can be determined based on their brevity and the cashier's ability to detect the presence or absence of a theft event. To this end we propose that a site-wide tracking system coupled with a clip nomination process that is based on a set of action saliency measures be used to produce the desired summery videos.

Effective perimeter protection mechanisms for industrial sites and critical infrastructure must contend with a large variety of potential threats as well as with the fact that normal site activity can be both complex and diverse. This paper documents the development of a system level approach capable of functioning under such challenging conditions. A multi-view tracking system is used to provide real-time site wide trajectories of all observed individuals. A Radar-based system is also used for tracking if and when camera coverage of various regions is not available. Track information is then analyzed with respect to articulated motion analysis, complex event analysis and normalcy analysis. In addition, object recognition is used to classify left behind objects using high resolution PTZ imagery. A real-time integrated version of this comprehensive approach to perimeter protection was deployed using a single standard off-the-shelf desktop computer.

Summary form only given. The rapid evolution of tools and software systems to design experiments, automatically monitor, collect and warehouse large amounts of data, from applications such as life sciences and industrial processes has resulted in a new paradigm shift. This change of paradigm is so fast that some of the practices for optimization and management of these processes that were valid only 5–10 years ago may no longer be fully acceptable or sufficient for today's business optimization and management. This has a direct influence on the best practices for knowledge discovery and management of the discovered knowledge in real-world data mining applications. Establishing and managing a real-world data mining project in any domain, in particular in today's life science industry, is not a trivial task. A few approaches have been proposed in the literature. However, initiation and successful management of such efforts may depend on where a given case study fits in the overall classification of data mining approaches. Today's knowledge discovery from data can be classified in several ways: (i) data mining on engineered systems (e.g. complex equipment) or systems designed by nature (e.g. life sciences), (ii) explanatory or predictive data mining, (iii) data mining from static data (e.g. data warehouse) or dynamic data (e.g. data streams), (iv) user operated or automated data mining. There could still be other ways to classify data mining applications. This talk provides an overview of the above listed knowledge discovery applications. We provide examples where we demonstrate how small or large amounts of data, when understood from a real-world data mining point of view and the required data is properly integrated, can result in novel knowledge discovery case studies. We explain motivations and challenges of establishing real-world dat

In this paper, we address the problem of online learning to recognize people from visual appearances, a prerequisite step towards building a fully intelligent and context-aware smart environment. While the trajectories of tracked individuals are responsible for producing samples to the appearance signature learning process, it is highly risky to directly label these appearance samples with tracker IDs, due to possible tracker switches and temporary tracker losses. Through the exploration of trajectory fidelity in terms of temporal continuity and spatial locality, we show that the side information from tracking, in the form of pairwise constraints, such as “must-link” and “cannot-link”, could significantly benefit signature learning. Furthermore, to learn and update an online identity signature pool, a two-step approach is proposed: 1) a data clustering step based on spectral kernel learning with pairwise constraints, and 2) a large-margin based discriminative signature model learning step. A real-world setup in a smart office environment is used to evaluate the performance of the learning paradigm. Consistent recognition of individuals from live videos verifies the efficacy and effectiveness of our proposal.

Environments such as schools, public parks and prisons and others that contain a large number of people are typically characterized by frequent and complex social interactions. In order to identify activities and behaviors in such environments, it is necessary to understand the interactions that take place at a group level. To this end, this paper addresses the problem of detecting and predicting suspicious and in particular aggressive behaviors between groups of individuals such as gangs in prison yards. The work builds on a mature multi-camera multi-target person tracking system that operates in real-time and has the ability to handle crowded conditions. We consider two approaches for grouping individuals: (i) agglomerative clustering favored by the computer vision community, as well as (ii) decisive clustering based on the concept of modularity, which is favored by the social network analysis community. We show the utility of such grouping analysis towards the detection of group activities of interest. The presented algorithm is integrated with a system operating in real-time to successfully detect highly realistic aggressive behaviors enacted by correctional officers in a simulated prison environment. We present results from these enactments that demonstrate the efficacy of our approach.

Intelligent video in urban settings can be challenging due the presence of crowds, clutter, poor camera placement and continuously changing light conditions. The surveillance of sports venues is particularly difficult, because thousands of people can enter or exit a venue in short periods of time. This paper presents a case study of successfully monitoring a sports venue using a multi-camera multi-target tracking system. The system performed site-wide tracking throughout a network of calibrated cameras and was able to accurately track thousands of people in real-time under challenging conditions. The extracted tracking information was used to detect a range of real-time events such as crowd formation, left luggage, and loitering. In addition all video,track and event information was indexed and stored to allow operators to perform playback and forensic search. This paper will present an overview of the deployed system and discuss the challenges that were encountered during the deployment.

This work addresses the important problem of the discovery and analysis of social networks from surveillance video. A computer vision approach to this problem is made possible by the proliferation of video data obtained from camera networks, particularly state-of-the-art Pan-Tilt-Zoom (PTZ) and tracking camera systems that have the capability to acquire high-resolution face images as well as tracks of people under challenging conditions. We perform “opportunistic” face recognition on captured images and compute motion similarities between tracks of people on the ground plane. To deal with the unknown correspondences between faces and tracks, we present a novel graph-cut based algorithm to solve this association problem. It enables the robust estimation of a social network that captures the interactions between individuals in spite of large amounts of noise in the datasets. We also introduce an algorithm that we call “modularity-cut”, which is an Eigen-analysis based approach for discovering community and leadership structure in the estimated social network. Our approach is illustrated with promising results from a fully integrated multi-camera system under challenging conditions over long period of time.

This paper studies image alignment, the problem of learning a shape and appearance model from labeled data and efficiently fitting the model to a non-rigid object with large variations. Given a set of images with manually labeled landmarks, our model representation consists of a shape component represented by a point distribution model and an appearance component represented by a collection of local features, trained discriminatively as a two-class classifier using boosting. Images with ground truth landmarks are the positive training samples while those with perturbed landmarks are considered as negatives. Enabled by piece-wise affine warping, corresponding local feature positions across all training samples form a hypothesis space for boosting. Image alignment is performed by maximizing the boosted classifier score, which is our distance measure, through iteratively mapping the feature positions to the image, and computing the gradient direction of the score with respect to the shape parameter. We apply this approach to human body alignment from surveillance-type images. We conduct experiments on the MIT pedestrian database where the body size is approximately 110 times 46 pixels, and demonstrate our real-time alignment capability.

This paper presents a novel distributed framework for multi-target tracking with an efficient data association computation. A decentralized representation of trackers’ motion and association variables is adopted. Considering the interleaved nature of data association and tracker filtering, the multi-target tracking is formulated as a missing data problem, and the solution is found by the proposed variational EM algorithm. We analytically show that 1) the posteriori distributions of trackers’ motions (the real interests in terms of tracking applications) can be effectively computed in the E-step of the EM iterations, and 2) the solution of trackers’ association variables can be pursued under a derived graph-based discrete optimization formulation, thus efficiently estimated in the M-step by the recently emerging graph optimization algorithms. The proposed approach is very general such that sophisticated data association priori and likelihood function can be easily incorporated. This general framework is tested with both simulation data and real world surveillance video. The reported qualitative and quantitative studies verify the effectiveness and low computational cost of the algorithm.

Computer vision approaches for retail applications can provide value far beyond the common domain of loss prevention. Gaining insight into the movement and behaviors of shoppers is of high interest for marketing, merchandizing, store operations and data mining. Of particular interest is the process of purchase decision making. What catches a customers attention? What products go unnoticed? What does a customer look at before making a final decision? Towards this goal we presents a system that detects and tracks both the location and gaze of shoppers in retail environments. While networks of standard overhead store cameras are used for tracking the location of customers, small in-shelf cameras are used for estimating customer gaze. The presented system operates robustly in real-time and can be deployed in a variety of retail applications.

Boosting has been widely applied in computer vision, especially after Viola and Jones's seminal work. The marriage of rectangular features and integral-image- enabled fast computation makes boosting attractive for many vision applications. However, this popular way of applying boosting normally employs an exhaustive feature selection scheme from a very large hypothesis pool, which results in a less-efficient learning process. Furthermore, this poses additional constraint on applying boosting in an onine fashion, where feature re-selection is often necessary because of varying data characteristic, but yet impractical due to the huge hypothesis pool. This paper proposes a gradient-based feature selection approach. Assuming a generally trained feature set and labeled samples are given, our approach iteratively updates each feature using the gradient descent, by minimizing the weighted least square error between the estimated feature response and the true label. In addition, we integrate the gradient-based feature selection with an online boosting framework. This new online boosting algorithm not only provides an efficient way of updating the discriminative feature set, but also presents a unified objective for both feature selection and weak classifier updating. Experiments on the person detection and tracking applications demonstrate the effectiveness of our proposal.

Video-based multiple target tracking (MTT) is a challenging task when similar targets are present in close vicinity. Because their visual observations are mixed and difficult to segment, their motions have to be estimated jointly. Most existing approaches perform this joint motion estimation in a centralized fashion and involve searching a rather high dimensional space, and thus leading to quite complicated joint trackers. This paper brings a new view to MTT from a game-theoretic perspective, bridging the joint motion estimation and the Nash equilibrium of a game. Instead of designing a centralized tracker, MTT is decentralized and a set of individual trackers is used, each of which tries to maximize its visual evidence for explaining its motion as well as generates interferences to others. Modelling this competition behavior, a special game is designed so that the difficult joint motion estimation is achieved at the Nash Equilibrium of this game where no individual tracker has incentives to change its motion estimate. This paper substantializes this novel idea in a solid case study where individual trackers are kernel-based trackers. An efficient best response updating procedure is designed to find the Nash equilibrium. The powerfulness of this game-theoretic MTT is shown by promising results on difficult real videos.

This paper describes a novel approach to optimal kernel placement in kernel-based tracking. If kernels are placed at arbitrary places, kernel-based methods are likely to be trapped in ill-conditioned locations, which prevents the reliable recovery of the motion parameters and jeopardizes the tracking performance. The theoretical analysis presented in this paper indicates that the optimal kernel placement can be evaluated based on a closed-form criterion, and achieved efficiently by a novel gradient-based algorithm. Based on that, new methods for temporal-stable multiple kernel placement and scale-invariant kernel placement are proposed. These new theoretical results and new algorithms greatly advance the study of kernel-based tracking in both theory and practice. Extensive real-time experimental results demonstrate the improved tracking reliability.

A fundamental issue in differential motion analysis is the compromise between the flexibility of the matching criterion for image regions and the ability of recovering the motion. Localized matching criteria, e.g., pixel-based SSD, may enable the recovery of all motion parameters, but it does not tolerate much appearance changes. On the other hand, global criteria, e.g., matching histograms, can accommodate dramatic appearance changes, but may be blind to some motion parameters, e.g., scaling and rotation. This paper presents a novel differential approach that integrates the advantages of both in a principled way based on a spatial-appearance model (SAM) that combines local appearances variations and global spatial structures. This model can capture a large variety of appearance variations that are attributed to the local non-rigidity. At the same time, this model enables efficient recovery of all motion parameters. A maximum likelihood matching criterion is defined and rigorous analytical results are obtained that lead to a closed form solution to motion tracking. Very encouraging results demonstrate the effectiveness and efficiency of the proposed method for tracking non-rigid objects that exhibit dramatic appearance deformations, large object scale changes and partial occlusions.

The large shape variability and partial occlusions challenge most object detection and tracking methods for nonrigid targets such as pedestrians. This paper presents a new approach based on a two-layer statistical field model that characterizes the prior of the complex shape variations as a Boltzmann distribution and embeds this prior and the complex image likelihood into a Markov field. A probabilistic variational analysis of this model reveals a set of fixed-point equations characterizing the equilibrium of the field. It leads to computationally efficient methods for calculating the image likelihood and for training the model. Based on that, effective algorithms for detecting nonrigid objects are developed. This new approach has several advantages. First, it is intrinsically suitable for capturing local nonrigidity. In addition, due to the distributed likelihood, this approach is robust to partial occlusions. Moreover, the two-layer structure provides large flexibility of modeling the image observations, which makes the new method robust to clutters. Extensive experiments demonstrate its effectiveness.

This paper presents a new statistical model for detecting and tracking deformable objects such as pedestrians, where large shape variations induced by local shape deformation can not be well captured by global methods such as PCA. The proposed model employs a Boltzmann distribution to capture the prior of local deformation, and embeds it into a Markov network which can be learned from data. A mean field variational analysis of this model provides computationally efficient algorithms for computing the likelihood of image observations and facilitate fast model training. Based on that, effective detection and tracking algorithms for deformable objects are proposed and applied to pedestrian detection and tracking. The proposed method has several advantages. Firstly, it captures local deformation well and thus is robust to occlusions and clutter. In addition, it is computationally tractable. Moreover, it divides deformation into local deformation and global deformation, then conquers them by combining bottom-up and top-down methodologies. Extensive experiments demonstrate the effectiveness of the proposed model for deformable objects.

This paper presents a decentralized approach to multiple target tracking. The novelty of this approach lies in the use of a set of autonomous while collaborative trackers to overcome the tracker coalescence problem with linear complexity. In this approach, the individual trackers are autonomous in the sense that they can select targets to track and evaluate themselves, and they are also collaborative since they need to compete for the targets against those trackers that are close to them through communication. The theoretical foundation of this new approach is based on the variational analysis of a Markov network that reveals the collaborative mechanism through fixed point iteration among these trackers and the existence of the equilibriums. In addition, a trained object detector is incorporated to help sense the potential newly appearing targets in the dynamic scene. Experimental results on challenging video sequences demonstrate the effectiveness and efficiency of the proposed method.

We propose a generative model approach to contour tracking against nonstationary clutter and to coping with occlusions by explicit modelling and inferring. The proposed dynamic Bayesian networks consist of multiple hidden processes, which model the target, the clutter and the occlusions. The image observation models, which depict the generation of the image features, are conditioned on all the hidden processes. Based on this framework, the tracker can automatically switch among different observation models according to the hidden states of the clutter and occlusions. In addition, the inference of these hidden states provides self-evaluations for the tracker. The tracking and inference are implemented based on sequence Monte Carlo techniques. The effectiveness of the proposed approach to robust tracking and inferring nonstationary clutter and occlusion is demonstrated for a variety of image sequences.

Occlusion is a difficult problem for appearance-based target tracking, especially when we need to track multiple targets simultaneously and maintain the target identities during tracking. To cope with the occlusion problem explicitly, this paper proposes a dynamic Bayesian network, which accommodates an extra hidden process for occlusion and stipulates the conditions on which the image observation likelihood is calculated. The statistical inference of such a hidden process can reveal the occlusion relations among different targets, which makes the tracker more robust against partial even complete occlusions. In addition, considering the fact that target appearances change with views, another generative model for multiple view representation is proposed by adding a switching variable to select from different view templates. The integration of the occlusion model and multiple view model results in a complex dynamic Bayesian network, where extra hidden processes describe the switch of targets' templates, the targets' dynamics, and the occlusions among different targets. The tracking and inferring algorithms are implemented by the sampling-based sequential Monte Carlo strategies. Our experiments show the effectiveness of the proposed probabilistic models and the algorithms.

A new method for visual tracking of articulated objects is presented. Analyzing articulated motion is challenging because the dimensionality increase potentially demands tremendous increase of computation. To ease this problem, we propose an approach that analyzes subparts locally while reinforcing the structural constraints at the mean time. The computational model of the proposed approach is based on a dynamic Markov network, a generative model which characterizes the dynamics and the image observations of each individual subpart as well as the motion constraints among different subparts. Probabilistic variational analysis of the model reveals a mean field approximation to the posterior densities of each subparts given visual evidence, and provides a computationally efficient way for such a difficult Bayesian inference problem. In addition, we design mean field Monte Carlo (MFMC) algorithms, in which a set of low dimensional particle filters interact with each other and solve the high dimensional problem collaboratively. Extensive experiments on tracking human body parts demonstrate the effectiveness, significance and computational efficiency of the proposed method.