We present the Moments in Time Dataset, a large-scale human-annotated collection of one million short videos corresponding to dynamic events unfolding within three seconds. Modeling the spatial-audio-temporal dynamics even for actions occurring in 3 second videos poses many challenges: meaningful events do not include only people, but also objects, animals, and natural phenomena; visual and auditory events can be symmetrical in time ("opening" is "closing" in reverse), and either transient or sustained. We describe the annotation process of our dataset (each video is tagged with one action or activity label among 339 different classes), analyze its scale and diversity in comparison to other large-scale video datasets for action recognition, and report results of several baseline models addressing separately, and jointly, three modalities: spatial, temporal and auditory. The Moments in Time dataset, designed to have a large coverage and diversity of events in both visual and auditory modalities, can serve as a new challenge to develop models that scale to the level of complexity and abstract reasoning that a human processes on a daily basis.

With the recent broad acceptance of body worn cameras (BWC) for police departments, there is an increased need to perform video analytics for this domain. However, body worn cameras pose several challenges including severe motion blur, barrel camera distortion from wide angle lenses, close proximity and odd viewing angles, and poor lighting conditions. In this paper, we evaluate the performance of several state of the art face detection approaches including Aggregate Channel Features [1] and Faster R-CNN [2] and show their limitations in this domain. We then describe how face detection can be improved for BWC by corroborating information from body parts detection. We design a system using 0–1 linear integer programming to optimize the matching of body parts detections for each person in the scene and to maximize the hit rate of faces with supporting evidence. By leveraging information from body parts detection, we are able to improve the average precision by nearly two percent.

Faster R-CNN achieves state-of-the-art performance on generic object detection. However, a simple application of this method to a large vehicle dataset performs unimpressively. In this paper, we take a closer look at this approach as it applies to vehicle detection. We conduct a wide range of experiments and provide a comprehensive analysis of the underlying structure of this model. We show that through suitable parameter tuning and algorithmic modification, we can significantly improve the performance of Faster R-CNN on vehicle detection and achieve competitive results on the KITTI vehicle dataset. We believe our studies are instructive for other researchers investigating the application of Faster R-CNN to their problems and datasets.

In this paper, we describe a method to automatically calibrate a static camera using only information regarding the vehicles in the scene. We use standard Adaboost vehicle detection combined with feature-based tracking to determine the size and direction of travel of vehicles at various locations in the scene. We then infer the scene geometry including the height of the camera, the camera focal length, and the angle of the camera with respect to the ground. Given the calibration, we can estimate the size and speed of objects and the distances travelled in the scene. We evaluate our system on 10 cameras with manually annotated calibration based on a 3D cube with user specified locations in 2D. We show our system can be used to accurately estimate vehicle sizes to within 2 feet for a range of camera configurations.

We address the problem of describing people based on fine-grained clothing attributes. This is an important problem for many practical applications, such as identifying target suspects or finding missing people based on detailed clothing descriptions in surveillance videos or consumer photos. We approach this problem by first mining clothing images with fine-grained attribute labels from online shopping stores. A large-scale dataset is built with about one million images and fine-detailed attribute sub-categories, such as various shades of color (e.g., watermelon red, rosy red, purplish red), clothing types (e.g., down jacket, denim jacket), and patterns (e.g., thin horizontal stripes, houndstooth). As these images are taken in ideal pose/lighting/background conditions, it is unreliable to directly use them as training data for attribute prediction in the domain of unconstrained images captured, for example, by mobile phones or surveillance cameras. In order to bridge this gap, we propose a novel double-path deep domain adaptation network to model the data from the two domains jointly. Several alignment cost layers placed inbetween the two columns ensure the consistency of the two domain features and the feasibility to predict unseen attribute categories in one of the domains. Finally, to achieve a working system with automatic human body alignment, we trained an enhanced RCNN-based detector to localize human bodies in images. Our extensive experimental evaluation demonstrates the effectiveness of the proposed approach for describing people based on fine-grained clothing attributes.

We develop a robust approach to detect parked vehicles in real time. Our approach particularly focuses on tracking vehicles in long term under challenging conditions such as lighting changes and occlusions. Vehicle tracking is performed by template matching based on fast-computed corner points. The template model is made self-adaptive over time to accommodate lighting changes. We also present an effective way to manage and track multiple vehicles when they are parked close together and occlude one another. We demonstrate the effectiveness of our approach on the challenging i-LIDs data set and another large one collected from real-world scenarios.

Detecting human behaviors in vast amounts of video is a challenging task in a variety of real-world applications. Thus an interactive tool designed to support this task with human in the loop is of significance in various domains including public safety and security. In this paper, we design and develop an interactive visual analytics system, RiskWheel, that enables effective analysis of detection results and utilization of user feedback to improve surveillance event detection. In particular, we propose 1) an interactive approach to visualize data with temporal relations and 2) a novel risk ranking method to differentiate detection results and present more informative ones to the user for better interaction. In our experiments, we demonstrate RiskWheel through a case study on TRECVID Surveillance Event Detection (SED) task [1]. The experimental results quantitatively show that RiskWheel outperforms multiple baselines, demonstrating the power of the risk ranking technique.

In this paper, we show how pedestrian detection accuracy and efficiency can be improved for static surveillance cameras using scene context and temporal non-maximal suppression. First, using the geometry of the scene, we derive the relationship between height in the image with respect to height in the real-world. This relationship is used to learn the range of scales to evaluate for potential detections. The geometry can also be used to estimate distances on the ground plane and to predict the image distance for pedestrian movement. Secondly, we show the error inherent in the standard non-maximum suppression method and demonstrate how the error can be reduced using a small temporal window. The scene context information and temporal non-maximum suppression (tNMS) can be applied to any detection algorithm. We evaluate the accuracy of each method across seven different videos. We show the scene context information and temporal non-maximum suppression improve accuracy on the order of ten and five percent respectively and over fifteen percent combined for seven different scenes for two publicly available detectors. Furthermore, both these approaches significantly reduce computational cost.

Practical surveillance systems deployed in urban scenarios need to operate 24/7 under a wide range of environmental conditions. As modern video analytics shift from blob-based to object-centered architectures, appearance-based object detection under different weather conditions and lighting effects emerges as a critical yet largely unaddressed problem. This paper investigates this research topic, using as a case study the problem of vehicle detection in urban surveillance environments. In particular, we show that a simple and efficient Winsorized lighting correction technique improves performance significantly when outliers due to shadows, specularities, headlights, and occluders are present. Moreover, we demonstrate that a self-training mechanism utilizing a balanced training set automatically acquired from the target domain yields superior performance. Our experimental results are carried out on a novel dataset of vehicle images collected from a public traffic camera and categorized according to multiple environmental conditions.

We address the problem of attribute-based people search in real surveillance environments. The system we developed is capable of answering user queries such as "show me all people with a beard and sunglasses, wearing a white hat and a patterned blue shirt, from all metro cameras in the downtown area, from 2pm to 4pm last Saturday". In this paper, we describe the lessons we learned from practical deployments of our system, and how we made our algorithms achieve the accuracy and efficiency required by many police departments around the world. In particular, we show that a novel set of multimodal integral filters and proper normalization of attribute scores are critical to obtain good performance. We conduct a comprehensive experimental analysis on video footage captured from a large set of surveillance cameras monitoring metro chokepoints, in both crowded and normal activity periods. Moreover, we show impressive results using images from the recent Boston marathon bombing event, where our system can rapidly retrieve the two suspects based on their attributes from a database containing more than one thousand people present at the event.

Several recent investigations attempt to classify vehicles into a small number (5-7) of colors. A significant complication arises, however; a large proportion of vehicles (>50%) are various shades of gray: white, black, silver, gray, and variations such as gun metal and pearly white. Distinguishing such shades of gray in vehicle body color from lighting changes is an unsolved problem. Furthermore, previous studies have evaluated their performance on private datasets precluding a comparison of methodologies. In this paper, we release a public dataset with ground truth color classification for future evaluations and comparisons based on the publicly available i-LIDS data [9]. We describe a method to perform vehicle color classification into 7 frequently occurring colors including dark red, dark blue and light silver, using pose dependent vehicle detection, vehicle alignment, and vehicle body part masks. We introduce new features for tree-based vehicle color classification based on the reliability of color information and the relative color of various vehicle parts.

We present a generic event detection system evaluated in the Surveillance Event Detection (SED) task of TRECVID 2012. We investigate a statistical approach with spatio-temporal features applied to seven event classes, which were defined by the SED task. This approach is based on local spatio-temporal descriptors, called MoSIFT and generated by pair-wise video frames. A Gaussian Mixture Model(GMM) is learned to model the distribution of the low level features. Then for each sliding window, the Fisher vector encoding [improvedFV] is used to generate the sample representation. The model is learnt using a Linear SVM for each event. The main novelty of our system is the introduction of Fisher vector encoding into video event detection. Fisher vector encoding has demonstrated great success in image classification. The key idea is to model the low level visual features as a Gaussian Mixture Model and to generate an intermediate vector representation for bag of features. FV encoding uses higher order statistics in place of histograms in the standard BoW. FV has several good properties: (a) it can naturally separate the video specific information from the noisy local features and (b) we can use a linear model for this representation. We build an efficient implementation for FV encoding which can attain a 10 times speed-up over real-time. We also take advantage of non-trivial object localization techniques to feed into the video event detection, e.g. multi-scale detection and non-maximum suppression. This approach outperformed the results of all other teams submissions in TRECVID SED 2012 on four of the seven event types.

Color information is an important feature for many vision algorithms including color correction, image retrieval and tracking. In this paper, we study the limitations of color measurement accuracy and explore how this information can be used to improve the performance of color correction. In particular, we show that a strong correlation exists between the error in hue measurements on one hand and saturation and intensity on the other hand. We introduce the notion of color strength, which is a combination of saturation and intensity information to determine when hue information in a scene is reliable. We verify the predictive capability of this model on two different datasets with ground truth color information. Further, we show how color strength information can be used to significantly improve color correction accuracy for the 11K real-world SFU gray ball dataset.

We present a novel approach for visual detection and attribute-based search of vehicles in crowded surveillance scenes. Large-scale processing is addressed along two dimensions: 1) large-scale indexing, where hundreds of billions of events need to be archived per month to enable effective search and 2) learning vehicle detectors with large-scale feature selection, using a feature pool containing millions of feature descriptors. Our method for vehicle detection also explicitly models occlusions and multiple vehicle types (e.g., buses, trucks, SUVs, cars), while requiring very few manual labeling. It runs quite efficiently at an average of 66 Hz on a conventional laptop computer. Once a vehicle is detected and tracked over the video, fine-grained attributes are extracted and ingested into a database to allow future search queries such as “Show me all blue trucks larger than 7 ft. length traveling at high speed northbound last Saturday, from 2 pm to 5 pm”. We perform a comprehensive quantitative analysis to validate our approach, showing its usefulness in realistic urban surveillance settings.

We present a novel application for searching for vehicles in surveillance videos based on semantic attributes. At the interface, the user specifies a set of vehicle characteristics (such as color, direction of travel, speed, length, height, etc.) and the system automatically retrieves video events that match the provided description. A key differentiating aspect of our system is the ability to handle challenging urban conditions such as high volumes of activity and environmental factors. This is achieved through a novel multi-view vehicle detection approach which relies on what we call motionlet classifiers, i.e. classifiers that are learned with vehicle samples clustered in the motion configuration space. We employ massively parallel feature selection to learn compact and accurate motionlet detectors. Moreover, in order to deal with different vehicle types (buses, trucks, SUVs, cars), we learn the motionlet detectors in a shape-free appearance space, where all training samples are resized to the same aspect ratio, and then during test time the aspect ratio of the sliding window is changed to allow the detection of different vehicle types. Once a vehicle is detected and tracked over the video, fine-grained attributes are extracted and ingested into a database to allow future search queries such as "Show me all blue trucks larger than 7ft length traveling at high speed northbound last Saturday, from 2pm to 5pm".

We present a novel approach for vehicle detection in urban surveillance videos, capable of handling unstructured and crowded environments with large occlusions, different vehicle shapes, and environmental conditions such as lighting changes, rain, shadows, and reflections. This is achieved with virtually no manual labeling efforts. The system runs quite efficiently at an average of 66Hz on a conventional laptop computer. Our proposed approach relies on three key contributions: (1) a co-training scheme where data is automatically captured based on motion and shape cues and used to train a detector based on appearance information; (2) an occlusion handling technique based on synthetically generated training samples obtained through Poisson image reconstruction from image gradients; (3) massively parallel feature selection over multiple feature planes which allows the final detector to be more accurate and more efficient. We perform a comprehensive quantitative analysis to validate our approach, showing its usefulness in realistic urban surveillance settings.

In this paper, we evaluate several low dimensional color features for object retrieval in surveillance video. Previous work in object retrieval in surveillance has been hampered by issues in low resolution, poor segmentation, pose and lighting variations and the cost of retrieval. To overcome these difficulties, we restrict our analysis to alarm-based vehicle detection and as a consequence, we restrict both pose and lighting variations. In addition, we study the utility of example-based retrieval to avoid the limitations of strict color classification. Finally, since we perform our evaluation at run-time for alarm-based detection, we do not need to index into a large database. We evaluate the efficiency and effectiveness of several color features including standard color histograms, weighted color histograms, variable bin size color histograms and color correlograms. Results show color correlogram to have the best performance for our datasets.

Urban environments present unique challenges from the perspective of surveillance and security. Threat activity in urban environments tends to be very similar to background activity, while the volume of activity is often very high. The widespread geographical area presents issues from the perspective of response. These characteristics of urban environments create challenges to traditional applications of video analytics technologies and opens up opportunities for novel approaches. This paper explores the applicability of video analytics in various scenarios presented in urban surveillance situations. We also describe novel technical solutions to some of the challenges of urban surveillance.

We propose a novel framework for searching for people in surveillance environments. Rather than relying on face recognition technology, which is known to be sensitive to typical surveillance conditions such as lighting changes, face pose variation, and low-resolution imagery, we approach the problem in a different way: we search for people based on a parsing of human parts and their attributes, including facial hair, eyewear, clothing color, etc. These attributes can be extracted using detectors learned from large amounts of training data. A complete system that implements our framework is presented. At the interface, the user can specify a set of personal characteristics, and the system then retrieves events that match the provided description. For example, a possible query is ¿show me the bald people who entered a given building last Saturday wearing a red shirt and sunglasses.¿ This capability is useful in several applications, such as finding suspects or missing people. To evaluate the performance of our approach, we present extensive experiments on a set of images collected from the Internet, on infrared imagery, and on two-and-a-half months of video from a real surveillance environment. We are not aware of any similar surveillance system capable of automatically finding people in video based on their fine-grained body parts and attributes.

In this paper, we present retrieval methods for extracting colored moving objects from surveillance cameras. In particular, we describe a method to classify moving objects into one of six colors. The method includes two sets of parameters. The first set can be used to compensate for illumination conditions and camera differences. The second set is used to tune the color extraction for specific object types and optimal retrieval. The tuning can be performed after video analysis has extracted metadata in real-time. The metadata is fed into a database which can be interactively queried by the user. The ability to query with feedback makes it possible to successfully find events of interest during an investigation without performing real-time adaptation which might be error-prone. In addition, for situations in which tracking moving objects is difficult, the system can generate significant colored object snippets of moving colored objects for retrieval. If the event of interest is not found by the initial system, this backup system can be invoked ensuring that the relevant event is found.

Moving object classification in far-field video is a key component of smart surveillance systems. In this paper, we propose a reliable system for person-vehicle classification which works well in challenging real-word conditions, including the presence of shadows, low resolution imagery, perspective distortions, arbitrary camera viewpoints, and groups of people. Our system runsin real-time (30 Hz) on conventional machines and has low memory consumption. We achieved accurate results by relying on powerful discriminative features, including a novel measure of object deformation based on differences of histograms of oriented gradients. We also provide an interactive user interface, enabling users to specify regions of interest for each class and correct for perspective distortions by specifying different sizes indifferent positions of the camera view. Finally, we use anautomatic adaptation process to continuously update the parameters of the system so that its performance increases for a particular environment. Experimental results demonstrate the effectiveness of our system in standard dataset and a variety of video clips captured with our surveillance cameras.

Surveillance video is used in two key modes, watching for known threats in real-time and searching for events of interest after the fact. Typically, real-time alerting is a localized function, e.g. airport security center receives and reacts to a “perimeter breach alert”, while investigations often tend to encompass a large number of geographically distributed cameras like the London bombing, or Washington sniper incidents. Enabling effective search of surveillance video for investigation & preemption, involves indexing the video along multiple dimensions. This paper presents a framework for surveillance search which includes, video parsing, indexing and query mechanisms. It explores video parsing techniques which automatically extract index data from video, indexing which stores data in relational tables, retrieval which uses SQL queries to retrieve events of interest and the software architecture that integrates these technologies.

We describe a set of tools for retail analytics based on a combination of video understanding and transaction-log. Tools are provided for loss prevention (returns fraud and cashier fraud), store operations (customer counting) and merchandising (display effectiveness). Results are presented on returns fraud and customer counting.

Pervasive sensor based systems are transforming Information Technology systems from being transactional in nature to being observational in nature. Observational systems are inherently distributed and capture information at a much finer grain of space and time. Enabling and building such systems also poses many technology challenges, extracting information from sensor signals, indexing and searching sensor meta-data, data mining and scalability. In this paper we use S3: the IBM smart surveillance system as an example of an observational system to explore several of these issues through real world deployment examples.

This paper proposes a fully automatic approach for Ca2+spark events detection in confocal image system. This system automatically detects the region of interest and separates it from the background, automatically finds the optimal thresholds for Ca2+sparks detection, and takes advantage of the correlation between two consecutive frames in detecting the Ca2+sparks. The experiment results show that the proposed method is very promising. Compared to the conventional system (proposed by Cheng et. al and later improved by Chun et. al), the preliminary experimental results show that the proposed system can achieve higher accuracy rate without compromising false alarm rate.

As smart surveillance technology becomes a critical component in security infrastructures, the system architecture assumes a critical importance. This paper considers the example of smart surveillance in an airport environment. We start with a threat model for airports and use this to derive the security requirements. These requirements are used to motivate an open-standards based architecture for surveillance. We discuss the critical aspects of this architecture and its implementation in the IBM S3 smart surveillance system. Demo results from a pilot deployment in Hawthorne, NY are presented.

The detection and tracking of people lie at the heart of many current and near-future applications of computer vision. We describe a background subtraction system designed to detect moving objects in a wide variety of conditions, and a second system to detect objects moving in front of moving backgrounds. Detected foreground regions are tracked with a tracking system which can initiate real-time alarms and generate a smart surveillance index which can be searched to find interesting events in stored video

The level of security at a facility is directly related to how well the facility can keep track of "who is where". The "who" part of this question is typically addressed through the use of face images for recognition either by a person or a computer face recognition system. The "where" part of this question can be addressed through 3D position tracking. The "who is where" problem is inherently multi-scale, wide angle views are needed for location estimation and high resolution face images for identification. A number of other people tracking challenges like activity understanding are multiscale in nature. An effective system to answer "who is where?" must acquire face images without constraining the users and must closely associate the face images with the 3D path of the person. Our solution to this problem uses computer controlled pan-tilt-zoom cameras driven by a 3D wide-baseline stereo tracking system. The pan-tilt-zoom cameras automatically acquire zoomed-in views of a person's head, while the person is in motion within the monitored space.

We have developed a fast robust 3D head tracking system based on rendering a texture-mapped cylinder. In order to handle the variable frame-to-frame motion changes, the system uses motion templates, which adapt to the current size of the motion increment. The relationship between measurable pixel energy and tracking error is used to design the parameters of the adaptive algorithm. To speedup processing and decouple rotational and translational motion, 2D positional information of the neckline is utilize. The confidence at each point is computed based on the amount of information used in creating the texture map and re-rendering the face. The system can also handle large out-of-plane rotations via additional templates. If the tracker fails, it can recover using an independent face detection routine. We compare the results of our approach with the extensive results of a closely related technique.

A software tool, which allows a middle school student to take measurements on images and videos, is presented. Using this software, students became scientists as they measure, collect and analyze data from a wide range of imagery. We demonstrate several novel techniques in image/video processing which enable students to take sophisticated measurements on a wide range of imagery. Routines which perform intelligent color segmentation and create visualizations of motion information are integrated into the software.

A method for determining local surface orientation from the autocorrelation function of statistically isotropic textures is introduced. It relies on the foreshortening that occurs in the image of an oriented surface, and the analogous foreshortening produced in the texture-autocorrelation function. This method assumes textural isotropy, but does not require the texture to be composed of texels or assume other texture regularities. The technique was applied to natural images of planar-textured surfaces and found to give good results. The simplicity of the method and its use of information from all parts of the image are emphasized.>

A method for determining local surface orientation from the autocorrelation function of statistically isotropic textures is introduced. It relies on the foreshortening that occurs in the image of an oriented surface, and the analogous foreshortening produced in the texture autocorrelation function. This method assumes textural isotropy, but does not require the texture to be composed of texels or assume texture regularities such as equal-area texels or equal spacing between texels. This technique is applied to natural images of planar textured surfaces and found to give good results in many instances. The simplicity of the method and its use of information from all parts of the image are emphasized.>