We address the problem of 24/7 object detection in urban surveillance videos, which presents unique challenges due to significant object appearance variations caused by lighting effects such as shadows and specular reflections, object pose variation, multiple weather conditions, and different times of the day. Rather than training a generic detector and adapting its parameters over time to handle all these variations, we rely on a large set of complementary and extremely efficient detector models, covering multiple overlapping appearance subspaces. At run time, our method continuously selects the most suitable detectors for a given scene and condition, using a novel approach inspired by parametric background modeling algorithms. We provide a comprehensive experimental analysis to show the effectiveness of our approach, considering traffic monitoring as our application domain. Our system runs at 100 frames per second on a standard laptop computer.

We address the problem of attribute-based people search in real surveillance environments. The system we developed is capable of answering user queries such as "show me all people with a beard and sunglasses, wearing a white hat and a patterned blue shirt, from all metro cameras in the downtown area, from 2pm to 4pm last Saturday". In this paper, we describe the lessons we learned from practical deployments of our system, and how we made our algorithms achieve the accuracy and efficiency required by many police departments around the world. In particular, we show that a novel set of multimodal integral filters and proper normalization of attribute scores are critical to obtain good performance. We conduct a comprehensive experimental analysis on video footage captured from a large set of surveillance cameras monitoring metro chokepoints, in both crowded and normal activity periods. Moreover, we show impressive results using images from the recent Boston marathon bombing event, where our system can rapidly retrieve the two suspects based on their attributes from a database containing more than one thousand people present at the event.

Many modern retail stores have self-checkout stations where customers can ring up their own orders without the assistance of any store personnel. To promote customer honesty these systems often weigh each item as it is placed in the bag to confirm that it has the expected mass for the product scanned. In our system we augment this basic check with an assessment of the item's visual appearance to further ensure that the correct code has been entered.

Urban environments present unique challenges from the perspective of surveillance and security. Threat activity in urban environments tends to be very similar to background activity, while the volume of activity is often very high. The widespread geographical area presents issues from the perspective of response. These characteristics of urban environments create challenges to traditional applications of video analytics technologies and opens up opportunities for novel approaches. This paper explores the applicability of video analytics in various scenarios presented in urban surveillance situations. We also describe novel technical solutions to some of the challenges of urban surveillance.

We present a novel framework for recognizing repetitive sequential events performed by human actors with strong temporal dependencies and potential parallel overlap. Our solution incorporates sub-event (or primitive) detectors and a spatiotemporal model for sequential event changes. We develop an effective and efficient method to integrate primitives into a set of sequential events where strong temporal constraints are imposed on the ordering of the primitives. In particular, the combination process is approached as an optimization problem. A specialized Viterbi algorithm is designed to learn and infer the target sequential events and handle the event overlap simultaneously. To demonstrate the effectiveness of the proposed framework, we report detailed quantitative analysis on a large set of cashier checkout activities in a retail store.

A significant portion of retail shrink is attributed to employees and occurs around the point of sale (POS). In this paper, we target a major type of retail fraud in surveillance videos, known as sweethearting (or fake scan), where a cashier intentionally fails to enter one or more items into the transaction in an attempt to get free merchandise for the customer. We first develop a motion-based algorithm to identify video segments as candidates for primitive events at the POS. We then apply spatio-temporal features to recognize true primitive events from the candidates and prune those falsely alarmed. In particular, we learn location-aware event models by Multiple-Instance Learning to address the location-sensitive issues that appear in our problem. Finally, we validate the entire transaction by combining primitive events according to temporal ordering constraints. We demonstrate the effectiveness of our approach on data captured from a real grocery store.

Video analytics have recently emerged as a promising technique of retail fraud detection for loss prevention. Efficient video analytic algorithms are highly desired for a practical fraud detection system. In this paper, we present a real-time algorithm for recognizing a cashier's actions at the point of sale (POS), which can be further used to analyze cashier behaviors for identifying fraudulent incidents. The algorithm uses a set of simple but effective features derived from a global representation of motion energy called polar motion map (PMM). These features capture the motion patterns exhibited in a cashier's actions as a focused beam of motion energy, characterizing the actions as the extension and retraction movement of the cashier's arm with respect to a prespecified region. Our algorithm demonstrates comparable accuracy against one of the state-of-the-art event recognition techniques while running significantly faster.

In virtually every business context there is a need to establish some form of monitoring system to ensure that employees comply with business processes and policies. Compliance failures range from organized theft to gaps in procedure that can be easily remedied through retraining. It is clearly important for businesses to capture and record these deviations to minimize loss prevention and maximize workplace safety and efficiency. In this workshop, we discuss the growing problem of compliance failure and how our system addresses this problem in a retail context to detect checkout-related fraud through the integration of visual and non-visual data.