Urban environments present unique challenges from the perspective of surveillance and security. Threat activity in urban environments tends to be very similar to background activity, while the volume of activity is often very high. The widespread geographical area presents issues from the perspective of response. These characteristics of urban environments create challenges to traditional applications of video analytics technologies and opens up opportunities for novel approaches. This paper explores the applicability of video analytics in various scenarios presented in urban surveillance situations. We also describe novel technical solutions to some of the challenges of urban surveillance.

A significant portion of retail shrink is attributed to employees and occurs around the point of sale (POS). In this paper, we target a major type of retail fraud in surveillance videos, known as sweethearting (or fake scan), where a cashier intentionally fails to enter one or more items into the transaction in an attempt to get free merchandise for the customer. We first develop a motion-based algorithm to identify video segments as candidates for primitive events at the POS. We then apply spatio-temporal features to recognize true primitive events from the candidates and prune those falsely alarmed. In particular, we learn location-aware event models by Multiple-Instance Learning to address the location-sensitive issues that appear in our problem. Finally, we validate the entire transaction by combining primitive events according to temporal ordering constraints. We demonstrate the effectiveness of our approach on data captured from a real grocery store.

Video analytics have recently emerged as a promising technique of retail fraud detection for loss prevention. Efficient video analytic algorithms are highly desired for a practical fraud detection system. In this paper, we present a real-time algorithm for recognizing a cashier's actions at the point of sale (POS), which can be further used to analyze cashier behaviors for identifying fraudulent incidents. The algorithm uses a set of simple but effective features derived from a global representation of motion energy called polar motion map (PMM). These features capture the motion patterns exhibited in a cashier's actions as a focused beam of motion energy, characterizing the actions as the extension and retraction movement of the cashier's arm with respect to a prespecified region. Our algorithm demonstrates comparable accuracy against one of the state-of-the-art event recognition techniques while running significantly faster.

Vision-based head trackers have been around for some years and are even beginning to be commercialized, but problems remain with respect to usability. Users without the ability to use traditional pointing devices - the intended audience of such systems - have no alternative if the automatic boot strapping process fails, there is room for improvement in face tracking, and the pointer movement dynamics do not support accurate and efficient pointing. This paper describes a novel head tracking pointer that addresses these problems.

Are the object manipulation techniques traditionally used in head-mounted displays (HMDs) applicable to augmented reality based projection systems? This paper examines the differences between HMD- and projector/camera-based AR interfaces in the light of a manipulation task involving documents and applications projected on common office surfaces such as tables, walls, cabinets, and floor. We report a Wizard of Oz study where subjects were first asked to create gesture/voice commands to move 2D objects on those surfaces and then exposed to gestures created by the authors. Among the options, subjects could select the object to be manipulated using voice command; touching, pointing, and grabbing gesture; or a virtual mouse. The results show a strong preference for a manipulation interface based on pointing gestures using small hand movements and involving minimal body movement. Direct touching of the object was also common when the object being manipulated was within the subjects' arm reach. Based on these results, we expect that the preferred interface resembles, in many ways, the egocentric model traditionally used in AR.

Symbiotic displays are intelligent network devices that can be enlisted on demand by mobile devices, offering higher resolutions and larger viewing areas. Users of small mobile devices can consequently map various kinds of information to the displays that are most appropriate for comfortably viewing them. The availability of these displays in public areas, where passersby may glimpse another's sensitive information, introduces a privacy issue. To address this, we suggest a technique that combines the blurring of sensitive words on the public symbiotic display with a user interface that allows these words to be read on the display of a personal mobile device. We describe three ways in which such interactions can be enabled and discuss the architectural trade-offs associated with these approaches. We also describe a prototype implementation of an email reading application utilizing an IBM WatchPad and an Everywhere Display projector. This prototype was demonstrated at PerCom'04

We demonstrate a system that combines steerable projection and computer vision technologies to create "touch-screen" style interactive displays on any flat surface in a space. A high-end version of the system -- the Everywhere Display (ED) -- combines an LCD projector with motorized focus and zoom, a computer controlled pan-tilt mirror, and a pan-tilt zoom camera to enable steering of interactive projections around space. A low-end version (ED-lite) enables creation of interactive displays using a portable projector and camera attached to a laptop computer. Unlike traditional augmented reality systems, the ED systems enable delivery of interactive multimedia content on ordinary objects without requiring users to wear head mounted displays or carry special input devices.

This paper introduces a new class of interactive interfaces that can be moved around to appear on ordinary objects and surfaces anywhere in a space. By dynamically adapting the form, function, and location of an interface to suit the context of the user, such steerable interfaces have the potential to offer radically new and powerful styles of interaction in intelligent pervasive computing spaces. We propose defining characteristics of steerable interfaces and present the first steerable interface system that combines projection, gesture recognition, user tracking, environment modeling and geometric reasoning components within a system architecture. Our work suggests that there is great promise and rich potential for further research on steerable interfaces.

Traditionally, a user has positioned himself/herself to be in front of a display in order to access information from it. In this information age, life at work and even at home is often confined to be in front of a display device that is the source of information or entertainment. The paper introduces another paradigm where the display follows the user rather than the user being tied to the display. We demonstrate how steerable projection and people tracking can be combined to achieve a display that automatically follows the user.

Tracking the face of a computer user as he looks at various parts of the screen is a fundamental tool for a variety of perceptual user interface applications. The authors have developed a simple but surprisingly robust tracking algorithm based on template matching and applied it successfully. This paper describes extensions to that algorithm, which improves performance at large facial rotation angles. The method is based on pre-distorting the single training template using 2D image transformations to simulate 3D facial rotations. The method avoids many of the problems associated with using a complex 3D head model. It is robust to variations in the environment and well-suited to use in practical applications in typical computing environments.

This paper will review the design of a working system that visually recognizes hand gestures for the control of a window based user interface. After an overview of the system, it will explore one aspect of gestural interaction in depth, hand tracking, and what is needed for the user to be able to interact comfortably with on-screen objects. We describe how the location of the hand is mapped to a location on the screen, and how it is both necessary and possible to smooth the camera input using a non-linear physical model of the cursor. The performance of the system is examined, especially with respect to abject selection. We show how a standard HCI model of object selection (Fitts' Law) can be extended to model the selection performance of free-hand pointing.

This work describes the design of a functioning user interface based on visual recognition of hand gestures, and details its performance. In the interface, gesture replaces the mouse for many actions including selecting, moving and resizing windows. A camera below the screen observes the user. The hand is segmented from the background using color. Features of the hand's motion are extracted from the sequence of segmented images, and when needed the hand's pose is classified using a neural net. This information is parsed by a task specific grammar. The system runs in real time on standard PC hardware. It has demonstrated its abilities with various users in several different office environments. Having experimented with a functioning gestural interface, the authors discuss the practicality and best applications of this technology.

This paper describes the techniques used to separate the hand from a cluttered background in a gesture recognition system. Target colors are identified using a histogram-like structure called a Color Predicate, which is trained in real-time using a novel algorithm. Running on standard PC hardware, the segmentation is of sufficient speed and quality to support an interactive user interface. The method has shown its flexibility in a range of different office environments, segmenting users with many different skin-tones. Variations have been applied to other problems including finding face candidates in video sequences.

We present a model for flexible extruded objects, such as wires, tubes, or grommets, and demonstrate a novel, self-adjusting, seven-dimensional Hough transform that derives their diameter and three-space curved axes from position and surface normal information. The method is purely local and is inexpensive to compute. The model considers such objects as piecewise toroidal, and decomposes the seven parameters of a torus into three nested subspaces, the structures of which counteract the errors implicit in the analysis of objects of great size and/or small curvature. We believe it is the first example of a parameter space structure designed to cluster ill-conditioned hypotheses together so that they can be easily detected and ignored. This work complements existing shape-from-contour approaches for analyzing tori: it uses no edge information, and it does not require the solution of high-degree nonlinear equations by iterative techniques.>

A framework for 3D object recognition is presented. Its flexibility and extensibility are accomplished through a uniform, parallel, and modular recognition architecture. Concurrent and stacked parameter transforms reconstruct a variety of features from the input scene. At each stage, constraint satisfaction networks collect and fuse the evidence obtained through the parameter transforms, ensuring a globally consistent interpretation of the input scene and allowing for the integration of diverse types of information. The final interpretation of the scene is a small consistent subset of the many initial hypotheses about partial features, primitive features, feature assemblies, and 3D objects computed by the various parameter transforms. A complete, integrated, and implemented system that extracts planar surfaces, patches of quadrics of revolution, and planar intersection curves of these surfaces from a depth map viewing 3D objects is described. Experimental results on the recognition behavior of the system are presented.>

A novel approach is presented for pruning the amount of search needed to match image features to object models. The technique relies on active networks which capture various visibility and geometric constraints between features of a model to prune these features from search space during matching. The networks, which can be efficiently implemented in Boolean logic, integrate harmoniously with the previous work in feature recognition and object matching. A method is proposed for clustering model features (vsets) and four types of constraints which assist in building the networks. The authors show, both analytically and empirically, the dramatic reduction in search provided by activation nets.>

A general framework for multiresolution visual recognition is introduced. The input is processed simultaneously at a coarse resolution throughout the image and at finer resolution within a small window. An approach for controlling the movement of the high-resolution window is described which allows for the unification of a variety of data and model-driven behavioral paradigms. Three modes have been implemented, one based on large unexplained areas in the data, one on conflicts in the object-model database, and one on a 2D space-filling algorithm. It is argued that this kind of multiresolution processing not only is useful in limiting the computational time, but also can be a deciding factor in making the entire vision problem a tractable and stable one. To demonstrate the approach, a class of 3D surface textures is introduced as a feature for recognition in the system considered. Surface texture recognition typically requires higher-resolution processing than required for the extraction of the underlying surface. As an example, surface texture is used to discriminate between a ping-pong ball and a golf ball.>

A vision system to recognize 3-D objects is presented. A novel notion of generalized feature makes it possible to develop a homogeneous architecture to support recognition from simple partial features to complex feature assemblies and 3-D objects. Layered concurrent parameter transforms vote for feature hypotheses on the basis of image data and previously reconstructed features. Recognition networks, motivated by connectionist systems, collect votes, fuse evidence from various sources and ensure global consistency. In addition, they provide an integrated solution to the segmentation problem. The highly modular design allows fundamentally different types of features to interact in a coherent way, utilizing redundancies for more robust recognition. Within this paradigm, a system extracts planar patches, patches of quadrics of revolution, and the intersection curves of these surfaces (lines and conic sections in three-space) from a depth map. Reconstructed features index into a model database to form consistent object hypotheses. Experimental results detailing the recognition behavior or real depth maps are included.>