Data privacy is a growing concern as more cloud-based solutions for extracting insights from images are available. Fully Homomorphic Encryption (FHE) is one of the state-of-the-art techniques to enable privacy preserving machine learning. However, encrypted deep neural network-based inference methods face the fundamental challenge of optimizing computational depth and complexity to achieve an acceptable trade-off between accuracy and practical feasibility. Existing works only report high level implementations without providing rigorous analysis of the intricacies involved in the FHE implementation of generic primitive operators of a convolutional neural network (CNN). In this paper, we use the CKKS encryption scheme available in the open-source HElib library to run encrypted inference experiments on the MNIST dataset. The experiments indicate that efficient ciphertext packing schemes, model optimization and multi-threading strategies play a critical role in determining the throughput and latency of the inference process. We also show that operational parameters of the chosen FHE scheme such as the degree of the cyclotomic polynomial, depth limitations of the underlying leveled HE scheme, and the computational precision parameters result in significant trade-offs between accuracy, security level and computational time of the machine learning model. The key contribution of the paper is the analysis and recommendation of optimization techniques for efficient encrypted CNN inference.

We present a remote sensing pipeline that processes LiDAR (Light Detection And Ranging) data through machine & deep learning for the application of archeological feature detection on big geo-spatial data platforms such as e.g. IBM PAIRS Geoscope [1], [2].Today, archeologists get overwhelmed by the task of visually surveying huge amounts of (raw) LiDAR data in order to identify areas of interest for inspection on the ground. We showcase a software system pipeline that results in significant savings in terms of expert productivity while missing only a small fraction of the artifacts.Our work employs artificial neural networks in conjunction with an efficient spatial segmentation procedure based on domain knowledge. Data processing is constraint by a limited amount of training labels and noisy LiDAR signals due to vegetation cover and decay of ancient structures. We aim at identifying geo-spatial areas with archeological artifacts in a supervised fashion allowing the domain expert to flexibly tune parameters based on her needs.

The assessment of the vegetation growth rate given remote sensing data is a challenging task in the Earth Observation sciences. LiDAR data acquisition is commonly used to extract height information at a given moment in time, however, the associated cost and complexity restrict continuous acquisitions. Frequently captured aerial imagery can be used to identify and separate vegetation from bare land, water, impervious surface, or built infrastructure. A combination of LiDAR data with aerial and radar imagery allows to track dynamic seasonal growth of vegetation around critical infrastructure such as power lines. We present a general framework that integrates tree identification and growth assessment around power lines with the goal to identify locations of high risk where trees potentially cause power outages.

While significant advancements have been made in the field of multimedia forensics to detect altered content, existing techniques mostly focus on enabling the content recipient to verify the content integrity without any inputs from the content creator. In many application scenarios, the creator has a strong incentive to establish the provenance and integrity of the multimedia data created and released by him. Hence, there is a strong need for mechanisms that allow the content creator to prove the authenticity of the released content. Since blockchain technology provides an immutable distributed database, it is an ideal solution for reliably time-stamping content with its creation time and storing an irrefutable signature of the content at the time of its creation. However, a simple digital signature scheme does not allow modification of the content after the initial commitment. Authorized multimedia content alteration by its creator is often necessary (e.g., redaction of faces to protect the privacy of individuals in a video, redaction of sensitive fields in a text document) before the content is distributed. The main contributions of this paper are: (i) a novel sanitizable signature scheme that enables the content creator to prove the integrity of the redacted content, while preventing the recipients from reconstructing the redacted segments based on the published commitment, and (ii) a blockchain-based solution for securely managing the sanitizable signature. The proposed solution employs a robust hashing scheme using chameleon hash function and Merkle tree to generate the initial signature, which is stored on the blockchain. The auxiliary data required for the integrity verification step is retained by the content creator and only a signature of this auxiliary data is stored on the blockchain. Any modifications to the multimedia content requires only updating the signature of the auxiliary data, which is securely recorded on the blockchain. We demonstrate that the proposed approach enables verification of integrity of redacted multimedia content without compromising the content privacy requirements.

A personal or enterprise collection of a large set of face images may contain many types of tags used for querying the collection. Often the tags have many irrelevant content that may not reflect the image content in terms of the facial characteristics. In this paper, we propose a data curation method to filter out the irrelevant face images using a face recognition based subgraph identification. Results on retrievals from the Internet using popular celebrities show the efficacy of our approach after we cleanse the images collection retrieved and applying our algorithm to the collection.

Often deep learning methods are associated with huge amounts of training data. The deeper the network gets, the larger is the need for training data. A large amount of labeled data helps the network learn about the variations it needs to handle in the prediction stage. It is not easy for everyone to get access to huge amounts of labeled data leaving a few to have the luxury to design very deep networks. In this paper, we propose to flatten the disparity by using the modeling methods to minimize the need for huge amounts of data for training a deep network. Using face recognition as an example, we demonstrate how limited labeled data can be leveraged to obtain near state of the art performance with generalization capability across multiple databases. In addition, we show that the normalization in the overall network can improve the speed and resource requirement for the prediction/inferencing stage.

Exploiting contextual cues has been a key idea to improve people detection in crowded scenes. Along this line we present a novel context-driven approach to detect people in crowded scenes. Based on a context graph that incorporates both geometric and social contextual patterns in crowds, we apply label propagation to discover weak detections contextually compatible with true detections while suppressing irrelevant false alarms. Compared to previous approaches for context modeling limited to only pairwise spatial interactions between local object neighbors, our approach provides a more effective way to model people interactions in a global context. Our approach achieves performance comparable to state of the art on two challenging datasets for people and pedestrian detection.

We address the problem of 24/7 object detection in urban surveillance videos, which presents unique challenges due to significant object appearance variations caused by lighting effects such as shadows and specular reflections, object pose variation, multiple weather conditions, and different times of the day. Rather than training a generic detector and adapting its parameters over time to handle all these variations, we rely on a large set of complementary and extremely efficient detector models, covering multiple overlapping appearance subspaces. At run time, our method continuously selects the most suitable detectors for a given scene and condition, using a novel approach inspired by parametric background modeling algorithms. We provide a comprehensive experimental analysis to show the effectiveness of our approach, considering traffic monitoring as our application domain. Our system runs at 100 frames per second on a standard laptop computer.

The goal of image stitching is to create natural-looking mosaics free of artifacts that may occur due to relative camera motion, illumination changes, and optical aberrations. In this paper, we propose a novel stitching method, that uses a smooth stitching field over the entire target image, while accounting for all the local transformation variations. Computing the warp is fully automated and uses a combination of local homography and global similarity transformations, both of which are estimated with respect to the target. We mitigate the perspective distortion in the non-overlapping regions by linearizing the homography and slowly changing it to the global similarity. The proposed method is easily generalized to multiple images, and allows one to automatically obtain the best perspective in the panorama. It is also more robust to parameter selection, and hence more automated compared with state-of-the-art methods. The benefits of the proposed approach are demonstrated using a variety of challenging cases.

In this paper, we propose a hierarchical computational system architecture to support the target domain of realtime mobile computing in the context of unmanned aerial vehicles (UAVs). The overall architectural vision includes support for system resilience in the presence of uncertainties in the operational environment of surveillance UAVs. We report measurement-based results that are obtained from a UAV proxy demonstration apparatus. The apparatus consists of a Raspberry Pi (RPi) board that serves as an on-board UAV computer, working with support from a laptop that serves as the on-ground computing infrastructure where an operator "consumes" video information received from the UAV. We quantify the gap between the on-board UAV camera frame rate (input) and the on-ground operator-observed frame rate (output) for a specialized class of computer vision applications germane to the UAV-based aerial surveillance domain. The goal is to keep the frame rate observed by the ground operator as close (or ideally equal) to the on-board UAV camera frame rate (i.e. to preserve the real-time aspect) despite the unstable bandwidth availability in the channel connecting both ends. The proposed hierarchical approach significantly outperforms two considered baselines: one in which computation takes place entirely on the UAV computer and another in which computation takes place entirely on the ground. This improved performance is due to a more balanced resource sharing between the on-board UAV computer and UAV-to-ground communication channel. Later, we show how the observed frame rate improves when the RPi board is replaced with an NVIDIA Jetson TK1 board. Based on the observations gleaned from these "proxy" experiments, we sketch the fundamentals of our ongoing work in model-based predictive analysis of resilient "UAV swarm" computational architectures of the future.

The importance of the internet-of-things (IOT) is now an established reality. With that backdrop, the phenomenal emergence of cameras/sensors mounted on unmanned aerial, ground and marine vehicles (UAVs, UGVs, UMVs) and body worn cameras is a notable new development. The swarms of cameras and real-time computing thereof are at the heart of new technologies like connected cars, drone-based city-wide surveillance and precision agriculture, etc. Smart computer vision algorithms (with or without dynamic learning) that enable object recognition and tracking, supported by baseline video content summarization or 2D/3D image reconstruction of the scanned environment are at the heart of such new applications. In this article, we summarize our recent innovations in this space. We focus primarily on algorithms and architectural design considerations for video summarization systems.

It is estimated that less than five percent of videos are currently analyzed to any degree. In addition to petabyte-sized multimedia archives, continuing innovations in optics, imaging sensors, camera arrays, (aerial) platforms, and storage technologies indicates that for the foreseeable future existing and new applications will continue to generate enormous volumes of video imagery. Contextual video summarizations and activity maps offers one innovative direction to tackling this Big Data problem in computer vision. The goal of this work is to develop semi-automatic exploitation algorithms and tools to increase utility, dissemination and usage potential by providing quick dynamic overview geospatial mosaics and motion maps. We present a framework to summarize (multiple) video streams from unmanned aerial vehicles (UAV) or drones which have very different characteristics compared to structured commercial and consumer videos that have been analyzed in the past. Using both metadata geospatial characteristics of the video combined with fast low-level image-based algorithms, the proposed method first generates mini-mosaics that can then be combined into geo-referenced meta-mosaics imagery. These geospatial maps enable rapid assessment of hours long videos with arbitrary spatial coverage from multiple sensors by generating quick look imagery, composed of multiple mini-mosaics, summarizing spatiotemporal dynamics such as coverage, dwell time, activity, etc. The overall summarization pipeline was tested on several DARPA Video and Image Retrieval and Analysis Tool (VIRAT) datasets. We evaluate the effectiveness of the proposed video summarization framework using metrics such as compression and hours of viewing time.

Action recognition is an important problem in computer vision and has received substantial attention in recent years. However, it remains very challenging due to the complex interaction of static and dynamic information, as well as the high computational cost of processing video data. This paper aims to apply the success of static image semantic recognition to the video domain, by leveraging both static and motion based descriptors in different stages of the semantic ladder. We examine the effects of three types of features: low-level dynamic descriptors, intermediate-level static deep architecture outputs, and static high-level semantics. In order to combine such heterogeneous sources of information, we employ a scalable method to fuse these features. Through extensive experimental evaluations, we demonstrate that the proposed framework significantly improves action classification performance. We have obtained an accuracy of 89.59% and 62.88% on the well-known UCF-101 and HMDB-51 benchmarks, respectively, which compare favorably with the state-of-the-art.

We develop a robust approach to detect parked vehicles in real time. Our approach particularly focuses on tracking vehicles in long term under challenging conditions such as lighting changes and occlusions. Vehicle tracking is performed by template matching based on fast-computed corner points. The template model is made self-adaptive over time to accommodate lighting changes. We also present an effective way to manage and track multiple vehicles when they are parked close together and occlude one another. We demonstrate the effectiveness of our approach on the challenging i-LIDs data set and another large one collected from real-world scenarios.

We present a novel approach for event detection in video by temporal sequence modeling. Exploiting temporal information has lain at the core of many approaches for video analysis (i.e., action, activity and event recognition). Unlike previous works doing temporal modeling at semantic event level, we propose to model temporal dependencies in the data at sub-event level without using event annotations. This frees our model from ground truth and addresses several limitations in previous work on temporal modeling. Based on this idea, we represent a video by a sequence of visual words learnt from the video, and apply the Sequence Memoizer [21] to capture long-range dependencies in a temporal context in the visual sequence. This data-driven temporal model is further integrated with event classification for jointly performing segmentation and classification of events in a video. We demonstrate the efficacy of our approach on two challenging datasets for visual recognition.

User authentication in the context of a secure transaction needs to be continuously evaluated for the risks associated with the transaction authorization. The situation becomes even more critical when there are regulatory compliance requirements. Need for such systems have grown dramatically with the introduction of smart mobile devices which make it far easier for the user to complete such transaction quickly but with a huge exposure to risk. Biometrics can play a very significant role in addressing such problems as a key indicator of the user identity and thus reducing the risk of fraud. While unimodal biometrics authentication systems are being increasingly experimented by mainstream mobile system manufacturers (e.g., fingerprint in iOS), we explore various opportunities of reducing risk in a multimodal biometrics system. The multimodal system is based on fusion of several biometrics combined with a policy manager. A new biometric modality: chirography which is based on user writing on multi-touch screens using their finger is introduced. Coupling with chirography, we also use two other biometrics: face and voice. Our fusion strategy is based on inter-modality score level fusion that takes into account a voice quality measure. The proposed system has been evaluated on an in-house database that reflects the latest smart mobile devices. On this database, we demonstrate a very high accuracy multi-modal authentication system reaching an EER of 0.1% in an office environment and an EER of 0.5% in challenging noisy environments.

Detecting human behaviors in vast amounts of video is a challenging task in a variety of real-world applications. Thus an interactive tool designed to support this task with human in the loop is of significance in various domains including public safety and security. In this paper, we design and develop an interactive visual analytics system, RiskWheel, that enables effective analysis of detection results and utilization of user feedback to improve surveillance event detection. In particular, we propose 1) an interactive approach to visualize data with temporal relations and 2) a novel risk ranking method to differentiate detection results and present more informative ones to the user for better interaction. In our experiments, we demonstrate RiskWheel through a case study on TRECVID Surveillance Event Detection (SED) task [1]. The experimental results quantitatively show that RiskWheel outperforms multiple baselines, demonstrating the power of the risk ranking technique.

In this paper, we show how pedestrian detection accuracy and efficiency can be improved for static surveillance cameras using scene context and temporal non-maximal suppression. First, using the geometry of the scene, we derive the relationship between height in the image with respect to height in the real-world. This relationship is used to learn the range of scales to evaluate for potential detections. The geometry can also be used to estimate distances on the ground plane and to predict the image distance for pedestrian movement. Secondly, we show the error inherent in the standard non-maximum suppression method and demonstrate how the error can be reduced using a small temporal window. The scene context information and temporal non-maximum suppression (tNMS) can be applied to any detection algorithm. We evaluate the accuracy of each method across seven different videos. We show the scene context information and temporal non-maximum suppression improve accuracy on the order of ten and five percent respectively and over fifteen percent combined for seven different scenes for two publicly available detectors. Furthermore, both these approaches significantly reduce computational cost.

Practical surveillance systems deployed in urban scenarios need to operate 24/7 under a wide range of environmental conditions. As modern video analytics shift from blob-based to object-centered architectures, appearance-based object detection under different weather conditions and lighting effects emerges as a critical yet largely unaddressed problem. This paper investigates this research topic, using as a case study the problem of vehicle detection in urban surveillance environments. In particular, we show that a simple and efficient Winsorized lighting correction technique improves performance significantly when outliers due to shadows, specularities, headlights, and occluders are present. Moreover, we demonstrate that a self-training mechanism utilizing a balanced training set automatically acquired from the target domain yields superior performance. Our experimental results are carried out on a novel dataset of vehicle images collected from a public traffic camera and categorized according to multiple environmental conditions.

We address the problem of attribute-based people search in real surveillance environments. The system we developed is capable of answering user queries such as "show me all people with a beard and sunglasses, wearing a white hat and a patterned blue shirt, from all metro cameras in the downtown area, from 2pm to 4pm last Saturday". In this paper, we describe the lessons we learned from practical deployments of our system, and how we made our algorithms achieve the accuracy and efficiency required by many police departments around the world. In particular, we show that a novel set of multimodal integral filters and proper normalization of attribute scores are critical to obtain good performance. We conduct a comprehensive experimental analysis on video footage captured from a large set of surveillance cameras monitoring metro chokepoints, in both crowded and normal activity periods. Moreover, we show impressive results using images from the recent Boston marathon bombing event, where our system can rapidly retrieve the two suspects based on their attributes from a database containing more than one thousand people present at the event.

In this paper, we present a real-time automatic vision-based rail inspection system, which performs inspections at 16 km/h with a frame rate of 20 fps. The system robustly detects important rail components such as ties, tie plates, and anchors, with high accuracy and efficiency. To achieve this goal, we first develop a set of image and video analytics and then propose a novel global optimization framework to combine evidence from multiple cameras, Global Positioning System, and distance measurement instrument to further improve the detection performance. Moreover, as the anchor is an important type of rail fastener, we have thus advanced the effort to detect anchor exceptions, which includes assessing the anchor conditions at the tie level and identifying anchor pattern exceptions at the compliance level. Quantitative analysis performed on a large video data set captured with different track and lighting conditions, as well as on a real-time field test, has demonstrated very encouraging performance on both rail component detection and anchor exception detection. Specifically, an average of 94.67% precision and 93% recall rate has been achieved for detecting all three rail components, and a 100% detection rate is achieved for compliance-level anchor exception with three false positives per hour. To our best knowledge, our system is the first to address and solve both component and exception detection problems in this rail inspection area.

Several recent investigations attempt to classify vehicles into a small number (5-7) of colors. A significant complication arises, however; a large proportion of vehicles (>50%) are various shades of gray: white, black, silver, gray, and variations such as gun metal and pearly white. Distinguishing such shades of gray in vehicle body color from lighting changes is an unsolved problem. Furthermore, previous studies have evaluated their performance on private datasets precluding a comparison of methodologies. In this paper, we release a public dataset with ground truth color classification for future evaluations and comparisons based on the publicly available i-LIDS data [9]. We describe a method to perform vehicle color classification into 7 frequently occurring colors including dark red, dark blue and light silver, using pose dependent vehicle detection, vehicle alignment, and vehicle body part masks. We introduce new features for tree-based vehicle color classification based on the reliability of color information and the relative color of various vehicle parts.

Effective reduction of false alarms in large-scale video surveillance is rather challenging, especially for applications where abnormal events of interest rarely occur, such as abandoned object detection. We develop an approach to prioritize alerts by ranking them, and demonstrate its great effectiveness in reducing false positives while keeping good detection accuracy. Our approach benefits from a novel representation of abandoned object alerts by relative attributes, namely static ness, foreground ness and abandonment. The relative strengths of these attributes are quantified using a ranking function[19] learnt on suitably designed low-level spatial and temporal features. These attributes of varying strengths are not only powerful in distinguishing abandoned objects from false alarms such as people and light artifacts, but also computationally efficient for large-scale deployment. With these features, we apply a linear ranking algorithm to sort alerts according to their relevance to the end-user. We test the effectiveness of our approach on both public data sets and large ones collected from the real world.

We present a generic event detection system evaluated in the Surveillance Event Detection (SED) task of TRECVID 2012. We investigate a statistical approach with spatio-temporal features applied to seven event classes, which were defined by the SED task. This approach is based on local spatio-temporal descriptors, called MoSIFT and generated by pair-wise video frames. A Gaussian Mixture Model(GMM) is learned to model the distribution of the low level features. Then for each sliding window, the Fisher vector encoding [improvedFV] is used to generate the sample representation. The model is learnt using a Linear SVM for each event. The main novelty of our system is the introduction of Fisher vector encoding into video event detection. Fisher vector encoding has demonstrated great success in image classification. The key idea is to model the low level visual features as a Gaussian Mixture Model and to generate an intermediate vector representation for bag of features. FV encoding uses higher order statistics in place of histograms in the standard BoW. FV has several good properties: (a) it can naturally separate the video specific information from the noisy local features and (b) we can use a linear model for this representation. We build an efficient implementation for FV encoding which can attain a 10 times speed-up over real-time. We also take advantage of non-trivial object localization techniques to feed into the video event detection, e.g. multi-scale detection and non-maximum suppression. This approach outperformed the results of all other teams submissions in TRECVID SED 2012 on four of the seven event types.

We present a novel approach to automatically create efficient and accurate object detectors tailored to work well on specific video surveillance cameras (specific-domain detectors), using samples acquired with the help of a more expensive, general-domain detector (trained using images from multiple cameras). Our method requires no manual labels from the target domain. We automatically collect training data using tracking over short periods of time from high-confidence samples selected by the general-domain detector. In this context, a novel confidence measure is proposed for detectors based on a cascade of classifiers, which are frequently adopted for computer vision applications that require real-time processing. We demonstrate our proposed approach on the problem of vehicle detection in crowded surveillance videos, showing that an automatically generated detector significantly outperforms the original general-domain detector with much less feature computations.

We present a robust system for large-scale abandoned object detection (AOD) with low false positive rates and good detection accuracy under complex realistic scenarios. The robustness of our system is largely attributed to an approach we develop for foreground analysis, which can effectively differentiate foreground objects from background under challenging conditions such as lighting changes, low textureness and low contrast as well as cluttered background. This significantly eliminates false positives caused by lighting changes while retaining true drops better. We further perform abandonment analysis to reduce more false positives including those related to people, at a small cost of accuracy (≤ 2%). We demonstrate the effectiveness of our approach on two large data sets collected in various challenging scenes, providing detailed analysis of experiments.

Substantial ambiguities arise in hand tracking due to issues such as small hand size, deformable hand shapes and similar hand appearances. These issues have greatly limited the capability of current multi-target tracking techniques in hand tracking. As an example, state-of-the-art approaches for people tracking handle indentity switching by exploiting the appearance cues using advanced object detectors. For hand tracking, such approaches will fail due to similar, or even identical hand appearances. The main contribution of our work is a global optimization framework based on binary quadratic programming (BQP) that seamlessly integrates appearance, motion and complex interactions between hands. Our approach effectively handles key challenges such as occlusion, detection failure, identity switching, and robustly tracks both hands in two challenging real-life scenarios: retail surveillance and sign languages. In addition, we demonstrate that an automatic method based on hand trajectory analysis outperforms state-of-the-art on checkout-related activity recognition in grocery stores.

We address the problem of learning robust and efficient multi-view object detectors for surveillance video indexing and retrieval. Our philosophy is that effective solutions for this problem can be obtained by learning detectors from huge amounts of training data. Along this research direction, we propose a novel approach that consists of strategically partitioning the training set and learning a large array of complementary, compact, deep cascade detectors. At test time, given a video sequence captured by a fixed camera, a small number of detectors is automatically selected per image location. We demonstrate our approach on the problem of vehicle detection in challenging surveillance scenarios, using a large training dataset composed of around one million images. Our system runs at an impressive average rate of 125 frames per second on a conventional laptop computer.

Color information is an important feature for many vision algorithms including color correction, image retrieval and tracking. In this paper, we study the limitations of color measurement accuracy and explore how this information can be used to improve the performance of color correction. In particular, we show that a strong correlation exists between the error in hue measurements on one hand and saturation and intensity on the other hand. We introduce the notion of color strength, which is a combination of saturation and intensity information to determine when hue information in a scene is reliable. We verify the predictive capability of this model on two different datasets with ground truth color information. Further, we show how color strength information can be used to significantly improve color correction accuracy for the 11K real-world SFU gray ball dataset.

For safety purposes, railroad tracks need to be inspected on a regular basis for physical defects or design noncompliances. Such track defects and non-compliances, if not detected in a timely manner, may eventually lead to grave consequences such as train derailments. In this paper, we present a real-time automatic vision-based rail inspection system, with main focus on anchors - an important rail component type, and anchor-related rail defects, or exceptions. Our system robustly detects important rail components including ties, tie plates, anchors with high accuracy and efficiency. Detected objects are then consolidated across video frames and across camera views to map to physical rail objects, by combining the video data streams from all camera views with GPS information and speed information from the distance measuring instrument (DMI). After these rail components are detected and consolidated, further data integration and analysis is followed to detect sequence-level track defects, or exceptions. Quantitative analysis performed on a real online field test conducted on different track conditions demonstrates that our system achieves very promising performance in terms of rail component detection, anchor condition assessment, and compliance-level exception detection. We also show that our system outperforms another advanced rail inspection system in anchor detection.

In retail stores, cashier non-compliance activities at the Point of Sale (POS) are one of the prevalent sources of retail loss. In this paper, we propose a novel approach to reliably rank the list of detected non-compliance activities of a given retail surveillance system, thereby provide a means of significantly reducing the false alarms and improving the precision in non-compliance detection. Our approach represents each detected non-compliance activity using multi-modal features coming from video data, transaction logs (TLog) data and intermediate results of the video analytics. We then learn a binary classifier that successfully separate true positives and false positives in a labeled training set. A confidence score for each detection can then be computed using the decision value of the trained classifier, and a ranked list of detections can be formed based on this score. The benefit from having this ranked list is two-fold. First, a large number of false alarms can be avoided by simply keeping the top part of the list and discarding the rest. Second, a trade off between precision and recall can easily be performed by sliding the discarding threshold along this ranked list. Experimental results on a large scale dataset captured from real stores demonstrate that our approach achieves better precision than a state-of-the-art system at the same recall. Our approach can also reach an operating point that exceeds the retailers' expectation in terms of precision, while retaining an acceptable recall of more than 60%.

We present a novel approach for visual detection and attribute-based search of vehicles in crowded surveillance scenes. Large-scale processing is addressed along two dimensions: 1) large-scale indexing, where hundreds of billions of events need to be archived per month to enable effective search and 2) learning vehicle detectors with large-scale feature selection, using a feature pool containing millions of feature descriptors. Our method for vehicle detection also explicitly models occlusions and multiple vehicle types (e.g., buses, trucks, SUVs, cars), while requiring very few manual labeling. It runs quite efficiently at an average of 66 Hz on a conventional laptop computer. Once a vehicle is detected and tracked over the video, fine-grained attributes are extracted and ingested into a database to allow future search queries such as “Show me all blue trucks larger than 7 ft. length traveling at high speed northbound last Saturday, from 2 pm to 5 pm”. We perform a comprehensive quantitative analysis to validate our approach, showing its usefulness in realistic urban surveillance settings.

We propose a robust approach for abandoned object detection in urban surveillance with over thousands of cameras. For such a large-scale monitoring based on intelligent video analysis, it is critical that a system be designed with careful control of false alarms. Our approach is based on proactive modeling of temporally static objects (TSO) such as cars stopping at red light and still pedestrians in the street. We develop a finite state machine to track the entire life cycles of TSOs from creation to termination. The semantically meaningful object information provided by the state machine in turn allows adaptive region-level updating of the background model without using any sophisticated object classification techniques. We demonstrate that our approach significantly mitigates the problematic issue of false alarm related to people in city surveillance, using both a small publicly available data set and a large one collected from various realistic urban scenarios.

Cashiers in retail stores usually exhibit certain repetitive and periodic activities when processing items. Detecting such activities plays a key role in most retail fraud detection systems. In this paper, we propose a highly efficient, effective and robust vision technique to detect checkout-related primitive activities, based on a hierarchical finite state machine (FSM). Our deterministic approach uses visual features and prior spatial constraints on the hand motion to capture particular motion patterns performed in primitive activities. We also apply our approach to the problem of retail fraud detection. Experimental results on a large set of video data captured from retail stores show that our approach, while much simpler and faster, achieves significantly better results than state-of-the-art machine learning-based techniques both in detecting checkout-related activities and in detecting checkout-related fraudulent incidents.

We propose a robust abandoned object detection algorithm for real-time video surveillance. Different from conventional approaches that mostly rely on pixel-level processing, we perform region-level analysis in both background maintenance and static foreground object detection. In background maintenance, region-level information is fed back to adaptively control the learning rate. In static foreground object detection, region-level analysis double-checks the validity of candidate abandoned blobs. Attributed to such analysis, our algorithm is robust against illumination change, "ghosts" left by removed objects, distractions from partially static objects, and occlusions. Experiments on nearly 130,000 frames of i-LIDS dataset show the superior performance of our approach.

We describe a system that controls whether the headlights of a vehicle are in the highbeam or lowbeam state based on input from a forward looking video camera. The core of the system relies on conventional computer vision techniques, albeit with a sophisticated spot finder front-end. Despite this architecture we are able to use an automated supervised learning technique to tune the system to yield high performance. Using a customer-imposed metric we present both in-car and off-line results from our system along with several competitors, and investigate the system's performance under different weather conditions.

This paper describes our recent work on developing an intelligent headlight control system using machine learning-based approaches. Specifically, such a system aims to automatically control a vehicle's beam state (high beam or low beam) during a night-time drive based on the detection of oncoming/overtaking/leading traffics as well as urban areas from the videos captured by a camera. Two machine learning-based approaches, namely, support vector machine (SVM) and AdaBoost, have been applied to accomplish this task. The architect of each approach, as well as its detailed processing modules, will be elaborated in the paper. The system has been extensively tested both online and offline to validate the robustness and effectiveness of the two proposed approaches. A detailed performance study along with some comparisons between the two approaches will be reported at the end.

A major source of revenue shrink in retail stores is the intentional or unintentional failure of proper checking out of items by the cashier. More recently, a few automated surveillance systems have been developed to monitor cashier lanes and detect non-compliant activities such as fake item checkouts or scans done with the intention of deriving monetary benefit. These systems use data from surveillance video cameras and transaction logs (TLog) recorded at the Point-of-Sale (POS). In this paper, we present a pattern discovery based approach to detect fraudulent events at the POS. Our approach is based on mining time-ordered text streams, representing retail transactions, formed from a combination of visually detected checkout related activities called primitives and barcodes from TLog data. Patterns representing single item checkouts, i.e. anchored around a single barcode, are discovered from these text streams using an efficient pattern discovery technique called Teiresias. The discovered patterns are used to build models for true and fake item scans by retaining or discarding the anchoring barcodes in those patterns respectively. A pattern matching and classification scheme is designed to robustly detect non-compliant cashier activities in the presence of noise in either the TLog or the video data. Different weighting schemes for quantifying the relative importance of the discovered patterns are explored: Frequency, Support Vector Machine (SVM) and Frequency+SVM. Using a large scale dataset recorded from retail stores, our approach discovers semantically meaningful cashier scan patterns. Our experiments also suggest that different weighting schemes result in varied false and true positive performances on the task of fake scan detection.

We present a novel application for searching for vehicles in surveillance videos based on semantic attributes. At the interface, the user specifies a set of vehicle characteristics (such as color, direction of travel, speed, length, height, etc.) and the system automatically retrieves video events that match the provided description. A key differentiating aspect of our system is the ability to handle challenging urban conditions such as high volumes of activity and environmental factors. This is achieved through a novel multi-view vehicle detection approach which relies on what we call motionlet classifiers, i.e. classifiers that are learned with vehicle samples clustered in the motion configuration space. We employ massively parallel feature selection to learn compact and accurate motionlet detectors. Moreover, in order to deal with different vehicle types (buses, trucks, SUVs, cars), we learn the motionlet detectors in a shape-free appearance space, where all training samples are resized to the same aspect ratio, and then during test time the aspect ratio of the sliding window is changed to allow the detection of different vehicle types. Once a vehicle is detected and tracked over the video, fine-grained attributes are extracted and ingested into a database to allow future search queries such as "Show me all blue trucks larger than 7ft length traveling at high speed northbound last Saturday, from 2pm to 5pm".

In this paper, we present our latest research engagement with a railroad company to apply machine vision technologies to automate the inspection and condition monitoring of railroad tracks. Specifically, we have proposed a complete architecture including imaging setup for capturing multiple video streams, important rail component detection such as tie plate, spike, anchor and joint bar bolt, defect identification such as raised spikes, defect severity analysis and temporal condition analysis, and long-term predictive assessment. This paper will particularly present various video analytics that we have developed to detect rail components, which form the building block of the entire framework. Our preliminary performance study has achieved an average of 98.2% detection rate, 1.57% false positive rate and 1.78% false negative rate on the component detection. Finally, with the lack of sufficient representative data and annotations to evaluate system performance on exception detection at both sequence and compliance levels, we proposed a mathematical modeling approach to calculate the probabilities of detecting such exceptions. Such analysis shows that there is still big room for us to improve our approaches in order to achieve desired false positive rate and miss detection rate at the sequence level.

Many modern retail stores have self-checkout stations where customers can ring up their own orders without the assistance of any store personnel. To promote customer honesty these systems often weigh each item as it is placed in the bag to confirm that it has the expected mass for the product scanned. In our system we augment this basic check with an assessment of the item's visual appearance to further ensure that the correct code has been entered.

We present a novel approach for vehicle detection in urban surveillance videos, capable of handling unstructured and crowded environments with large occlusions, different vehicle shapes, and environmental conditions such as lighting changes, rain, shadows, and reflections. This is achieved with virtually no manual labeling efforts. The system runs quite efficiently at an average of 66Hz on a conventional laptop computer. Our proposed approach relies on three key contributions: (1) a co-training scheme where data is automatically captured based on motion and shape cues and used to train a detector based on appearance information; (2) an occlusion handling technique based on synthetically generated training samples obtained through Poisson image reconstruction from image gradients; (3) massively parallel feature selection over multiple feature planes which allows the final detector to be more accurate and more efficient. We perform a comprehensive quantitative analysis to validate our approach, showing its usefulness in realistic urban surveillance settings.

In this paper, we first present the architecture of an intelligent headlight control (IHC) system that we developed in our earlier work. This IHC system aims to automatically control a vehicle's beam state (high beam or low beam) during a night-time drive. A three-level decision framework built around a support vector machine (SVM) learning engine is then briefly discussed. Next, we switch our focus to the study of system performance by varying the SVM feature set, as well as by exploiting various SVM training options and adjustments through a set of experiments. We believe that what we learned from this performance study can provide readers useful guidelines on extracting effective SVM features within the IHC problem domain, as well as on training an effective SVM learning engine for more generalized applications.

We propose a novel approach for enhancing precision in a leading video analytics system that detects cashier fraud in grocery stores for loss prevention. While intelligent video analytics has recently become a promising means of loss prevention for retailers, most of the real-world systems suffer from a large number of false alarms, resulting in a significant waste of human labor during manual verification. Our proposed approach starts with the candidate fraudulent events detected by a state-of-the-art system. Such fraudulent events are a set of visually recognized checkout-related activities of the cashier without barcode associations. Instead of conducting costly video analysis, we extract a few keyframes to represent the essence of each candidate fraudulent event, and compare those keyframes to identify whether or not the event is a valid check-out process that involves consistent appearance changes on the lead-in belt, the scan area and the take-away belt. Our approach also performs a margin-based soft classification so that the user could trade off between saving human labor and preserving high recall. Experiments on days of surveillance videos collected from real grocery stores show that our algorithm can save about 50% of human labor while preserving over 90% of true alarms with small computational overhead.

We study the problem of event detection from realistic videos with repetitive sequential human activities. Despite the large body of work on event detection and recognition, very few have addressed low-quality videos captured from realistic environments. Our framework is based on solving the shortest path on a temporal-event graph constructed from the video content. Graph vertices correspond to detected event primitives, and edge weights are set according to generic knowledge of the event patterns and the discrepancy between event primitives based on a greedy matching of their visual features. Experimental results on videos collected from a retail environment validate the usefulness of the proposed approach.

Visual compliance has emerged as a new paradigm to ensure that employees comply with processes and policies in a business context. In this paper, we focus on videos from retail stores and formulate a binary integer program for detecting checkout events, which enables enforcing visual compliance in such an environment. The proposed integer program maximizes the essential quantities that characterize true events of interest, subject to an array of constraints. In particular, the binary decision variables correspond to the presence of a set of hypothesized visual events. In the objective function, the binary variables are weighted by quality measures derived from infinite Gaussian mixture modeling of the video content, such that maximizing the overall quality measure is expected to uncover the meaningful visual events. Our framework is tested and validated on videos recorded at checkout lanes, and leads to better performance than previous methods.

This paper presents an experimental evaluation of three different traffic sign detection approaches, which detect or localize various types of traffic signs from real-time videos. Specifically, the first approach exploits geometric features to identify traffic signs, while the other two are developed based on SVM (Support Vector Machine) and AdaBoost learning mechanisms. We describe each of the three approaches, conduct a detailed comparison among them, and examine their pros and cons. Our conclusions should lead to useful guidelines for developing a real-time traffic sign detector.

We present a novel framework for recognizing repetitive sequential events performed by human actors with strong temporal dependencies and potential parallel overlap. Our solution incorporates sub-event (or primitive) detectors and a spatiotemporal model for sequential event changes. We develop an effective and efficient method to integrate primitives into a set of sequential events where strong temporal constraints are imposed on the ordering of the primitives. In particular, the combination process is approached as an optimization problem. A specialized Viterbi algorithm is designed to learn and infer the target sequential events and handle the event overlap simultaneously. To demonstrate the effectiveness of the proposed framework, we report detailed quantitative analysis on a large set of cashier checkout activities in a retail store.

A significant portion of retail shrink is attributed to employees and occurs around the point of sale (POS). In this paper, we target a major type of retail fraud in surveillance videos, known as sweethearting (or fake scan), where a cashier intentionally fails to enter one or more items into the transaction in an attempt to get free merchandise for the customer. We first develop a motion-based algorithm to identify video segments as candidates for primitive events at the POS. We then apply spatio-temporal features to recognize true primitive events from the candidates and prune those falsely alarmed. In particular, we learn location-aware event models by Multiple-Instance Learning to address the location-sensitive issues that appear in our problem. Finally, we validate the entire transaction by combining primitive events according to temporal ordering constraints. We demonstrate the effectiveness of our approach on data captured from a real grocery store.

Video analytics have recently emerged as a promising technique of retail fraud detection for loss prevention. Efficient video analytic algorithms are highly desired for a practical fraud detection system. In this paper, we present a real-time algorithm for recognizing a cashier's actions at the point of sale (POS), which can be further used to analyze cashier behaviors for identifying fraudulent incidents. The algorithm uses a set of simple but effective features derived from a global representation of motion energy called polar motion map (PMM). These features capture the motion patterns exhibited in a cashier's actions as a focused beam of motion energy, characterizing the actions as the extension and retraction movement of the cashier's arm with respect to a prespecified region. Our algorithm demonstrates comparable accuracy against one of the state-of-the-art event recognition techniques while running significantly faster.

In virtually every business context there is a need to establish some form of monitoring system to ensure that employees comply with business processes and policies. Compliance failures range from organized theft to gaps in procedure that can be easily remedied through retraining. It is clearly important for businesses to capture and record these deviations to minimize loss prevention and maximize workplace safety and efficiency. In this workshop, we discuss the growing problem of compliance failure and how our system addresses this problem in a retail context to detect checkout-related fraud through the integration of visual and non-visual data.

Reliable information security mechanisms are required to combat the rising magnitude of identity theft in our society. While cryptography is a powerful tool to achieve information security, one of the main challenges in cryptosystems is to maintain the secrecy of the cryptographic keys. Though biometric authentication can be used to ensure that only the legitimate user has access to the secret keys, a biometric system itself is vulnerable to a number of threats. A critical issue in biometric systems is to protect the template of a user which is typically stored in a database or a smart card. The fuzzy vault construct is a biometric cryptosystem that secures both the secret key and the biometric template by binding them within a cryptographic framework. We present a fully automatic implementation of the fuzzy vault scheme based on fingerprint minutiae. Since the fuzzy vault stores only a transformed version of the template, aligning the query fingerprint with the template is a challenging task. We extract high curvature points derived from the fingerprint orientation field and use them as helper data to align the template and query minutiae. The helper data itself do not leak any information about the minutiae template, yet contain sufficient information to align the template and query fingerprints accurately. Further, we apply a minutiae matcher during decoding to account for nonlinear distortion and this leads to significant improvement in the genuine accept rate. We demonstrate the performance of the vault implementation on two different fingerprint databases. We also show that performance improvement can be achieved by using multiple fingerprint impressions during enrollment and verification.

Establishing identity is becoming critical in our vastly interconnected society. Questions such as "Is she really who she claims to be?," "Is this person authorized to use this facility?," or "Is he in the watchlist posted by the government?" are routinely being posed in a variety of scenarios ranging from issuing a driver's license to gaining entry into a country. The need for reliable user authentication techniques has increased in the wake of heightened concerns about security and rapid advancements in networking, communication, and mobility. Biometrics, described as the science of recognizing an individual based on his or her physical or behavioral traits, is beginning to gain acceptance as a legitimate method for determining an individual's identity. Biometric systems have now been deployed in various commercial, civilian, and forensic applications as a means of establishing identity. In this paper, we provide an overview of biometrics and discuss some of the salient research issues that need to be addressed for making biometric technology an effective tool for providing information security. The primary contribution of this overview includes: 1) examining applications where biometric scan solve issues pertaining to information security; 2) enumerating the fundamental challenges encountered by biometric systems in real-world applications; and 3) discussing solutions to address the problems of scalability and security in large-scale authentication systems.

Popular clustering algorithms, such as K-means (KM) and expectation maximization (EM), are sensitive to the initialization of cluster centers. In contrast, recently proposed K-harmonic means (KHM) algorithm is more robust to the randomness of the initialization. However, KHM works best when the dimensionality of the data (N) is small (usually less than 8). Because the dimensionality of features that are used for many clustering problems in image/video and speech processing is large, the benefits of KHM cannot be exploited. Based upon this observation, this paper proposes a novel method to employ KHM for high-dimensional data so as to realize initialization-independent clustering. The proposed method employs efficient spectral clustering techniques whereby the affinity matrix of the data is decomposed into its eigenvectors and k (total number of clusters) eigenvectors corresponding to the k largest eigenvalues are retained. That is, we represent N-D data by k-D transformed data when k < N and propose to employ KHM over this k-D transformed data. The assumption of k < N indeed encompasses a large number of significant video processing and computer vision problems where the use of KHM was not beneficial before. We demonstrate the effectiveness and the efficiency of the proposed algorithm for face clustering in the domains where the number of persons (k) is not large, such as anchorperson grouping, video-based speaker clustering in videoconferencing, and identity-based tracking in small office environments.

The detection and tracking of people lie at the heart of many current and near-future applications of computer vision. We describe a background subtraction system designed to detect moving objects in a wide variety of conditions, and a second system to detect objects moving in front of moving backgrounds. Detected foreground regions are tracked with a tracking system which can initiate real-time alarms and generate a smart surveillance index which can be searched to find interesting events in stored video

In traditional cryptosystems, user authentication is based on possession of secret keys; the method falls apart if the keys are not kept secret (i.e., shared with non-legitimate users). Further, keys can be forgotten, lost, or stolen and, thus, cannot provide non-repudiation. Current authentication systems based on physiological and behavioral characteristics of persons (known as biometrics), such as fingerprints, inherently provide solutions to many of these problems and may replace the authentication component of traditional cryptosystems. We present various methods that monolithically bind a cryptographic key with the biometric template of a user stored in the database in such a way that the key cannot be revealed without a successful biometric authentication. We assess the performance of one of these biometric key binding/generation algorithms using the fingerprint biometric. We illustrate the challenges involved in biometric key generation primarily due to drastic acquisition variations in the representation of a biometric identifier and the imperfect nature of biometric feature extraction and matching algorithms. We elaborate on the suitability of these algorithms for digital rights management systems.

The level of security at a facility is directly related to how well the facility can keep track of "who is where". The "who" part of this question is typically addressed through the use of face images for recognition either by a person or a computer face recognition system. The "where" part of this question can be addressed through 3D position tracking. The "who is where" problem is inherently multi-scale, wide angle views are needed for location estimation and high resolution face images for identification. A number of other people tracking challenges like activity understanding are multiscale in nature. An effective system to answer "who is where?" must acquire face images without constraining the users and must closely associate the face images with the 3D path of the person. Our solution to this problem uses computer controlled pan-tilt-zoom cameras driven by a 3D wide-baseline stereo tracking system. The pan-tilt-zoom cameras automatically acquire zoomed-in views of a person's head, while the person is in motion within the monitored space.

Fingerprint identification is based on two basic premises: (1) persistence and (2) individuality. We address the problem of fingerprint individuality by quantifying the amount of information available in minutiae features to establish a correspondence between two fingerprint images. We derive an expression which estimates the probability of a false correspondence between minutiae-based representations from two arbitrary fingerprints belonging to different fingers. Our results show that (1) contrary to the popular belief, fingerprint matching is not infallible and leads to some false associations, (2) while there is an overwhelming amount of discriminatory information present in the fingerprints, the strength of the evidence degrades drastically with noise in the sensed fingerprint images, (3) the performance of the state-of-the-art automatic fingerprint matchers is not even close to the theoretical limit, and (4) because automatic fingerprint verification systems based on minutia use only a part of the discriminatory information present in the fingerprints, it may be desirable to explore additional complementary representations of fingerprints for automatic matching.

Fingerprint identification is based on two basic premises: (i) persistence: the basic characteristics of fingerprints do not change with time; and (ii) individuality: the fingerprint is unique to an individual. The validity of the first premise has been established. While the second premise is generally accepted to be true, the underlying scientific basis of fingerprint individuality has not been formally tested. We address the problem of fingerprint individuality by quantifying the amount of information available in minutiae points to establish a correspondence between two fingerprint images. We derive an expression which estimates the probability of falsely associating minutiae-based representations from two arbitrary fingerprints. For example, the probability that a fingerprint with 36 minutiae points will share 12 minutiae points with another arbitrarily chosen fingerprint with 36 minutiae points is 6.10/spl times/10/sup -8/. These probability estimates are compared with typical fingerprint matcher accuracy results. Our results show that: (i) fingerprint matching is not infallible and leads to some false associations, (ii) the performance of automatic fingerprint matcher does not even come close to the theoretical performance, and (iii) due to the limited information content of the minutiae-based representation, automatic system designers should explore the use of non-minutiae-based information present in the fingerprints.

Biometrics-based authentication is becoming popular because of increasing ease-of-use and reliability. Performance evaluation of such systems is an important issue. We attempt to address two aspects of performance evaluation that have been conventionally neglected. First, the "difficulty" of the data that is used in a study influences the evaluation results. We propose some measures to characterize the data set so that the performance of a given system on different data sets can be compared. Second, conventional studies often have reported the false reject and false accept rates (FRR, FAR) in the form of match score distributions. However, no confidence intervals are computed for these distributions, hence no indication of the significance of the estimates is given. In this paper, we systematically study and compare parametric and nonparametric (bootstrap) methods for measuring confidence intervals. We give special attention to false reject rate estimates.

It is desirable to enhance the fingerprint image for achieving good fingerprint matching performance. In our earlier work, we have proposed a method for learning a set of partitioned least-squares filters from a given set of images and ground truth pairs. In this paper, we propose a refined approach to learn a class of hierarchical filter banks to extend our enhancement technique. The new technique is shown to obtain much better results particularly when the input image is not easily restored using our earlier technique. We evaluate the performance of our new approach to fingerprint enhancement filter design by assessing the effect of enhancement on performance of (i) fingerprint feature extraction and (ii) fingerprint matching.

We propose a feedback path for the feature extraction stage, followed by a feature refinement stage for improving the matching performance. This performance improvement is illustrated in the context of a minutia-based fingerprint verification system. We show that a minutia verification stage based on re-examining the gray-scale profile in a detected minutia's spatial neighborhood in the sensed image can improve the matching performance by /spl sim/4% on our database. Further, we show that a feature refinement stage which assigns a class label to each detected minutia (ridge ending and ridge bifurcation) before matching can also improve the matching performance by /spl sim/3%. A combination of feedback (minutia verification) in the feature extraction phase and feature refinement (minutia classification) improves the overall performance of the fingerprint verification system by /spl sim/8%.

Fingerprint images contain varying amount of noise because of the limitations of the fingerprint acquisition process. It is often necessary to enhance such noisy fingerprint images so that the features extracted from them are reliable. We propose a novel approach to fingerprint enhancement where a set of filters are learned using the "learn-from-example" paradigm. An expert provides the ground truth information for ridges in a small set of representative fingerprint images. The space of local fingerprint patterns in a small neighborhood is partitioned into a set of expressive yet computationally simple classes. A filter is learnt for each partition by finding the optimal linear mapping (in least-square sense) from the input to the enhanced space. The proposed approach offers distinct performance and speed advantages for a wide variety of fingerprint images.

With identity fraud in our society reaching unprecedented proportions and with an increasing emphasis on the emerging automatic personal identification applications, biometrics-based verification, especially fingerprint-based identification, is receiving a lot of attention. There are two major shortcomings of the traditional approaches to fingerprint representation. For a considerable fraction of population, the representations based on explicit detection of complete ridge structures in the fingerprint are difficult to extract automatically. The widely used minutiae-based representation does not utilize a significant component of the rich discriminatory information available in the fingerprints. Local ridge structures cannot be completely characterized by minutiae. Further, minutiae-based matching has difficulty in quickly matching two fingerprint images containing a different number of unregistered minutiae points. The proposed filter-based algorithm uses a bank of Gabor filters to capture both local and global details in a fingerprint as a compact fixed length FingerCode. The fingerprint matching is based on the Euclidean distance between the two corresponding FingerCodes and hence is extremely fast. We are able to achieve a verification accuracy which is only marginally inferior to the best results of minutiae-based algorithms published in the open literature. Our system performs better than a state-of-the-art minutiae-based system when the performance requirement of the application system does not demand a very low false acceptance rate. Finally, we show that the matching performance can be improved by combining the decisions of the matchers based on complementary (minutiae-based and filter-based) fingerprint information.

With the identity fraud in our society reaching unprecedented proportions and with an increasing emphasis on the emerging automatic positive personal identification applications, biometrics-based identification, especially fingerprint-based identification, is receiving a lot of attention. There are two major shortcomings of the traditional approaches to fingerprint representation. For a significant fraction of population, the representations based on explicit detection of complete ridge structures in the fingerprint are difficult to extract automatically. The widely used minutiae-based representation does not utilize a significant component of the rich discriminatory information, available in the fingerprints. The proposed filter-based algorithm uses a bank of Gabor filters to capture both the local and the global details in a fingerprint as a compact 640-byte fixed length FingerCode. The fingerprint matching is based on the Euclidean distance between the two corresponding FingerCodes and hence is extremely fast. Our initial results show identification accuracies comparable to the best results of minutiae-based algorithms published in the open literature. Finally, we show that the matching performance can be improved by combining the decisions of the matchers based on complementary fingerprint information.

We view fingerprints as oriented texture patterns, textures which exhibit an inherent and well-defined sense of directionality. Given a fingerprint image, we demonstrate that reliable translation and rotation invariant representations can be built based entirely on the inherent properties of the underlying fingerprint texture. We also illustrate that the representations thus derived are useful for robust discrimination of the fingerprints. In particular, we present the application of our novel representation scheme for solving the problems of fingerprint classification and matching on large datasets of fingerprint images acquired in real situations. The proposed scheme of generic representation for oriented textures relies on extracting one or more invariant frames of reference of the oriented texture based on an analysis of its orientation field. The fingerprint classification is based on a two-stage classifier which uses a K-NN classifier in its first stage and a set of neural network classifiers in its second stage.

Fingerprint verification is an important biometric technique for personal identification. We describe the design and implementation of a prototype automatic identity-authentication system that uses fingerprints to authenticate the identity of an individual. We have developed an improved minutiae-extraction algorithm that is faster and more accurate than our earlier algorithm (1995). An alignment-based minutiae-matching algorithm has been proposed. This algorithm is capable of finding the correspondences between input minutiae and the stored template without resorting to exhaustive search and has the ability to compensate adaptively for the nonlinear deformations and inexact transformations between an input and a template. To establish an objective assessment of our system, both the Michigan State University and the National Institute of Standards and Technology NIST 9 fingerprint data bases have been used to estimate the performance numbers. The experimental results reveal that our system can achieve a good performance on these data bases. We also have demonstrated that our system satisfies the response-time requirement. A complete authentication procedure, on average, takes about 1.4 seconds on a Sun ULTRA I workstation (it is expected to run as fast or faster on a 200 HMz Pentium).

Fingerprint images vary in quality. In order to ensure that the performance of an automatic fingerprint identification system (AFIS) will be robust with respect to the quality of input fingerprint images, it is essential to incorporate a fingerprint enhancement module in the AFIS system. We introduce a new fingerprint enhancement algorithm which decomposes the input fingerprint image into a set of filtered images. From the filtered images, the orientation field is estimated and a quality mask which distinguishes the recoverable and unrecoverable corrupted regions in the input image is generated. The input fingerprint image is adaptively enhanced in the recoverable regions. The performance of our algorithm has been evaluated on an online fingerprint verification system using the MSU fingerprint database containing over 600 fingerprint images. Experimental results show that our enhancement algorithm improves the performance of the online fingerprint verification system and makes it more robust with respect to the quality of input fingerprint images.

It is generally agreed that individual visual cues are fallible and often ambiguous. This has generated a lot of interest in design of integrated vision systems which are expected to give a reliable performance in practical situations. The design of such systems is challenging since each vision module works under a different and possibly conflicting set of assumptions. We have proposed and implemented a multiresolution system which integrates perceptual organization (grouping), segmentation, stereo, shape from shading, and line labeling modules. We demonstrate the efficacy of our approach using images of several different realistic scenes. The output of the integrated system is shown to be insensitive to the constraints imposed by the individual modules. The numerical accuracy of the recovered depth is assessed in case of synthetically generated data. Finally, we have qualitatively evaluated our approach by reconstructing geons from the depth data obtained from the integrated system. These results indicate that integrated vision systems are likely to produce better reconstruction of the input scene than the individual modules.>

Individual cues from visual modules are fallible and often ambiguous. As a result, only integrated vision systems can be expected to give a reliable performance in practice. The design of such systems is challenging since each vision module works under different and possibly conflicting sets of assumptions. We have proposed and implemented a multiresolution system which integrates perceptual grouping, segmentation, stereo, shape from shading, and line labelling modules. The output of the integrated system is shown to be relatively insensitive to the constraints imposed by the individual modules.>