In this paper, we define meta-recognition, a performance prediction method for recognition algorithms, and examine the theoretical basis for its postrecognition score analysis form through the use of the statistical extreme value theory (EVT). The ability to predict the performance of a recognition system based on its outputs for each match instance is desirable for a number of important reasons, including automatic threshold selection for determining matches and nonmatches, and automatic algorithm selection or weighting for multi-algorithm fusion. The emerging body of literature on postrecognition score analysis has been largely constrained to biometrics, where the analysis has been shown to successfully complement or replace image quality metrics as a predictor. We develop a new statistical predictor based upon the Weibull distribution, which produces accurate results on a per instance recognition basis across different recognition problems. Experimental results are provided for two different face recognition algorithms, a fingerprint recognition algorithm, a SIFT-based object recognition system, and a content-based image retrieval system.

Measuring system performance seems conceptually straightforward. However, the interpretation of the results and predicting future performance remain as exceptional challenges in system evaluation. Robust experimental design is critical in evaluation, but there have been very few techniques to check designs for either overlooked associations or weak assumptions. For biometric & vision system evaluation, the complexity of the systems make a thorough exploration of the problem space impossible - this lack of verifiability in experimental design is a serious issue. In this paper, we present a new evaluation methodology that improves the accuracy of variance estimator via the discovery of false assumptions about the homogeneity of cofactors - i.e., when the data is not ''well mixed". The new methodology is then applied in the context of a biometric system evaluation with highly influential cofactors.

In this paper, a new framework for evaluating a variety of computer vision systems and components is introduced. This framework is particularly well suited for domains such as classification or recognition systems, where blind application of the i.i.d. assumption would reduce an evaluation's accuracy, such as with classification or recognition systems. With few exceptions, most previous work on vision system evaluation does not include confidence intervals, since they are difficult to calculate, and are often coupled with strict requirements. We show how a set of previously overlooked replicate statistics tools can be used to obtain tighter confidence intervals of evaluation estimates while simultaneously reducing the amount of data and computation required to reach such sound evaluatory conclusions. In the included application of the new methodology, the well-known FERET face recognition system evaluation is extended to incorporate standard errors and confidence intervals.

Autonomous video surveillance and monitoring of human subjects in video has a rich history. Many deployed systems are able to reliably track human motion in indoor and controlled outdoor environments, e.g., parking lots and university campuses. A challenging domain of vital military importance is the surveillance of noncooperative and camouflaged targets within cluttered outdoor settings. These situations require both sensitivity and a very wide field of view and, therefore, are a natural application of omnidirectional video. Fundamentally, target finding is a change detection problem. Detection of camouflaged and adversarial targets implies the need for extreme sensitivity. Unfortunately, blind change detection in woods and fields may lead to a high fraction of false alarms, since natural scene motion and lighting changes produce highly dynamic scenes. Naturally, this desire for high sensitivity leads to a direct tradeoff between miss detections and false alarms. This paper discusses the current state of the art in video-based target detection, including an analysis of background adaptation techniques. The primary focus of the paper is the Lehigh Omnidirectional Tracking System (LOTS) and its components. This includes adaptive multibackground modeling, quasi-connected components (a novel approach to spatio-temporal grouping), background subtraction analyses, and an overall system evaluation.

Recently, S. Nayar (1997) introduced a parabolic imaging system that has a field of view of a full hemisphere or more. When used with a video camera the result is an omni-directional video stream that captures everything going around it. In the VAST Lab at Lehigh, we have been experimenting with these cameras, developing new variants, and developing omni-directional vision applications. We present an overview of omni-directional imaging and then two of our applications which we will be demonstrating. The first application is a frame-rate multi-body tracking system. The system uses an omni-directional imager and a standard PC to track multiple moving objects in all directions. The system is designed to provide perspective views of the most significant targets, either locally or over a network. The second application is something we call Remote Reality, which provides an immersive environment via omnidirectional imaging. It can use live or pre-recorded video from a remote location. While less interactive than traditional VR, remote reality has important advantages: there is no need for "model building" and the objects, textures and motions are not graphical approximations.