In surveillance situations, computer vision systems are often deployed to help humans perform their tasks more effectively. In a typical installation human observers are required to simultaneously monitor a number of video signals. Psychophysical research indicates that there are severe limitations in the ability of humans to monitor simultaneous signals. Do these same limitations extend to surveillance? We present a method for evaluating human surveillance performance in a situation that mimics demands of real world surveillance. A single computer monitor contained either nine display cells or four display cells. Each cell contained a stream of 2 to 4 moving objects. Observers were instructed to signal when a target event occurred - - when one of the objects entered a small square ldquoforbiddenrdquo region in the center of the display. Target events could occur individually or in groups of 2 or 3 temporally close events. The results indicate that the observers missed many targets (60%) when required to monitor 9 displays and many fewer when monitoring 4 displays (20%). Further, there were costs associated with target events occurring in close temporal succession. Understanding these limitations would help computer visions researchers to design algorithms and human-machine interfaces that result in improved overall performance.

A new method for evaluating edge detection algorithms is presented and applied to measure the relative performance of algorithms by Canny, Nalwa-Binford, Iverson-Zucker, Bergholm, and Rothwell. The basic measure of performance is a visual rating score which indicates the perceived quality of the edges for identifying an object. The process of evaluating edge detection algorithms with this performance measure requires the collection of a set of gray-scale images, optimizing the input parameters for each algorithm, conducting visual evaluation experiments and applying statistical analysis methods. The novel aspect of this work is the use of a visual task and real images of complex scenes in evaluating edge detectors. The method is appealing because, by definition, the results agree with visual evaluations of the edge images.

The purpose of this paper is to describe a new (to computer vision) experimental framework which allows us to make quantitative comparisons using subjective ratings made by people. This approach avoids the issue of pixel-level ground truth. As a result, it does not allow us to make statements about the frequency of false positive and false negative errors at the pixel level. Instead, using experimental design and statistical techniques borrowed from Psychology, we make statements about whether the outputs of one edge detector are rated statistically significantly higher than the outputs of another. This approach offers itself as a nice complement to signal-based quantitative measures. Also, the evaluation paradigm in this paper is goal oriented; in particular, we consider edge detection in the context of object recognition. The human judges rate the edge, detectors based on how well the capture the salient features of real objects. So far, edge detection modules have been designed and evaluated in isolation, except for the recent work by Ramesh and Haralick (1992). The only prior work (that we are aware of) which also uses humans to rate image algorithms is that of Reeves and Higdon (1995). They use human ratings to decide on regularization parameters of image restoration. Fram and Deutch (1975) also used human subjects, however, the focus was on human versus machine performance rather than using human ratings to compare different edge detectors. The use of human judges to rate image outputs mist be approached systematically. Experiments must be designed and conducted carefully, and results interpreted with appropriate statistical tools. The use of statistical analysis in vision system performance characterization has been rare. The only prior work in the area that we are aware of is that of Nair et al. (1995), who used statistical ranking procedures to compare neural network based object recognition systems.