Computer scientists and software engineers seldom rely on using experimental methods despite frequent calls to do so. The problem may lie with the shortcomings of traditional experimental methods. We introduce a new form of experimental designs, synthetic designs, which address these shortcomings. Compared with classical experimental designs (between-subjects, within-subjects, and matched-subjects), synthetic designs can offer substantial reductions in sample sizes, cost, time and effort expended, increased statistical power, and fewer threats to validity (internal, external, and statistical conclusion). This new design is a variation of within-subjects design in which each system user serves in only a single treatment condition. System performance scores for all other treatment conditions are derived synthetically without repeated testing of each subject. This design, though not applicable in all situations, can be used in the development and testing of some computer systems provided that user behavior is unaffected by the version of computer system being used. We justify synthetic designs on three grounds: this design has been used successfully in the development of computerized mug shot systems, showing marked advantages over traditional designs; a detailed comparison with traditional designs showing their advantages on 17 of the 18 criteria considered; and an assessment showing these designs satisfy all the requirements of true experiments (albeit in a novel way).

When a witness in a criminal investigation is asked to identify a suspect in a conventional, randomly ordered album of photographs, identification success deteriorates rapidly as more photos are examined. An alternative is a feature-based retrieval system in which mug shots of previous offenders are displayed in order of similarity to a witness's description of a suspect's facial features. Previous studies have shown that the use of a feature system increases identification success by reducing the number of photographs witnesses examine. However, those studies failed to test feature systems in naturalistic settings. Two experiments are reported showing that feature systems are, indeed, effective in more naturalistic settings. In one, subject witnesses viewed a 37-second videotape of a crime in progress. In the other, they witnessed a live confrontation between two people. Afterwards, witness descriptions of the facial features of each suspect were compared with those of 1000 official police mug shots on file. The number of photos to be examined to find target suspects was reduced from the 500 expected for the traditional album to an average of less than 25.

In suspect identification, witnesses examine photos of known offenders in mugshot albums. The probability of correct identification deteriorates rapidly, however, as more photos are examined. Computerized feature systems, which display mugshots in order of similarity to witness descriptions increase success by reducing the number of mugshots examined. Previous experimental tests failed to examine the effects of delay (in eliciting witness descriptions) on recall memory and system performance, as well as the effects of number of raters per mugshot, and live target suspects (rather than using photos of men as targets). In two studies (one with live and one with photo target suspects), subject witnesses described targets after delays varying from several days up to four weeks. Delay had no effect on the number of photos examined, although in the fourth week the accuracy of witness feature descriptions of suspects began to deteriorate. Two raters per mugshot significantly improved performance. More raters did not improve performance further. Regardless of delay, witness searches resulted in photos of target suspects retrieved on average among the first 20-45 mugshots for a database of 1000 mugshots before a target suspect's photo was displayed. The effects of delay on recall memory were similar to those reported previously for recognition memory of faces.

This paper focuses on the problem of information retrieval from databases containing images rather than text. We propose an error-tolerant alternative to menus and keywords—the feature-matching approach—in which users describe what they want to retrieve in response to a set of queries. The system matches the user's description with descriptions of images already in the database. Database images are then presented to the user in order of similarity to the user's description. The present paper serves four purposes: application of our feature-matching approach to a new image domain (trademarks); systematization of the process for developing these systems (articulation of five stages in the process of system development); specification of criteria for selecting features to maximize system performance; and introduction of concepts of power of discrimination and error tolerance to show how measures of these two factors can be used for evaluating system performance and optimizing system development. Evaluation of the system (including experiments on a small pilot database of trademarks and simulations of large databases) show the proposed set of 150 features would, without modification, be capable of handling expansion of the database to over 50 000 trademarks while still retrieving the target within the first 10 items on average. Analysis suggested several changes that should further improve the feature set.

Correct suspect identification of known offenders by witnesses deteriorates rapidly as more are examined in mugshot albums. Feature approaches, where mugshots are displayed in order of similarity to witnesses' descriptions, increase identification success by reducing this number. System performance depends on selection of system features. Four methods of selecting features are evaluated empirically: theory, random, hill-climbing algorithm, and hybrid. The theory asserts success depends on five properties of system features: informativeness, orthogonality, sufficiency, consistency, and observability. Comparing system performance on the best 10 features selected (from a pool of 90) by each method supports our contention. In four experimental tests of a system with 1000 official mugshots, over 90% of witness searches resulted in photos of target suspects retrieved in the first ten mugshots displayed for examination (using all 90 system features). On average, suspects were retrieved in the first 54, 7, 22, and 70 mugshots when using only the best 10 model features. Hybrid and hill-climbing algorithms did not improve on this performance, and performance of randomly selected sets of 10 features was poor.