Computer scientists and software engineers seldom rely on using experimental methods despite frequent calls to do so. The problem may lie with the shortcomings of traditional experimental methods. We introduce a new form of experimental designs, synthetic designs, which address these shortcomings. Compared with classical experimental designs (between-subjects, within-subjects, and matched-subjects), synthetic designs can offer substantial reductions in sample sizes, cost, time and effort expended, increased statistical power, and fewer threats to validity (internal, external, and statistical conclusion). This new design is a variation of within-subjects design in which each system user serves in only a single treatment condition. System performance scores for all other treatment conditions are derived synthetically without repeated testing of each subject. This design, though not applicable in all situations, can be used in the development and testing of some computer systems provided that user behavior is unaffected by the version of computer system being used. We justify synthetic designs on three grounds: this design has been used successfully in the development of computerized mug shot systems, showing marked advantages over traditional designs; a detailed comparison with traditional designs showing their advantages on 17 of the 18 criteria considered; and an assessment showing these designs satisfy all the requirements of true experiments (albeit in a novel way).

When a witness in a criminal investigation is asked to identify a suspect in a conventional, randomly ordered album of photographs, identification success deteriorates rapidly as more photos are examined. An alternative is a feature-based retrieval system in which mug shots of previous offenders are displayed in order of similarity to a witness's description of a suspect's facial features. Previous studies have shown that the use of a feature system increases identification success by reducing the number of photographs witnesses examine. However, those studies failed to test feature systems in naturalistic settings. Two experiments are reported showing that feature systems are, indeed, effective in more naturalistic settings. In one, subject witnesses viewed a 37-second videotape of a crime in progress. In the other, they witnessed a live confrontation between two people. Afterwards, witness descriptions of the facial features of each suspect were compared with those of 1000 official police mug shots on file. The number of photos to be examined to find target suspects was reduced from the 500 expected for the traditional album to an average of less than 25.

The focus of this paper is on flexibility in a “query-answering” system which includes the user who generates the queries as well as other components of the system responsible for interpreting and fulfilling the query. Furthermore, the emphasis is expanded from pure “queries” which can be satisfied by information alone to generalized queries which may be satisfied by some combination of information, services, and physical goods not initially known in detail by the user. In order to carry out these comparisons and contrasts in a manner that uncovers potentially useful implications for internet infrastructure providers, consumer oriented enterprises, and business-to-business enterprises, we will use an analytic approach based on the post-structural theories of Michel Foucault. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In suspect identification, witnesses examine photos of known offenders in mugshot albums. The probability of correct identification deteriorates rapidly, however, as more photos are examined. Computerized feature systems, which display mugshots in order of similarity to witness descriptions increase success by reducing the number of mugshots examined. Previous experimental tests failed to examine the effects of delay (in eliciting witness descriptions) on recall memory and system performance, as well as the effects of number of raters per mugshot, and live target suspects (rather than using photos of men as targets). In two studies (one with live and one with photo target suspects), subject witnesses described targets after delays varying from several days up to four weeks. Delay had no effect on the number of photos examined, although in the fourth week the accuracy of witness feature descriptions of suspects began to deteriorate. Two raters per mugshot significantly improved performance. More raters did not improve performance further. Regardless of delay, witness searches resulted in photos of target suspects retrieved on average among the first 20-45 mugshots for a database of 1000 mugshots before a target suspect's photo was displayed. The effects of delay on recall memory were similar to those reported previously for recognition memory of faces.

Modern financial institutions require sophisticated risk assessment tools to integrate human expertise and historical data in a market that is changing and broadening qualitatively, quantitatively, and geographically. The need is especially acute in newly developed countries where expertise and data are scarce, and knowledge bases and assumptions imported from the West may be of limited applicability. Second order logical models can be a valuable tool in such situations. They integrate the robustness of neural or statistical modeling of data, the perspicuity of logical rule induction, and the experience and understanding of skilled human experts. The approach is illustrated in the context of risk assessment in the Korean surety insurance industry.

This paper focuses on the problem of information retrieval from databases containing images rather than text. We propose an error-tolerant alternative to menus and keywords—the feature-matching approach—in which users describe what they want to retrieve in response to a set of queries. The system matches the user's description with descriptions of images already in the database. Database images are then presented to the user in order of similarity to the user's description. The present paper serves four purposes: application of our feature-matching approach to a new image domain (trademarks); systematization of the process for developing these systems (articulation of five stages in the process of system development); specification of criteria for selecting features to maximize system performance; and introduction of concepts of power of discrimination and error tolerance to show how measures of these two factors can be used for evaluating system performance and optimizing system development. Evaluation of the system (including experiments on a small pilot database of trademarks and simulations of large databases) show the proposed set of 150 features would, without modification, be capable of handling expansion of the database to over 50 000 trademarks while still retrieving the target within the first 10 items on average. Analysis suggested several changes that should further improve the feature set.

Correct suspect identification of known offenders by witnesses deteriorates rapidly as more are examined in mugshot albums. Feature approaches, where mugshots are displayed in order of similarity to witnesses' descriptions, increase identification success by reducing this number. System performance depends on selection of system features. Four methods of selecting features are evaluated empirically: theory, random, hill-climbing algorithm, and hybrid. The theory asserts success depends on five properties of system features: informativeness, orthogonality, sufficiency, consistency, and observability. Comparing system performance on the best 10 features selected (from a pool of 90) by each method supports our contention. In four experimental tests of a system with 1000 official mugshots, over 90% of witness searches resulted in photos of target suspects retrieved in the first ten mugshots displayed for examination (using all 90 system features). On average, suspects were retrieved in the first 54, 7, 22, and 70 mugshots when using only the best 10 model features. Hybrid and hill-climbing algorithms did not improve on this performance, and performance of randomly selected sets of 10 features was poor.

An important factor for determining the success of any natural language program is the variability in the users' questions that must be handled by the system. Thus, it is important to find methods of measuring and handling this variability. In this article we examine the tendency for subjects to use the topics and terms introduced in the computer's answers in forming their questions. It is shown that the last and next‐to‐last answers have a strong effect on the questions users ask. It is also shown that although users often do ask questions that are related to their previous questions, the most frequent case is for users to introduce new questions related to the computer's answers. Thus, the effect of the computer's answers is stronger than the effect of users' own questions when forming new questions. Finally, it is shown that the accuracy of the computer's responses in answering the questions has important effects on the tendency to use previous questions and answers. These results suggest that natural language systems that take advantage of these features of human‐computer interaction may be successful.

A field trial was conducted to test a natural-language technology developed by the authors (COMODA) and to dispense AIDS information to the public. This trial allowed users with computers and modems to dial-in to an AIDS information system and ask questions or browse through the information. The system received nearly 500 calls during a two month period. The calls lasted an average of 10 minutes and involved an average of 27 interactions between the user and the computer. Approximately 45% of the interactions were direct naturallanguage questions, and the COMODA system was quite successful in answering these questions. Also, the comments left by the users were extremely positive, with 96% of the users who rated the system giving it a positive rating. The users commented that the COMODA system was an easy-to-use method of accessing valuable AIDS information, and they would like to see the system expanded to cover more topic areas. The users also made useful suggestions on how the system could be improved. It was concluded that the COMODA system is a viable natural-language access system for presenting information to the public.

One need not create a natural language understanding system in order to create a hypertext data base that can be traversed with unconstrained natural language. The task is simplified because the computer creates a constrained context, imposes a non-negotiable topic, and elicits simple questions. Two small hypertext data bases describing the authors' organi7ation and the terms and rules of baseball were implemented on an IBM PC. When ten untrained people were allowed to search through these data bases, 59 per cent of their queries were answered correctly by the first data base and 64 per cent by the second.

Supervising student teams who worked with expert practitioners to build prototype expert systems has yielded insights into the frustrations and rewards of the early stages of knowledge engineering. This paper presents these insights to the expert systems community, particularly those beginning their first project or entering a wholly new application area. We focus on example based rule induction shells without imprecision. While such tools are generally too limited for implementing a finished, deliverable expert system, they are useful for “breaking the ice” with a quick, inexpensive demonstration system while acquiring the vital first elements of the eventual expert knowledge base.

Six different varieties of methodology are surveyed for choosing one of a fixed number of alternative actions in the context of uncertainty. Within this context a fixed number of possible states of the world can actually hold, where the outcome of each alternative action is dependent on the state of the world. The six approaches differ from each other primarily in their assumptions about the quality and quantity of information that is available regarding: the relative possibility or likelihood of the various states of the world, and the relative utility of the various outcomes defined by (action, state) pairs. The six approaches are illustrated using a single example. Finally, the prospects for an integrated approach to decision support that is sensitive to the quality and quantity of information are discussed, and some fruitful areas for further research are suggested.