This paper begins with a discussion of the perceived requirement for general psychological models to underpin the design of human interaction with complex systems. A proposal is made that general psychological models may not be necessary, if indeed possible to create, for many applications of interest. For the applications of interest, cognitive engineering or human-machine systems models, such as the operator function model (OFM), may suffice. Next, a brief overview of the current state of OFM and its computational implementation, OFMspert, is presented. It describes how OFMspert has been extended to support the design of two intelligent tutoring systems (ITS) for operational control of safety-critical systems. An intelligent tutoring system requires two models: an expert model and a student model. This paper describes how the OFM can serve as the basis of the expert model and the OFMspert architecture as the basis for the architecture of the tutor itself.

The paper presents a brief overview of the current implementation and evolution of the operator function model (OFM) (C.M. Mitchell, 1987) and OFMspert, its computational implementation. It describes the proposed extension of these models to design operations automation, that is, automation that controls a system in a manner similar to that of a human operator. It then describes two operations automation research projects: AutoPass and Apprentice. AutoPass, now concluded, is presented, together with the empirical evaluation and results. A description of Apprentice, still in progress, concludes the paper.

With modern technology, rapid change characterizes complex dynamic systems, including the practitioners' role and required knowledge. Training in safety-critical domains must occur in a timely manner and ensure that practitioners understand changing systems and procedures. Web-based training provides an excellent opportunity to address these issues because distribution of training can be timely, easily accessed and cost-efficient. Few sophisticated training systems exist for practitioners of complex dynamic systems. GT-ITACS (Georgia Tech Intelligent Architecture for Complex Systems) was designed and implemented as an intelligent tutoring system (ITS) "shell" to support several ongoing Georgia Tech ITS research efforts. Access to domain-specific education and training from a Web-based library may meet the needs of training programs by allowing easy access to ongoing education and training material.

Even with extensive formal training, new operators of modern, large-scale technological systems typically have extended periods of on-the-job training before they become experts. In some of these environments, the complex and changing nature of operational requirements is exacerbated by the cost of training and need to meet certification requirements. A case-based intelligent tutoring system (CBITS) has been proposed as one alternative to respond to the need for additional training. This paper describes the development of cases for a case-based intelligent tutor and illustrates how an example case can be used in a particular implementation-GT-CBITS (Georgia Tech case-based intelligent tutoring system). The case addresses the use of a newly certified submode and control feature of the MD-11 aircraft. The paper illustrates how GT-CBITS represents expert knowledge of a case with the operator function model (OFM) and uses OFMspert, the computational implementation of the operator function model, as the intelligence to maintain dynamic expert and student models.

OFMspert (operator function model expert system) is a computational software system to support operator intent inferencing in a wide range of domains and applications. This paper presents a brief history of OFMspert, followed by a description of various enhancements resulting in the current domain- and application-independent architecture.

This paper proposes a technique, Apprentice, for constructing, incrementally and primarily passively, a knowledge base for operations automation. It blends machine learning and knowledge elicitation techniques similar to those recently proposed by the knowledge acquisition community. Apprentice allows operators to understand what an autonomous system is doing and why, and then repair or extend the underlying knowledge base as necessary to meet the wide range of operating conditions that characterize complex systems operations. The four phases of Apprentice's knowledge building technique are presented followed by a brief discussion of the advantages of this approach.

One of the major tasks of pilots of modern aircraft is monitoring and understanding the status and behavior of the auto-flight system, i.e., mode awareness. In order to maintain mode awareness in the dynamic environment, pilots must be continuously vigilant of indications from several locations within the cockpit. Lacking accurate and complete system knowledge and interfaces that clearly present the system state and constraints, pilots may misunderstand the control modes, pilots often cite vertical path navigation (VNAV) as a flight management system function that "surprises" them. The VNAV Tutor, a computer-based training system, was developed to address this issue. The VNAV Tutor attempts to improve pilots' understanding of VNAV control modes and the interaction of the mode control panel functions with the flight management system during VNAV usage. An evaluation showed that the VNAV Tutor enhanced both the conceptual understanding and the operational use of the vertical navigation function by pilots transitioning to aircraft with sophisticated auto-flight systems.

Training is a critical issue for operators responsible for the safe and efficient operation of large-scale complex dynamic systems. This paper proposes and articulates a set of requirements for an intelligent tutoring system. The requirements specify what (instructional content) and how (instructional strategies) to teach a novice operator to supervise and control a complex dynamic system. The instructional content teaches system structure and behavior (i.e., declarative knowledge), system procedures (i.e., procedural knowledge), and how to use this declarative and procedural knowledge to manage a complex dynamic system in real time (i.e., operational skill). Using the underlying representations of the operator function model (OFM) and OFMspert, the OFM's computational implementation. GT-VITA (Georgia Tech Visual and Inspectable Tutor and Assistant) realizes these requirements. As a proof-of-concept demonstration, an instance of the generic GT-VITA tutoring architecture was implemented for satellite ground controllers. The empirical evaluation, utilizing NASA satellite ground control personnel, showed that GT-VITA was a flexible and useful training system. In fact, NASA has adopted VITA as the foundation for required training for all satellite ground control personnel. In addition to an intelligent tutoring system architecture, by using and extending the operator function model and OFMspert, GT-VITA demonstrates a robust methodology for conceptualizing the tutor-aid paradigm. The tutor-aid paradigm defines a conceptual framework in which learning with a intelligent tutor gradually becomes collaboration with an intelligent associate. Using the same structures (i.e., OFM and OFMspert) and the same domain knowledge, GT-VITA specifies a tutor and GT-MOCA (Jones and Mitchell, 1995) specifies an aid.>

This paper proposes a methodology for carrying out human-machine systems engineering research and in particular focuses on the development of knowledge-based support for training and aiding. The process is characterized by the definition of knowledge requirements, a normative model structure of human-machine interaction, a knowledge architecture that implements the model, and an interactive real-time simulation environment. The knowledge requirements largely specify the model, knowledge architecture, and simulation. The model and the knowledge architecture are the basis for the design of intelligent systems. The simulation serves as an experimental testbed that provides a feasible, safe, and effective means for exploring alternative human-machine systems designs and also facilitates technology transfer. The aim is to create an understanding of a complex work domain that forms the context for ongoing research in human-machine interaction. A specific example from satellite ground control is described.< >

One approach to aiding the human supervisory controller of a complex dynamic system is to provide an intelligent operator's associate. We propose a prescriptive theory of human-computer cooperative problem solving and describes the design and evaluation of a prototype system based on the theory. The theory consists of five principles: human-in-charge, mutual intelligibility, openness and honesty, management of trouble, and multiple perspectives. A prototype intelligent associate system, the Georgia Tech Mission Operations Cooperative Assistant (GT-MOCA), is an embodiment of these principles that provides a collection of context-sensitive resources for the human operator of a simulated satellite ground control system. These resources include an interactive visualization of current activities, an organized message lists of important events, and interactive graphics depicting the current state of the controlled system. An evaluation study utilizing actual NASA satellite ground controllers showed that GT-MOCA was perceived to be useful and provided performance benefits for certain portions of the control task.>

The design, implementation, and evaluation of ALLY, a computer-based associate for the human supervisor of a simulated satellite ground control system, is described. ALLY's dynamic intent inferencing abilities and system control properties are described. Experimental evaluation of ALLY compared performance of a two-person team with a person-ALLY team in the control of a simulated satellite ground control system.>

HCI research at Georgia Tech is found in three cooperating groups: the Engineering Psychology and Experimental Psychology Programs in the School of Psychology, the Center for Human-Machine Systems Research in the School of Industrial and Systems Engineering, and the interdisciplinary Graphics, Visualization and Usability (GVU) Center. We cooperate via cross-listed courses, having students in one area take a minor in another area, collaborative research projects, serving on Ph. D. committees, joint colloquia and brown bag lunches, and joint appointments. The GVU Center (housed in the College of Computing) and Cognitive Science Program (sponsored by Psychology, Industrial and Systems Engineering, and the College of Computing) involves a number of the same faculty, further enhancing our collaborations.

Ahab, an intelligent computer-aided instruction (ICAI) program, illustrates an architecture for simulator-based ICAI programs to teach troubleshooting in complex, dynamic environments. The architecture posits three elements of a computerized instructor: the task model, the student model, and the instructional module. The task model is a prescriptive model of expert performance that uses symptomatic and topographic search strategies to provide students with directed problem-solving aids. The student model is a descriptive model of student performance in the context of the task model. The instructional module coordinates information presented by the instructional media, the task model, and the student model so that each student receives individualized instruction. Concept and metaconcept knowledge that supports these elements is contained in frames and production rules, respectively. The results of an experimental evaluation are discussed.>

The use of direct manipulation interface technology as a valuable design technique for operator interaction and control with a real-time supervisory control system was examined. To evaluate the effectiveness of a direct manipulation interface, two user interfaces to a simulation of a flexible manufacturing system were evaluated. Five measures that reflected operator performance were analyzed. Subjects using the direct manipulation interface operated the system significantly better on three of the five measures.>

The authors explore the key characteristics of an intelligent advisory system. A central feature is that human-machine cooperation should be based on a metaphor of human-to-human cooperation. ALLY, a computer-based operator's associate, is discussed which is based on a preliminary theory of human-to-human cooperation. ALLY assists the operator in carrying out the supervisory control functions for a simulated NASA ground control system. Experimental evaluation of ALLY indicates that operators using ALLY performed at least as well as they did when using a human associate, and in some cases they performed even better.>

A research project was undertaken to address the problems and issues of operator training in complex engineering systems. The research proposes a tutor/aid paradigm for the design of an intelligent tutoring system (ITS) that evolves from a tutor to an operator's assistant for supervisory control of complex dynamic systems. The goal is to design a joint cognitive system that achieves a symbiosis of human and computer skills critical in ensuring safe and reliable operation of complex dynamic systems. The implementation of the tutor/aid concept capitalizes on OFMspert, a successful computer-based associate that represents the final end in the evolution of the tutor to an assistant. Characteristics of an intelligent tutoring/aiding system are identified with respect to the representation of domain knowledge, the tutor's pedagogical structure, and the student knowledge representation. The OFMspert architecture provides preliminary structures for realizing some of these characteristics. Areas in which OFMspert must be augmented to support tutoring are identified. Finally, the architecture is illustrated in the context of a tutor/assistant for NASA satellite ground controllers.>

An evaluation is made of the decision processes of humans in a flexible manufacturing system (FMS) environment. GT-FMS, a real-time simulator of an FMS, was implemented with data from an actual FMS installation. An experiment was conducted in which humans interacted with the control system of GT-FMS at an aggregate level. The humans described each of their control actions, and their decision processes were evaluated by mapping these descriptions onto a standard model of human decision-making. The experimental results support making humans an integral part of the FMS control process, since an intricate knowledge of the system state and system sensitivity were crucial to human decision-making in GT-FMS. Humans in this experiment used detailed system status information, rather than system performance history, as the basis of their control decisions and were inconsistent in defining their goals.>

The authors propose the concept of an operator's associate to serve as an intelligent aid to the human operator. They note that a necessary prerequisite for a useful and effective 'intelligent' automated aid is the ability to understand, or infer the intentions of, the human operator. This ability in turn relies on a sound model of operator performance goals in the control of the system. A good model of operator performance defines the structure of operator intentions or goals with respect to current system state. In the authors' system, this intentional structure is implemented as a blackboard architecture called ACTIN (Action Interpreter). Beyond implementation, however, the intent inferencing process must be validated. The authors describe two studies that experimentally evaluated the system's ability to infer operator intentions involving (1) post hoc analysis by a human domain expert and (2) concurrent verbal protocols by the actual human operator. Finally, the authors offer suggestions toward a general validation methodology.>

A description is given of the interaction of a human operator and an automated transportation system within a cell of an FMS. The interaction of a human operator with operations-research-based scheduling algorithms is the research focus. The human role in management of the cell transportation system consists of monitoring the status of parts requiring transportation and, when necessary, interacting with the automated scheduling system to affect the produced schedules in such a way that compensation is made for late parts. Specific operator interaction methods are proposed. Experiments to test the detailed supervisory control philosophy are then discussed.>

A description is given of a research domain developed to study the role of human supervisory control in discrete manufacturing. A discussion is presented of the problems found when attempting to map J. Rasmussen's hierarchy (1986) directly to the discrete manufacturing domain. A methodology is proposed for extending Rasmussen's hierarchy to discrete manufacturing, and a plan to validate the hypothesis is suggested.>

The authors propose and evaluate ITSSO (intelligent tutoring system for satellite operators), an architecture for the design of an intelligent tutoring system for operators of NASA ground control systems. An experiment compared the performance of subjects trained with ITSSO to the performance of subjects trained via conventional hands-on experience. Although ITSSO did not result in uniformly better performance, the study provides a number of important insights into the issues and problems involved in the successful design and evaluation of intelligent tutoring systems for operators of complex systems.>

Presents a review of some of the areas to which supervisory control theory has been applied and indicate some of the ideas which have as yet not been followed up but which were introduced by Sheridan and his students. Topics covered include: modeling manual and supervisory control tasks; supervisory control of deep-ocean vehicles; technology problems and technology solutions; human integration in a complex world; supervisory control in distributed decision-making; and human supervisor control models in advanced manufacturing systems.>

An intelligent tutoring system (ITS) is discussed that helps to organize system knowledge and operational information, including symptom-cause relationships, to enhance operator performance. The ITS incorporates the structure, function and behavior of the controlled system in an appropriate form that achieves cognitive compatibility with the operators. In addition, the ITS contains a normative task model that provides it with the ability to infer the trainee's misconceptions. The example system considered is an intelligent tutor to be used for training operators to troubleshoot large dynamic systems.>

This paper describes the use of Smalltalk in the design and implementation of OFMspert. OFMspert is an architecture for an intelligent operator's associate. In order to verify the architecture, an OFMspert was implemented to act as an assistant to an operator of a NASA satellite ground control system. OFMspert is a large system that utilizes a Smalltalk implementation of a knowledge-based problem solving method known as the blackboard architecture. The entire system was designed and implemented in Smalltalk/V™ and later ported to the Smalltalk-80™ system. The object-oriented paradigm in general, and Smalltalk‡ in particular, greatly facilitated the rapid design and implementation of this system. This paper summarizes the OFMspert architecture, emphasizing its implementation in an object-oriented paradigm.

Control of flexible manufacturing systems (FMS) requires the complex interaction of two components: (1) computers to perform automated control and routing activities, and (2) humans to supervise the automation, to monitor system flows and outputs, and to intervene to diagnose and either correct or compensate for unanticipated events. Current academic FMS scheduling research has focused mainly on the first component in the control loop, development of scheduling algorithms for optimization and control. Here, the second component is included in both the definition of the FMS control problem and the corresponding control approach. A more realistic definition of the FMS control environment is presented using a supervisory control framework. Within this context, the concept of aiding a human operator who supervises the predominantly automated FMS operations is developed.>

The results of an experimental study of the supervisory control of a simulated flexible manufacturing system (FMS) are discussed. A real-time simulator of an FMS, (Georgia Tech-FMS (GT-FMS)) was implemented and configured with data from a real manufacturing system. An experiment was run in which humans interacted with the automatic control system of GT-FMS with the goal of improving overall system performance by meeting due data while simultaneously minimizing inventory. Experimental results show that with human supervision both due data and inventory performance of GT-FMS can be improved. The results strongly support the idea of actively integrating humans into operational controls of automated manufacturing environments.>

The authors proposes an architecture for an expert system that can function as an operator's associate in the supervisory control of a complex dynamic system. Called OFMspert (operator function model (OFM) expert system), the architecture uses the operator function modeling methodology as the basis for the design. The authors put emphasis on the understanding capabilities, i.e. the intent referencing property, of an operator's associate. They define the generic structure of OFMspert, particularly those features that support intent inferencing. They also describe the implementation and validation of OFMspert in GT-MSOCC (Georgia Tech-Multisatellite Operations Control Center), a laboratory domain designed to support research in human-computer interaction and decision aiding in complex, dynamic systems.>

Modeling and aiding operators in supervisory control environments are necessary prerequisites to the effective use of automation in complex dynamic systems. A research program is described that explores these issues within the context of the Georgia Tech-Multisatellite Operations Control Center (GT-MSOCC). GT-MSOCC is a real-time interactive simulation of the operator interface to a NASA ground control system for unmanned Earth-orbiting satellites. GT-MSOCC is a high fidelity domain in which a range of modeling, decision aiding, and workstation design issues addressing human-computer interaction may be explored. GT-MSOCC is described in detail. The use of high-fidelity domains as research tools is also discussed, and the validity and generalizability of such results to other domains are examined. In addition to a description of GT-MSOCC, several other related parts are included. A GT-MSOCC operator function model (OFM) is presented. The GT-MSOCC model illustrates an enhancement to the general operator function modeling methodology that extends the model's utility for design applications. The proposed methodology represents operator actions as the lowest level discrete control network nodes. Actions may be cognitive or manual; operator action nodes are linked to information needs or system reconfiguration commands. Thus augmented, the operator function model provides a formal representation of operator interaction with the controlled system and serves as the foundation for subsequent theoretical and empirical research. A brief overview of experimental and methodological studies using GT-MSOCC is also given.

The use of a voice-augmented keyboard input modality is evaluated in a supervisory control application. An implementation of voice recognition technology in supervisory control is proposed: voice is used to request display pages, while the keyboard is used to input system reconfiguration commands. Twenty participants controlled GT-MSOCC, a highfidelity simulation of the operator interface to a NASA ground control system, via a workstation equipped with either a single keyboard or a voice-augmented keyboard. Experimental results showed that in all cases where significant performance differences occurred, performance with the voice-augmented keyboard modality was inferior to and had greater variance than the keyboard-only modality. These results suggest that, current moderately priced voice recognition systems are an inappropriate human-computer interaction technology in supervisory control systems.

Model-based qualitative icons and adaptive window display interfaces may be valuable tools to enhance the effectiveness of operators in real-time data-intensive supervisory control systems. Qualitative icons may be used to integrate low-level quantitative data into high-level qualitative error detection mechanisms. Using windowing technology, multiple data sources that reflect different aspects of system state can be displayed simultaneously on a single screen. Both technologies were combined and implemented to design an operator interface to the Georgia Tech-Multisatellite Operation Control Center (GT-MSOCC). An operator function model for GT-MSOCC was used to derive workstation features, including hardware configuration, the function of qualitative icons for monitoring, fault detection and identification, and the contents and placement of computer windows. The model also determined sets of windows needed by the operator to undertake major operator control functions. An experiment was performed to evaluate the effectiveness of a workstation incorporating model-based qualitative icons and dynamic operator function window sets. Subjects controlled GT-MSOCC via either a conventional operator interface or the model-based interface. Eleven measures that reflected operator performance were analyzed. Subjects using the model-based workstation operated the system significantly better on nine of these measures. On all measures, performance with the model-based workstation was uniformly better on average and had less variability than performance with the conventional workstation.

Recent advances in computer technology and the changing rule of the human in complex systems require changes in design strategies for information displays. The use of discrete control models to represent the human operator's cognitive and decisionmaking activities is described. The analytic procedures required to build a discrete control model show promise as a basis of a design methodology for the definition of an information display system for supervisory control tasks. The discrete control modeling procedures and their application for a simulated system is demonstrated.