Describes the Collaborative Apprenticeship Learning Object Toolkit (CALOT), an architecture that forms the basis of a family of tutoring systems. It utilizes the apprenticeship model of learning for the study of operational procedures where theory and practice are both important. Systems built upon the CALOT architecture can be utilized by multiple, non-colocated students in a synchronous manner. A distributed set of students can collaborate on the tutorial in real-time. The development of this type of computer tutorial system is driven by the goal of linking theoretical knowledge with practical operational experience. Active, exploratory, apprentice-style learning is supported via modes of operation within the system. The student can flexibly choose to observe the expert perform and explain operational steps, or to act as an apprentice and carry out the steps autonomously. The student can switch between these modes at their discretion, giving the student control of the level of intervention by the system. Furthermore, the student can explore and reflect on an information space of object designations, procedures and related concepts. Two examples tutorials based on the CALOT architecture are the Collaborative Virtual Spectrometer and the Virtual X-Ray Diffractometer, which teach about NMR spectroscopy and X-ray diffraction, respectively. Finally, BAUEN (Basic AUthoring ENvironment) is a GUI system that helps the software developer designer author a CALOT-based computer tutorial system with maximum efficiency, focusing the developer's efforts on domain content, and not on the system implementation.

Early CAI efforts focused on the transmission or delivery of knowledge to the student. Subsequent work in intelligent tutoring systems took a more cognitive approach and attempted to emulate the behavior of skilled human tutors via software. More recent work has focused on collaborative learning processes, and the idea of a collaboratory as a virtual space for work. This paper describes a design process to support the development and use of collaborative learning technologies. This process, the Learning Collaboratory Design Framework (LuCiDiFy), integrates methods and concepts from cognitive systems engineering, theories of learning and instruction, distributed computing and computer-supported collaborative learning (CSCL). LuCiDiFy is instantiated in an actual software testbed, the Collaborative Learning Environment for Operational Systems (CLEOS). CLEOS is a collaboratory for teachers, students and practitioners in the physical sciences, and in particular for NMR spectroscopy and X-ray diffraction experiments. CLEOS includes two virtual instrument tutorials, CVS (Collaborative Virtual Spectrometer) and VXRD (Virtual X-Ray Diffractometer); an asynchronous messaging system, the Question Board (QB); a project-based design environment, TOSP (Tool for Organizing and Supervising Projects); and an architecture for developing collaborative apprenticeship learning systems, CALOT (Collaborative Apprenticeship Learning Object Toolkit); an authoring environment, BAUEN (Basic AUthoring ENvironment); and a collaborative multi-user domain (MUD) infrastructure, MudSpot.

Describes a model-based design method to develop CoRaven, a decision support tool that is intended to assist military intelligence analysts in managing and interpreting large quantities of battlefield information. In this method, we use observations of practitioners solving specific tasks in order to understand and model how they use information. We use this model of the task to help identify user needs that the tool must support, and, during initial prototyping, to guide usability analyses. We have found task models to be an important consideration in the decision support tool design process that can help to constrain the design space and reduce the time required to develop an effective decision support tool prototype.

The paper provides an overview of several related research projects on collaborative planning. The task of collaborative planning in dynamic environments is described and two related projects are presented. We constructed two interactive systems based on a MOO and performed a usability evaluation using task performance and subjective measures. The first system is a collection of various intelligent agent capabilities implemented in a MOO environment. We situate our system in the context of debates about definitions of agency; our agents provide online assistance to users but the extent to which they embody various theoretical ideas of agency is also of interest. The second project examines the use of hand-held computing devices in a MOO based collaborative planning environment. Basic issues of usability as well as theoretical issues in distributed collaborative planning are discussed.

The field of computer supported cooperative work (CSCW) has led to many advances in both the study of cooperative work groups and the development of collaboration technologies. However, a number of key research issues remain. Two of these issues are: (1) the seamless integration of support for asynchronous and synchronous cooperative work, and (2) the ability for individual users to flexibly change or tailor their mode of collaboration while engaged in real time interaction. The paper describes a new approach to handling both these issues. The approach is called the Flexible Collaborative Support (FLECS) framework. It relies on the use of active database technology, an explicit organizational model, and flexible transaction management. This approach bridges the gap between CSCW and multi-user database systems. It does so by supporting awareness of other users' presence in the information space as well as providing visualization of other users' actions, thus fulfilling the most fundamental requirement of CSCW systems. At the same time, the FLECS framework capitalizes on the data security and consistency management methods which have formed the foundation of multi-user database systems. The paper also presents CityCollab and its evaluation by a community of users. CityCollab is a collaborative system, which supports the processing of Requests for Work in the context of civil infrastructure management. CityCollab represents an instantiation of the more general FLECS architecture/framework.

The paper discusses the domain of Army logistics planning at the level of the ODSCLOG (Office of the Deputy Chief of Staff for Logistics) and describes tools to assist in this planning. In particular, we have developed a prototype collaboratory (CLINT) for logistics planning that relies on Microsoft NetMeeting/sup TM/ as the collaborative infrastructure, and a tool named THREADS to support logistics problem solving. THREADS is a "visual simulation" of logistics routes that allows planners to quickly identify bottlenecks and compare alternative plans.

The GISMOO project examined the collaborative planning behavior of an existing team of practitioners in the Public Works Division of a US Army base through the use of a simple Web-enhanced virtual environment. Comparisons were drawn between the behaviors of the work group while communicating face to face, and while communicating synchronously via an online virtual conference room based on MOO (Multi-user Object Oriented) technology enhanced by a Web interface. The results of this project showed that the use of an online virtual environment for collaborative planning is practical and could be beneficial, especially to physically dispersed work groups. The value of the conference room tools was not proven in this study, primarily as the structure of the scenarios did not support the asynchronous portion of the team's collaborative work well. The behavior of the work group using the collaboration tool changed in generally predictable ways as compared to face to face communication.

In complex problem solving domains such as supervisory control, one approach to human-centered automation is intelligent associate systems. This paper proposes a taxonomy of cooperative problem solving; provides an analysis of models used in cooperative problem solving; and discusses a variety of intelligent associate systems.

The emergence of computer technology has had profound implications for instruction. Recent efforts in the field have shifted the focus from knowledge-as-a-product to learning-as-a-process. Computer-supported collaborative learning (CSCL) focuses on learning as a collaborative activity that is situated in its environment. The paper describes the Apprenticeship Learning Object Toolkit (ALOT), a generalized architecture, that forms the basis of a family of computer tutoring systems. The system utilizes the apprenticeship model of learning to create a learning environment for the study of operational procedures of experiments in the physical sciences. Examples include nuclear magnetic resonance (NMR) spectroscopy or x-ray diffractometry (XRD). The development of a learning environment is driven by the goal of linking theoretical knowledge with practical operational experience. Active, exploratory, apprentice-style learning is supported via modes of operation within the system. The student can flexibly choose to "observe the expert" perform and explain operational steps, or "act as an apprentice" and carry out the steps autonomously. The student can switch between these modes at their discretion, giving the student control of the level of intervention by the system. In addition, the student can explore and reflect on an "information space" of object designations, procedures, and related concepts. Additionally, the BAUEN (basic authoring environment) is introduced. BAUEN is a graphical user interface system that is intended to help the designer author a computer tutorial system with maximum efficiency, focusing the developer's efforts on domain content and not on system implementation.

This work presents the development of a multi-user virtual environment in support of crisis management planning activities. Coupling ongoing work in activity representation with analysis of the domain, an extensible domain ontology was developed. Studying the domain, several challenges became apparent: (1) the need to support extensive dynamic and distributed collaboration, (2) the need for a flexible, open architecture, (3) the need for views of the information tailored to the activities of the planning team, and (4) the opportunity to leverage past and current related efforts in the domain. This project developed a Java-based collaboration architecture around a multi-user domain (MUD) to provide presence and access to collaborative services. The collaborative infrastructure layer provides persistence, user authentication, and access control. Built upon this substrate are collaborative services. The target use of MAST is to represent and track workflow in the crisis action planning done by the Operations Planning Team of the US Pacific Command. The design approach was to base MAST on a reusable class library which implements a rich ontology structure and represents key elements of the domain. Our ontology draws on the SPOT project and the Shared Planning and Activity Representation (SPAR) project. The resulting ontology, implemented in Java, is compliant with the SPAR Reference Object Model Specification. A strength of this work is the extensibility, flexibility and openness both of the representation and the system architecture.

Intelligence analysis is one of the major functions performed by an Army staff in battlefield management. In particular, intelligence analysts develop intelligence requirements based on the commander's information requirements, develop a collection plan, and then monitor messages from the battlefield with respect to the commander's information requirements. The goal of the CoRAVEN project is to develop an intelligent collaborative multimedia system to support intelligence analysts. Key ingredients of our design approach include: (1) significant knowledge engineering activities with domain experts, (2) representation of an explicit model of reasoning and activity to drive design, (3) the use of Bayesian belief networks as a way to structure inferences that relate observable data to the commander's information requirements, (4) collaborative graphical user interfaces to provide flexible support for the multiple tasks in which analysts are engaged, (5) sonification of data streams and alarms to support enhanced situation awareness, (6) detailed psychological studies of reasoning and judgment under uncertainty, and (7) iterative prototyping of candidate designs with domain experts for both formative and summative evaluation. The paper discusses our current progress on all these fronts.

Distributed supervisory control is the monitoring and control of complex, dynamic, highly automated systems by a team of human operators. As such, it is an example of a cooperative work environment. The goal of this paper is to analyze the nature of cooperative work in distributed supervisory control environments and describe implications for computer support. In particular, we argue that flexible support for activity management is a key requirement for computer support for distributed supervisory control, and we propose the intelligent support for activity management (ISAM) architecture as one infrastructure for doing so. The design and implementation of ISAM draws from previous work on intelligent operator's associate systems and computer-supported cooperative work (CSCW) systems. A graphical user interface to ISAM and an evaluation study of this system in the context of team supervisory control are also described.

Training is a critical issue for operators responsible for the safe and efficient operation of large-scale complex dynamic systems. This paper proposes and articulates a set of requirements for an intelligent tutoring system. The requirements specify what (instructional content) and how (instructional strategies) to teach a novice operator to supervise and control a complex dynamic system. The instructional content teaches system structure and behavior (i.e., declarative knowledge), system procedures (i.e., procedural knowledge), and how to use this declarative and procedural knowledge to manage a complex dynamic system in real time (i.e., operational skill). Using the underlying representations of the operator function model (OFM) and OFMspert, the OFM's computational implementation. GT-VITA (Georgia Tech Visual and Inspectable Tutor and Assistant) realizes these requirements. As a proof-of-concept demonstration, an instance of the generic GT-VITA tutoring architecture was implemented for satellite ground controllers. The empirical evaluation, utilizing NASA satellite ground control personnel, showed that GT-VITA was a flexible and useful training system. In fact, NASA has adopted VITA as the foundation for required training for all satellite ground control personnel. In addition to an intelligent tutoring system architecture, by using and extending the operator function model and OFMspert, GT-VITA demonstrates a robust methodology for conceptualizing the tutor-aid paradigm. The tutor-aid paradigm defines a conceptual framework in which learning with a intelligent tutor gradually becomes collaboration with an intelligent associate. Using the same structures (i.e., OFM and OFMspert) and the same domain knowledge, GT-VITA specifies a tutor and GT-MOCA (Jones and Mitchell, 1995) specifies an aid.>

This paper proposes a methodology for carrying out human-machine systems engineering research and in particular focuses on the development of knowledge-based support for training and aiding. The process is characterized by the definition of knowledge requirements, a normative model structure of human-machine interaction, a knowledge architecture that implements the model, and an interactive real-time simulation environment. The knowledge requirements largely specify the model, knowledge architecture, and simulation. The model and the knowledge architecture are the basis for the design of intelligent systems. The simulation serves as an experimental testbed that provides a feasible, safe, and effective means for exploring alternative human-machine systems designs and also facilitates technology transfer. The aim is to create an understanding of a complex work domain that forms the context for ongoing research in human-machine interaction. A specific example from satellite ground control is described.< >

One approach to aiding the human supervisory controller of a complex dynamic system is to provide an intelligent operator's associate. We propose a prescriptive theory of human-computer cooperative problem solving and describes the design and evaluation of a prototype system based on the theory. The theory consists of five principles: human-in-charge, mutual intelligibility, openness and honesty, management of trouble, and multiple perspectives. A prototype intelligent associate system, the Georgia Tech Mission Operations Cooperative Assistant (GT-MOCA), is an embodiment of these principles that provides a collection of context-sensitive resources for the human operator of a simulated satellite ground control system. These resources include an interactive visualization of current activities, an organized message lists of important events, and interactive graphics depicting the current state of the controlled system. An evaluation study utilizing actual NASA satellite ground controllers showed that GT-MOCA was perceived to be useful and provided performance benefits for certain portions of the control task.>

The design, implementation, and evaluation of ALLY, a computer-based associate for the human supervisor of a simulated satellite ground control system, is described. ALLY's dynamic intent inferencing abilities and system control properties are described. Experimental evaluation of ALLY compared performance of a two-person team with a person-ALLY team in the control of a simulated satellite ground control system.>

The authors explore the key characteristics of an intelligent advisory system. A central feature is that human-machine cooperation should be based on a metaphor of human-to-human cooperation. ALLY, a computer-based operator's associate, is discussed which is based on a preliminary theory of human-to-human cooperation. ALLY assists the operator in carrying out the supervisory control functions for a simulated NASA ground control system. Experimental evaluation of ALLY indicates that operators using ALLY performed at least as well as they did when using a human associate, and in some cases they performed even better.>

The authors propose the concept of an operator's associate to serve as an intelligent aid to the human operator. They note that a necessary prerequisite for a useful and effective 'intelligent' automated aid is the ability to understand, or infer the intentions of, the human operator. This ability in turn relies on a sound model of operator performance goals in the control of the system. A good model of operator performance defines the structure of operator intentions or goals with respect to current system state. In the authors' system, this intentional structure is implemented as a blackboard architecture called ACTIN (Action Interpreter). Beyond implementation, however, the intent inferencing process must be validated. The authors describe two studies that experimentally evaluated the system's ability to infer operator intentions involving (1) post hoc analysis by a human domain expert and (2) concurrent verbal protocols by the actual human operator. Finally, the authors offer suggestions toward a general validation methodology.>

This paper describes the use of Smalltalk in the design and implementation of OFMspert. OFMspert is an architecture for an intelligent operator's associate. In order to verify the architecture, an OFMspert was implemented to act as an assistant to an operator of a NASA satellite ground control system. OFMspert is a large system that utilizes a Smalltalk implementation of a knowledge-based problem solving method known as the blackboard architecture. The entire system was designed and implemented in Smalltalk/V™ and later ported to the Smalltalk-80™ system. The object-oriented paradigm in general, and Smalltalk‡ in particular, greatly facilitated the rapid design and implementation of this system. This paper summarizes the OFMspert architecture, emphasizing its implementation in an object-oriented paradigm.

The authors proposes an architecture for an expert system that can function as an operator's associate in the supervisory control of a complex dynamic system. Called OFMspert (operator function model (OFM) expert system), the architecture uses the operator function modeling methodology as the basis for the design. The authors put emphasis on the understanding capabilities, i.e. the intent referencing property, of an operator's associate. They define the generic structure of OFMspert, particularly those features that support intent inferencing. They also describe the implementation and validation of OFMspert in GT-MSOCC (Georgia Tech-Multisatellite Operations Control Center), a laboratory domain designed to support research in human-computer interaction and decision aiding in complex, dynamic systems.>