This paper revisits several concepts and model frameworks that have been in the literature of human-machine interaction and control engineering for up to 50 years. The purposes of the revisit are as follows: 1) to sharpen distinctions between adaptive automation, level of automation, allocation authority, supervisory control, and adaptive control engineering as the terms are currently used in the literature; 2) to define modes of human supervisory adaptation from the control engineering perspective; and 3) to suggest comparative taxonomies for adaptive automation in direct control and in supervisory control.

This paper offers two perspectives on the relation between music and supervisory control. The first is that music making can be viewed as a three-level hierarchy of control and feedback interactions. The top level is the composer or conductor interacting through hand and auditory signals with the musician's conscious (higher brain level) activity. The mid level is internal to the musician, the higher brain centers interacting with the musician's subconscious neuromuscular activity. The lowest level is the musician's neuromuscular system interacting with the musical instrument. At each level there are constraints that are discussed in the paper. A second perspective refers to a phenomenon of music listening, its capability to engender a sense of virtual reality (or virtual presence) in much the same way that observing a computer-generated visual scene can make an observer feel present in a place other than where he or she is physically located.

With increasing attention to patient safety, hospitals and other clinical facilities are developing practice guidelines and protocols with the specific intent of reducing harm to patients. However, the introduction of these protocols can have unanticipated negative consequences and if followed rigidly can become disabling. We use the manual count procedure that was designed to improve patient safety by reducing the likelihood of leaving an object (e.g., needle, sponge, or instrument) inside a patient body cavity during a surgical procedure to illustrate this point. Using results from a focus group of seven operating room nurses and an observational study of nine complex operations, we show that the count protocol has unanticipated negative consequences that need to be considered in evaluating the net positive gain in patient safety. The study highlights the importance of evaluating the overall impact of proposed protocols in assessing its potential benefits to patient safety.

Antilock brake system (ABS) technology with powerful electronic components has shown superior braking performance to conventional vehicles on test tracks. On real highways, however, the performance of the ABS-equipped car has been disappointing. The poor braking performance with ABS has resulted from the questionable design of the human-machine interface for the brake system. The goal of this study is to design brake systems that provide more intuitive brake control and proper braking-performance information for the driver. In this study automotive brake systems are modeled as a type of master-slave telemanipulator. Human force-displacement interaction at the brake pedal has a strong effect on braking performance. As a preliminary study in brake-system design, the characteristics of human leg motion and its underlying motor-control scheme are studied through experiments and simulations, and a model of braking motion by the driver's leg is developed. This paper proposes novel brake systems based on two new aspects. First, the mechanical impedance characteristics of the leg action of the driver are taken into consideration in designing the brake systems. Second, the brake systems provide the driver with kinesthetic feedback of braking conditions or performance. The effectiveness of the proposed designs in a combined driver-vehicle system is investigated using driving simulation.

In the evolution of any professional organization, it is informative and desirable to take stock of what has occurred, and to use this assessment to consider and plan for the future. The IEEE Systems, Man, and Cybernetics (SMC) Society is considered to be the leading professional society in the transdisciplinary area of systems engineering, cybernetics, and human machine systems, and has an international reputation for our efforts in developing and presenting innovative research results related to this area. In this paper, we - group of five current and former SMC Society presidents - consider the past, present, and future of the IEEE SMC Society; we are also doing this to commemorate the Society's 30th anniversary. In particular, we address our auspicious beginning; our transition from an incubatee to an incubator society; the breadth of our transactions; the international character of our membership; the appropriateness of our name; the move toward an "intelligent" systems-oriented umbrella organization; the evolving array of research areas; and the challenges and opportunities we face in the future.

Humans interact with their environments in many ways, and that interaction has become the focus of ever-greater interest as technology empowers the physical environment with new sensor and computer-based intelligence. This paper describes four different types of coupling between the human and environment, where "coupling" here refers to two-way interactions which are sufficiently tight that causality between human and environment is obscure and even arbitrary. These are not four mutually exclusive categories, but rather four ways of characterizing the relationship between the human and environment. Some generalizations of these relationships are discussed, along with hints on how we might improve our designs of intelligent technology, especially the interfaces to human users.

This paper identifies two contending metaphors of the mind, and outlines a model of comprehension informed by the parsimony principle ("Ockham's razor"). The model, called virtual associative network (VAN), is applied to explain human performance and improve decision aiding in complex tasks involving multiple variables and rapidly changing constraints. This model is compared to a conventional modeling paradigm ("Swiss army knife") representing the mind as a "toolkit" of special purpose "instruments" (or modules). The paper has four sections. The first section introduces the VAN model focusing on its key assumptions. The second section runs computational experiments to assess the mathematical validity of these assumptions. Next, some of the model's decision aiding applications are demonstrated. The concluding section discusses agreement and the lack of such between the VAN model and other cognitive theories. Discussion centers on assessing VAN's plausibility vis-a-vis recent neuropsychological findings.

BACKGROUND: Increasing data suggest that error in medicine is frequent and results in substantial harm. The recent Institute of Medicine report (LT Kohn, JM Corrigan, MS Donaldson, eds: To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press, 1999) described the magnitude of the problem, and the public interest in this issue, which was already large, has grown. GOAL: The goal of this white paper is to describe how the frequency and consequences of errors in medical care can be reduced (although in some instances they are potentiated) by the use of information technology in the provision of care, and to make general and specific recommendations regarding error reduction through the use of information technology. RESULTS: General recommendations are to implement clinical decision support judiciously; to consider consequent actions when designing systems; to test existing systems to ensure they actually catch errors that injure patients; to promote adoption of standards for data and systems; to develop systems that communicate with each other; to use systems in new ways; to measure and prevent adverse consequences; to make existing quality structures meaningful; and to improve regulation and remove disincentives for vendors to provide clinical decision support. Specific recommendations are to implement provider order entry systems, especially computerized prescribing; to implement bar-coding for medications, blood, devices, and patients; and to utilize modern electronic systems to communicate key pieces of asynchronous data such as markedly abnormal laboratory values. CONCLUSIONS: Appropriate increases in the use of information technology in health care- especially the introduction of clinical decision support and better linkages in and among systems, resulting in process simplification-could result in substantial improvement in patient safety.

In the present research, a special form of cluster analysis was successful in revealing cognitive groupings having appreciable influence on task performance in a target-tracking task designed to resemble ATC. The method of cluster analysis was derived from the "virtual associative network" model of memory organization and applied to the record of eye fixations in the course of task performance. Results using inexperienced subjects showed fixation clustering consistent with Gestalt factors. Task objectives did not seem to affect grouping as much. The subjects' recall was generally poor, except where direct manipulation of targets occurred. We conclude that: 1) cognitive grouping influences performance; 2) recall techniques may not be able to elicit subjects' cognitive groupings; and 3) such groupings can be determined via analysis of eye fixations. These findings have implications for studying workload assessment and information structuring in complex visual scanning tasks.

This paper discusses four ways that humans interact with their environments, plus four variables that determine the experience of virtual reality, and also which of the interactions support which of the VR-enhancing variables. Some philosophical issues abou immersion, the experience of presence, and the meaning of realit are then considered. The engineering paradigm of estimation is then reviewed as a way of bridging classical ontologica differences of opinion about reaility. Finally some VR research needs are discussed: haptics, mi nimally invasive diagnosis and surgery, driving simulation, decision aids in system operation, education, computer-aided synthesis, measures of presence and whether presence enhances performance, social ills of VR, and the relation of VR to spirituality.

Technical developments in computer hardware and software now make it possible to introduce automation into virtually all aspects of human-machine systems. Given these technical capabilities, which system functions should be automated and to what extent? We outline a model for types and levels of automation that provides a framework and an objective basis for making such choices. Appropriate selection is important because automation does not merely supplant but changes human activity and can impose new coordination demands on the human operator. We propose that automation can be applied to four broad classes of functions: 1) information acquisition; 2) information analysis; 3) decision and action selection; and 4) action implementation. Within each of these types, automation can be applied across a continuum of levels from low to high, i.e., from fully manual to fully automatic. A particular system can involve automation of all four types at different levels. The human performance consequences of particular types and levels of automation constitute primary evaluative criteria for automation design using our model. Secondary evaluative criteria include automation reliability and the costs of decision/action consequences, among others. Examples of recommended types and levels of automation are provided to illustrate the application of the model to automation design.

In aviation almost all decisions relate to safety, and most have therefore been proceduralized in order to reduce risk. There are very few decisions that are made on the basis of a value metric such as utility or economic cost. One decision that can be shown to be value-based is the selection of a flight profile. Fuel consumption and flight time have significant effects on aircraft operating cost, but they cannot be minimized simultaneously. In addition, winds, turbulence, and performance vary widely with altitude and time. These factors make it important and difficult for pilots to decide among alternative trajectories. A three-space, three-operation paradigm for decision-making was developed and used as a framework for the analysis of in-flight decision-making. Pilots and dispatchers from several US airlines were surveyed to determine which attributes of the outcome of a flight they considered most important. Avoiding turbulence-for passenger comfort-topped the list of items that were not directly related to safety. Pilots' decision-making about the selection of a flight profile on the basis of flight time, fuel burn, and exposure to turbulence was then observed. Each pilot's decisions were compared to those produced by a separately elicited utility model, the airline's minimum-cost objective function, and a lexicographic behavioral model. Of these decision models, utility maximization is shown to reproduce the pilots' decisions with the greatest accuracy. Furthermore, the airline's cost-minimization approach is shown to be inadequate for predicting in-flight decisions. Finally, we propose a decision aid that would present the pilot with attribute-space information (rather than feature-space information) about several pareto-optimal trajectories including minimum cost and maximum utility.

Decision making in the real world is concerned with both deterministic models and undeterministic human expertise. However. Both current decision-making methods use either mathematical programming or heuristic rules; each approach by itself loses the other piece of information. This paper proposes a new approach to integrate these two different types of information simultaneously and adaptively. It uses fuzzy mathematical programming to quantitatively model objectives and resource constraints, and a linguistic knowledge base to qualitatively model imprecise human expertise. The coordination between these two modules and the new idea of subjective knowledge adaptation for human-in-the-loop systems are discussed in detail.>

Information, as explicated by C.E. Shannon (1949), refers to the reduction in uncertainty about the state of an event after a message has been sent relative to the uncertainty about the state of the event before the message was sent. Information value, as explicated by R.A. Howard (1966) and others, refers to the difference in what one can gain by action taken knowing the state of the event relative to what one can gain by action taken without such knowledge. This paper considers the relation between the two concepts, information and information value, to the authors knowledge not heretofore discussed. It is concluded that the two concepts are independent, characterizing quite different aspects of information seeking and using. In application, the cost of the first term (information) subtracts from the second term (information value), and/or diminishes the latter in relation to information access time.>

A 30-year history of research on dealing with the effects of time delay in the control loop on human teleoperation in space is reviewed. Experiments on the effects of delay on human performance are discussed, along with demonstrations of predictive displays to help the human overcome the delay. Supervisory control is shown to offer a variety of options, from switching to local impedance control upon contact with the environment to higher-level local automation. Wave transformation techniques to ameliorate the effects of delay are also described. Space teleoperations have tended to deal with the problem of time delay by avoiding it and not attempting to teleoperate from the ground. It is suggested that the US space effort might have gotten further ahead and at a lower cost with a greater is commitment to space teleoperation from the ground through the delay. Predictive display works well for free positioning. Local impedance control is recommended for control in contact with the environment, possibly accompanied by wave transformation techniques. Higher level supervisory control is always an option for sufficiently predictable tasks and will continue to improve with better sensors and task models.>

A theory for optimizing the extent of a continuous action variable in a situation where either or both a discrete reward and discrete cost occur as probabilistic functions of the extent of the action selected is presented. It is shown how in many respects the approach is similar to signal detection theory, and also that there are the essential differences, especially since the decision is continuous rather than binary. Examples for application are suggested.>

Four current and former IEEE Systems, Man, and Cybernetics (SMC) Society presidents consider the past, present, and future of the IEEE SMC Society to commemorate the society's 20th anniversary. The authors address its auspicious beginning, its transition from an incubatee to an incubator society, the breadth of its transactions, the international character of its membership, the appropriateness of the name, the move toward a systems-oriented umbrella organization, the evolving array of research areas, and the challenges and opportunities the society faces in the future.>

A method for pragmatic inference as a necessary accompaniment to command languages is proposed. The approach taken focuses on the modeling and recognition of the human operator's intent, which relates sequences of domain actions ('plans') to changes in some model of the task environment. The salient feature of this module is that it captures some of the physical and linguistic contextual aspects of an instruction. This provides a basis for generalization and reinterpretation of the instruction in different task environments. The theoretical development is founded on previous work in computational linguistics and some models in the theory of action and intention. To illustrate these ideas, an experimental command language to a telerobot is implemented. The program consists of three different components: a robot graphic simulation, the command language itself, and the domain-independent pragmatic inference module. Examples of task instruction processes are provided to demonstrate the benefits of this approach.>

The planning problem associated with tactile exploration for object recognition and localization is addressed. Given that an object has been sensed and is one of a number of modeled objects, and given that the data obtained so far are insufficient for recognition and/or localization, the methods developed determin the paths along which a point contact sensor must be directed in order to obtain further highly diagnostic measurements. Three families of sensor paths are found. The first is the family of paths for which recognition and localization are guaranteed. The second guarantees only that something will be learned. The third represents paths to avoid because nothing new will be learned. The methods are based on a small but powerful set of geometric ideas and are developed for two-dimensional, planar-faced objects. They are conceptually easily generalized to handle three-dimensional objects, including objects with through holes.>

A two-port impedance-network model of a single-degree-of-freedom remote manipulation system in which a human operator at the master port interacts with a task object at the slave port in a remote location is presented. The design of the network involves the selection of feedback gains for the servomechanisms that transmit motion and force information from one port of the two-port to the other in both directions. The proposed methodology allows this selection to be based on both stability requirements and specifications of desired port impedances, given models of the task and the human operator. The resulting design guidelines guarantee stability for any passive task object at the slave port and any passive human impedance at the master port.>

A dynamic user aid was developed to help operators compensate for several-second time delays in telemanipulator systems. The approach includes state prediction, position feedback presentation, force feedback presentation, and slave impedance control. The aid helps the operator anticipate the delayed system response by providing predicted force and position feedback using dynamic models of both the manipulator and the environment. Three predictive force feedback techniques are also presented. Results of tests with a single-degree-of-freedom simulation showed operator performance to improve with the predictive aid. Tests also showed that 'dual' force reflection provides better performance than other techniques.>

An interactive decision-aiding technique is developed to assist a human operator in selecting a satisfactory goal in a control task. An attainability set of a system is introduced as the set of all goals that the system can potentially achieve. The goal selection problem is modeled as a decision problem of exploration of the attainability set. Experiments demonstrating the advantages of the proposed technique in comparison with traditional control methods are described. It was found that the performance of subjects using the decision mode was consistently better than that of subjects using either the automatic or the manual modes. The failure rate was much smaller (nonexistent) when the subjects used the decision mode. All subjects explicitly confirmed that the decision mode was the best and the easiest mode of controlling the experimental system.>

A framework is proposed for helping the operator of a redundant teleoperator to perform tasks in a 3D environment. The aid is implemented on a graphics workstation. Kinematic control of the redundant teleoperator is performed by specifying end-effector trajectories and using a numerical inverse kinematic algorithm to determine positions of the intermediate links of the teleoperator. Also proposed are graphic displays that aid the operator in the execution of tasks. Experiments with a twelve-degrees-of-freedom vehicle manipulator system that were conducted to compare these suggestions with other methods of kinematic control and display indicate that the proposed framework results in superior performance by reducing both task completion times and collision with obstacles in the environment.>

The effect of movement in each of the six degrees of freedom (d.o.f.) on operator tracking performance is studied. Subjects used a single six-d.o.f. hand controller to track a target ball with a control ball on a computer monitor. Depth cues in the displayed tracking environment made tracking in three dimensions possible. Three sets of experiments were conducted in which movement occurred in one, three, and six d.o.f. at a time. Root mean square error (r.m.s.e.) was calculated for each d.o.f. as the measure of performance. The experiments were performed with velocity and acceleration control plant inputs.>

The authors propose a model, or filter, for debiasing opinions from multiple experts and combining them into a single consistent estimate of some variable of interest. A distinguishing feature of the approach consists of making the calibration of experts an integral part of filtering. This enables the filter to learn from previous experience with the experts. The theoretical development takes a Bayesian perspective, using B. de Finetti's notion of exchangeability (1964). Experimental results with a preliminary computer implementation of the filter show that its estimates are better than those from comparable filters that do not involve calibration.>

An iterative method of computing the solution of the inverse kinematic problem is developed for redundant systems using the transpose of the Jacobian matrix instead of the pseudoinverse. The solutions may be optimized on a criterion function or on physical constraints, such as obstacle avoidance. Stability and convergence of the method are shown. Although its convergence rate is only about half that of Newton's method, the advantage of the method is that it remains easily tractable close to the singular configurations of the manipulator. A hybrid method combining the Jacobian transpose and Newton's methods is proposed. Results of the application of the method on a 10-link manipulator in 2-D space are shown.>

Manipulators are subject to interaction forces when they maneuver in a constrained work-space. Our goal is to develop a method for the design of controllers or constrained manipulators in the presence of model, uncertainties. The controller must carry out fine maneuvers when the manipulator is not constrained, and compliant motion, with or without Interaction-force measurement, when the manipulator is constrained. At the same time stability must be preserved If bounded uncertainties are allowed in modelling the manipulators. Stability of the manipulator and environment as a whole and the preservation of stability in the face of changes are two fundamental, issues that have been considered in the design method. We start with conventional controller design specifications concerning the treatment of external forces when the manipulator is not constrained. Generalizing this concept to include cases when the manipulator is constrained, we state a set of practical, design specifications in the frequency domain that is meaningful, from the standpoint of control, theory and assures the desired compliant motion. In the cartesian coordinate frame and stability in the presence of bounded uncertainties. This approach also assures the global stability of the manipulator and its environment. While this paper concerns the fundamentals of compliant motion, part 2 of this paper [Reference 24] is devoted to the controller design method.

This paper reviews some what's and whys of "supervisory control", "telerobotics" and "telepresence" as applied to operations in space. Specific salient problems and developments in control are described, including: (1) computer-based planning; (2) command language; (3) geometric resolution; and (4) arm/end-effector dexterity and impedance. Key problems and developments in telesensory feedback are also described, including: (i) predictive display; (2) teleproprioception; (3) force reflection; and (4) teletouch. Finally, some "policy" problems of design/development by computer simulation vs. hardware design, and the problems of evaluation of prototype systems are mentioned. Efforts from the author's laboratory are cited as examples; no comprehensive literature review is intended.

A controller design methodology to develop a robust compliant motion for robot manipulators is described. The achievement of the target dynamics (the target impedance is introduced in Part I) and preservation of stability robustness in the presence of bounded model uncertainties are the key issues in the design method. State-feedback and force-feedforward gains are chosen to guarantee the achievement of the target dynamics, while preserving stability in the presence of the model uncertainties. In general, the closed-loop behavior of a system cannot be shaped arbitrarily over an arbitrarily wide frequency range. It is proved that a special class of impedances that represent our set of performance specifications are mathematically achievable asymptotically through state-feedback and interaction-force feedforward as actuator bandwidths become large, and we offer a geometrical design method for achieving them in the presence of model uncertainties. The design method reveals a classical trade-off between a system's performance over a bounded frequency range and its stability relative to model uncertainties via multivariable Nyquits criteria. Two classes of such uncertainties are dealt with. While the first class of model uncertainties is formed from the uncertainties in the parameters of the modeled dynamics, the high-frequency unmodeled dynamics form the second class of model uncertainties. The multivariable Nyquist criterion is used to examine trade-offs in stability robustness against approximation of desired target impedances over bounded frequency ranges.

A method for the design of controllers of constrained manipulators in the presence of model uncertainties is developed. The controller must carry out fine maneuvers when the manipulator is not constrained, and compliant motion, with or without interaction-force measurement, when the manipulator is constrained. At the same time stability must be preserved if bounded uncertainties are allowed in modelling the manipulators. Stability of the manipulator and environment as a whole and the preservation of stability in the face of changes are two fundamental issues that have been considered in the design method. A set of practical design specifications in the frequency domain is presented that is meaningful from the standpoint of control theory and assures the desired compliant motion in the Cartesian coordinate frame and stability in the presence of bounded uncertainties. This approach also assures the global stability of the manipulator and its environment. The consequence of inexact achievement of performance specifications on stability is also specified. Part I concerns the fundamentals of compliant motion, while Part II is devoted to the controller design method.

A paradigm is developed for the problem of allocating in time a single resource to multiple simultaneous task demands which appear randomly, last for various periods, and offer varying rewards for service. Based upon a dynamic optimizing algorithm plus an estimator, and including response time and future discounting constraints, a model of the human decisionmaker is compared to experimental results for human subjects performing such a task at a computer-graphics terminal. Results indicate a reasonable fit, under various model parameters and task conditions, and suggest interesng hypotheses about the nature of human "planning ahead" and mental work load.

Criteria for selecting from alternative measures of plant information have been derived by information theory and the expected-value model of decision theory. Two criteria derived by information theory are minimum " uncertainty" of judgment of plant states, and maximum "surprise" of human operators. Two criteria derived by decision theory are maximum expected reward to operators, and maximum ratio of the maximum and next-to-maximum expected rewards. These four criteria have been evaluated experimentally in application to human monitoring problems of three "plants": an abstract laboratory exercise, a small research reactor, and a conventional nuclear power plant (high-fidelity training simulator). The results show that the criteria derived by decision theory are superior to the others from the viewpoint of correct rate of human subjects' judgment.