This article assessed the effectiveness of an augmented reality (AR) authoring tool for aviation weather training developed to support flight instructors without requiring technical expertise. While AR-based training showed promise in aviation weather education, there are limited tools available for nontechnologist instructors. To address this challenge, an instructor-centered AR authoring tool was developed and evaluated using a multiphase framework. The AR authoring approach was validated on three dimensions: instructors’ authoring experience; instructional quality of produced AR lessons; and students’ learning outcomes and engagement. Flight instructors successfully developed interactive AR lessons on aviation weather topics of their choice, and reported high usability and manageable workload for authoring. Instructors reported increased confidence in creating AR content, while their concerns regarding technical complexity, mental effort, and time demands decreased. Subject matter experts rated the instructional quality of instructor-authored AR lessons highly. Using the produced AR lessons, students showed significant gains in knowledge and motivation. These findings demonstrate the potential of instructor-centered authoring approach to enable the creation of high-quality, interactive educational content without requiring technical expertise, and to provide a replicable, evidence-based framework for assessing educational technologies and supporting broader integration of AR in education.

Cybersickness, or sickness caused by virtual reality (VR), represents a significant threat to the usability of VR applications. Repeated exposure to the same VR stimulus causes a reduction in cybersickness, referred to as Cybersickness Abatement from Repeated Exposure (CARE). This study examined whether the benefits of CARE generalize across distinct VR contexts, which was operationalized as three distinct games (a climbing game, a puzzle game, and a stealth survival game). Participants played a VR game for up to 20 minutes. Those in the Repeated Exposure condition played one VR game (either a puzzle game or a climbing game) on three separate days followed by a different VR game (a survival game) on the fourth day. Those in the Single Exposure condition played the survival game once. The three games all differed in several ways, including environment and task, whereas the puzzle and survival games shared a similar joystick locomotion interface that differed from the locomotion interface in the climbing game. Results indicate that cybersickness on Day 4 of the Repeated Exposure condition was significantly lower than that in the Single Exposure condition, regardless of which game was experienced on Days 1-3. The practical implication of this finding is that CARE that occurs in one VR context can generalize to a novel context with a distinct environment, task, and locomotion interface. Results are considered in the context of multiple theoretical explanations for CARE, including sensory rearrangement and habituation. These results support systematic exposure as an approach to reducing cybersickness.

Cybersickness, or sickness induced by virtual reality (VR), negatively impacts the enjoyment and adoption of the technology. One method that has been used to reduce sickness is repeated exposure to VR, herein Cybersickness Abatement from Repeated Exposure (CARE). However, high sickness levels during repeated exposure may discourage some users from returning. Field of view (FOV) restriction reduces cybersickness by minimizing visual motion in the periphery, but also negatively affects the user's visual experience. This study explored whether CARE that occurs with FOV restriction generalizes to a full FOV experience. Participants played a VR game for up to 20 minutes. Those in the Repeated Exposure Condition played the same VR game on four separate days, experiencing FOV restriction during the first three days and no FOV restriction on the fourth day. Results indicated significant CARE with FOV restriction (Days 1--3). Further, cybersickness on Day 4, without FOV restriction, was significantly lower than that of participants in the Single Exposure Condition, who experienced the game without FOV restriction only on one day. The current findings show that significant CARE can occur while experiencing minimal cybersickness. Results are considered in the context of multiple theoretical explanations for CARE, including sensory rearrangement, adaptation, habituation, and postural control.

Multiple tools are available to reduce cybersickness (sickness caused by virtual reality), but past research has not investigated the combined effects of multiple mitigation tools. Field of view (FOV) restriction limits peripheral vision during self-motion, and ample evidence supports its effectiveness for reducing cybersickness. Snap turning involves discrete rotations of the user's perspective without presenting intermediate views, although reports on its effectiveness at reducing cybersickness are limited and equivocal. Both mitigation tools reduce the visual motion that can cause cybersickness. The current study (N = 201) investigated the individual and combined effects of FOV restriction and snap turning on cybersickness when playing a consumer virtual reality game. FOV restriction and snap turning in isolation reduced cybersickness compared to a control condition without mitigation tools. Yet, the combination of FOV restriction and snap turning did not further reduce cybersickness beyond the individual tools in isolation, and in some cases the combination of tools led to cybersickness similar to that in the no mitigation control. These results indicate that caution is warranted when combining multiple cybersickness mitigation tools, which can interact in unexpected ways.

Apple's 1987 Knowledge Navigator video contains a vision of a sophisticated digital personal assistant, but the natural human-agent conversational dialog shown does not currently exist. To investigate why, the authors analyzed the video using three theoretical frameworks: the DiCoT framework, the HAT Game Analysis framework, and the Flows of Power framework. These were used to codify the human-agent interactions and classify the agent's capabilities. While some barriers to creating such agents are technological, other barriers arise from privacy, social and situational factors, trust, and the financial business case. The social roles and asymmetric interactions of the human and agent are discussed in the broader context of HAT research, along with the need for a new term for these agents that does not rely on a human social relationship metaphor. This research offers designers of conversational agents a research roadmap to build more highly capable and trusted non-human teammates.

User interface design patterns that document Human-Agent Team (HAT) interface best practices were developed by analyzing team video games with different types of agents, ranging from virtual players in sports games to advisors contributing to strategy and planning in combat games. One factor that may contribute to the enormous success of some games is the cohesive team interactions between humans and agents. A design pattern collection was developed by identifying 25 design patterns under interface themes most relevant to HATs. The design patterns describe how multiple successful game interfaces handle challenging aspects of HAT interaction, addressing contexts including planning, spatial orientation, task management, communication, and more. Based on the complex and well-tested interface solutions in video games, these design patterns can provide HAT user interface developers practical guidance for interface development and a starting point for developing a HAT design pattern language.

A Playbook delegation approach was evaluated for human-autonomy teaming (HAT) in Single Pilot Operations (SPO). In SPO, a single pilot makes flight decisions, performs flight tasks, and collaborates with an autonomous teammate. The autonomous teammate shares responsibility, authority, and tasks. Challenges include the design of functions, interactions, and teaming skills. HAT often requires the ability to dynamically allocate functions, and timely methods to accurately express intent to teammates. A Playbook delegation interface was developed to enables the pilot to call and modify plays in collaboration with the autonomous teammate. Twenty pilots evaluated the Playbook interface to explore real-time function allocation, and identified teaming skills needed to support HAT. Pilots preferred the Playbook interface for better collaboration with the autonomous teammate. Interviews revealed that supporting human-like communication in HAT is critical to facilitate decision-making. Four major teaming skills (communication, coordination, cooperation, and cognition) are discussed to support HAT in SPO.

Sensor-based vision systems enable approaches to altitudes closer to the runway in low visibility conditions. These systems utilize imaging sensors capable of providing forward vision of the runway and the resulting imagery is displayed on a head-up display. Previous studies were primarily limited to nominal cases. Off-nominal cases assessments are limited only to display failures. The sensors integrated to these systems have known limitations and can produce degraded display output with respect to atmospheric conditions. An evaluation is conducted to assess the potential human factors implications of a simulated display of poor sensor output, operationalized as a difference in contrast ratio of the display output for simulated approach and landing operations. Pilots fly six different approach and landing scenarios in a simulator for two visibility levels and three levels of display information quality (none, poor, and good display output). Measures of performance include approach and landing performance, attention allocation, workload, and decision-making. The experiment results indicate that landing performance and decision making are negatively impacted by poor display output, while there is no evidence of an impact on workload or situation awareness.

An empirical study evaluated key human factors issues related to automation visibility and information quality, based on a refined definition of information automation. Next-generation air transportation system operational concepts will dramatically affect the types and amount of information available on flight decks. Information automation systems collect, process, and present information to support pilot tasks and awareness. The definition of flight deck information automation was refined to differentiate it from other types of automation. Pilots interacted with an example information automation system to investigate the premise that automation visibility will have an impact on the ability of pilots to detect problems resulting from poor information quality. Poor information quality appeared to be difficult for pilots to detect, even when presented with high automation visibility. Pilots tended to over-trust automation, so when reporting high workload and information was missing, they chose the top plan suggested by the automation even though it was not the best. Trust in automation was reduced by low information quality, but compensated for by increased automation visibility. Added information to help pilots understand information automation state and outputs, given a level of information quality, should be balanced against potential increases in pilot workload due to the time and attention needed to process the extra information.

This work developed and evaluated a human-machine interface for the control of a roadable air vehicle (RAV), capable of surface driving, vertical takeoff, sustained flight, and landing. Military applications seek to combine the benefits of ground and air vehicles to maximize flexibility of movement but require that the operator have minimal pilot training. This makes the operator vulnerable to automation complexity issues; however, the operator will expect to be able to interact extensively and control the vehicle during flight. A mixed-initiative control approach mitigates these vulnerabilities by integrating the operator into many complex control domains in the way that they often expect---flexibly in charge, aware, but not required to issue every command. Intrinsic safety aspects were evaluated by comparing performance, decision making, precision, and workload for three RAV control paradigms: human-only, fully automated, and mixed-initiative control. The results suggest that the mixed-initiative paradigm leverages the benefits of human and automated control while also avoiding the drawbacks associated with each.

A method was developed to produce a set of consistent and independent characteristics of adaptive systems. Previous work identified human performance risks and benefits of adaptive systems through a systematic analysis of 25 characteristics. A potential limitation of the original analysis was the reliance on 25 characteristics that were potentially inconsistent. It could be that some characteristics were redundant, thereby giving more weight to the riskier characteristics, and increasing the average risk rating. A method was developed to evaluate each of the characteristics from a usability perspective. The results were then used to generate a Pearson's rank correlation of each pair wise combination of characteristics. The utility of the method was triage for identifying those pair wise comparisons needing further human factors analysis without having to consider each combination. A final set of seven consistent and independent characteristics were defined.

The Function Allocation Matrix Tool (FAMT) is applied as the basis of a reach zone analysis for the controls of the Orion Crew Exploration Vehicle. NASA's Constellation Program intends to return humans to the moon by 2020, followed by exploration to Mars and beyond. Orion will serve as the primary crew transport vehicle, and will be equipped with a modern `glass cockpit' to allow operators to command and control all of the vehicle's systems from one of two operator stations. It will have a fraction of the buttons, switches, and dials found on the Space Shuttle flight deck. Instead, operators will monitor and command the vehicle's systems via graphics-based displays; thus the design of Orion's displays and controls places an increased emphasis on human-computer interaction and usability. It is therefore necessary to place controls in appropriate reach zones, defined as three-dimensional envelopes that define reach limits and boundaries within the Orion cockpit operator station. The FAMT is a systematic analysis tool with which to determine the appropriate location of cockpit controls and displays needed to support mission priorities. While the tool was originally designed for investigating incremental functionality additions to existing cockpits, it has been applied to Orion toward design of a completely new cockpit, and has played a crucial role in determining the minimum recommended reach zones for each control needed by the operator to complete their assigned tasks.

Honeywell has designed a joint human-computer cognitive system to support rapid decision making demands of dismounted soldiers. In highly networked environments the sheer magnitude of communication amid multiple tasks could overwhelm individual soldiers. Key cognitive bottlenecks constrain information flow and the performance of decision-making, especially under stress. The adaptive decision-support system mitigates non-optimal human performance via automation when the system detects a breakdown in the human's cognitive state. The human's cognitive state is assessed in real-time via a suite of neuro-physiological and physiological sensors. Adaptive mitigation strategies can include task management, optimizing information presentation via modality management, task sharing, and task loading. Mitigations are designed with consideration for both the costs and benefits of intermittent augmentation. The paper describes the system development and evolution, explorations of usable cognitive mitigation strategies, and four evaluations that show adaptive automaton can effectively, mitigate human decision-making performance at extremes (overload and underload) of workload.

A conceptual framework for designing human-computer cognitive systems is proposed. The Cognitive Bottleneck Framework (CBF) identifies four significant "cognitive bottlenecks" that negatively impact the quality and tempo of decision making: (1) Information overload; humans cannot manage the vast amounts of information delivered by their computing environment, (2) Sequential cognitive processing; while information arrives in parallel, humans are essentially serial processors that can only address a single thread or task at a time, (3) Narrow user input capabilities: the system has more sophisticated means of communicating to the human than the human has of communicating to the system, and (4) Function mis-allocation; tasks are allocated to humans by default rather than by design, leaving them with tasks for which they are cognitively ill-suited. The overall purpose of CBF is to "right-size" these bottlenecks, remove constraints that restrict information flow, and better fit information channels to the abilities of either the human or computer.

This paper discusses the evolution of the Alerting and Notification of Conditions Outside the Aircraft (ANCOA) concept, based on an integrated alerting framework. The development of ANCOA was prompted by the alert proliferation and the lack of alert standardization as independent alerting systems for conditions outside the aircraft (e.g. TCAS, EGPWS) are introduced onto the flight deck. These separate alerting systems, each using different alerting and display philosophies to present information to the crew, have no integrated standard protocol for prioritizing and organizing the information to reduce the demands on pilot attention and information processing. In response, a framework that integrated, organized, prioritized, and displayed information from new flight deck alerting systems was developed. ANCOA exemplifies the framework by integrating information from independent alerting systems to enable multiple alerts to be prioritized and de-conflicted, in order to support prompt and appropriate responses to adverse conditions based on good situation awareness.

This paper describes the diversion off-gate management assistant (DOGMA) system a decision-support tool that uses airline-articulated policies to critique dispatcher initiated diversion decisions to ensure that these plans will not only maintain safe operating practices, but will also go further toward providing decision makers with the broad and diverse set of concerns from various stakeholders in their diversion decisions. Diversion management is the decision of which incoming flights to divert and where to divert them to in the face of reduced landing capacity at original destination airports. Policy is used to capture the goals and priorities of all stakeholders that currently do not have a prominent voice in diversion decisions. DOGMA is a policy based system that interacts with the user via a critiquing approach in which dispatchers' input to a diversion plan is reviewed by a computer partner that offers feedback on the potential solutions and their ramifications for all stakeholders. The impact of this increased awareness and broader input into the diversion decision should be better and more consistent diversion decisions that minimize the negative impact of diversions and improve an airline's ability to recover from severe schedule disruptions.

This paper describes a design process to support the development of a learning collaboratory, a distributed, computer-based, virtual space for learning and work. A learning collaboratory, as a distributed distance learning environment, offers great opportunities to expand the way people teach and learn and to broaden educational opportunities to an ever increasing range of learners. The challenge is to design distance learning technologies that engender meaningful learning experiences that take full advantage of the power of computer-mediated communication to support innovative learner-centered and collaborative interactions between students, teachers, subject experts, and resources. First, the paper describes the learning collaboratory design framework (LUCIDIFY), a design process that integrates methods and concepts from cognitive systems engineering, theories of learning and instruction, distributed computing, and computer-supported collaborative learning to guide the principled design of learning collaboratories. Next, the paper describes how LUCIDIFY was used in the design and implementation of the collaborative learning environment for operational systems (CLEOS), a learning collaboratory for teachers, students, and practitioners in the physical sciences. CLEOS features two virtual instrument tutorials, an asynchronous messaging system, a project-based design and management application, and a collaborative multi-user domain infrastructure.

Describes the Collaborative Apprenticeship Learning Object Toolkit (CALOT), an architecture that forms the basis of a family of tutoring systems. It utilizes the apprenticeship model of learning for the study of operational procedures where theory and practice are both important. Systems built upon the CALOT architecture can be utilized by multiple, non-colocated students in a synchronous manner. A distributed set of students can collaborate on the tutorial in real-time. The development of this type of computer tutorial system is driven by the goal of linking theoretical knowledge with practical operational experience. Active, exploratory, apprentice-style learning is supported via modes of operation within the system. The student can flexibly choose to observe the expert perform and explain operational steps, or to act as an apprentice and carry out the steps autonomously. The student can switch between these modes at their discretion, giving the student control of the level of intervention by the system. Furthermore, the student can explore and reflect on an information space of object designations, procedures and related concepts. Two examples tutorials based on the CALOT architecture are the Collaborative Virtual Spectrometer and the Virtual X-Ray Diffractometer, which teach about NMR spectroscopy and X-ray diffraction, respectively. Finally, BAUEN (Basic AUthoring ENvironment) is a GUI system that helps the software developer designer author a CALOT-based computer tutorial system with maximum efficiency, focusing the developer's efforts on domain content, and not on the system implementation.

Early CAI efforts focused on the transmission or delivery of knowledge to the student. Subsequent work in intelligent tutoring systems took a more cognitive approach and attempted to emulate the behavior of skilled human tutors via software. More recent work has focused on collaborative learning processes, and the idea of a collaboratory as a virtual space for work. This paper describes a design process to support the development and use of collaborative learning technologies. This process, the Learning Collaboratory Design Framework (LuCiDiFy), integrates methods and concepts from cognitive systems engineering, theories of learning and instruction, distributed computing and computer-supported collaborative learning (CSCL). LuCiDiFy is instantiated in an actual software testbed, the Collaborative Learning Environment for Operational Systems (CLEOS). CLEOS is a collaboratory for teachers, students and practitioners in the physical sciences, and in particular for NMR spectroscopy and X-ray diffraction experiments. CLEOS includes two virtual instrument tutorials, CVS (Collaborative Virtual Spectrometer) and VXRD (Virtual X-Ray Diffractometer); an asynchronous messaging system, the Question Board (QB); a project-based design environment, TOSP (Tool for Organizing and Supervising Projects); and an architecture for developing collaborative apprenticeship learning systems, CALOT (Collaborative Apprenticeship Learning Object Toolkit); an authoring environment, BAUEN (Basic AUthoring ENvironment); and a collaborative multi-user domain (MUD) infrastructure, MudSpot.

The paper discusses the domain of Army logistics planning at the level of the ODSCLOG (Office of the Deputy Chief of Staff for Logistics) and describes tools to assist in this planning. In particular, we have developed a prototype collaboratory (CLINT) for logistics planning that relies on Microsoft NetMeeting/sup TM/ as the collaborative infrastructure, and a tool named THREADS to support logistics problem solving. THREADS is a "visual simulation" of logistics routes that allows planners to quickly identify bottlenecks and compare alternative plans.

The emergence of computer technology has had profound implications for instruction. Recent efforts in the field have shifted the focus from knowledge-as-a-product to learning-as-a-process. Computer-supported collaborative learning (CSCL) focuses on learning as a collaborative activity that is situated in its environment. The paper describes the Apprenticeship Learning Object Toolkit (ALOT), a generalized architecture, that forms the basis of a family of computer tutoring systems. The system utilizes the apprenticeship model of learning to create a learning environment for the study of operational procedures of experiments in the physical sciences. Examples include nuclear magnetic resonance (NMR) spectroscopy or x-ray diffractometry (XRD). The development of a learning environment is driven by the goal of linking theoretical knowledge with practical operational experience. Active, exploratory, apprentice-style learning is supported via modes of operation within the system. The student can flexibly choose to "observe the expert" perform and explain operational steps, or "act as an apprentice" and carry out the steps autonomously. The student can switch between these modes at their discretion, giving the student control of the level of intervention by the system. In addition, the student can explore and reflect on an "information space" of object designations, procedures, and related concepts. Additionally, the BAUEN (basic authoring environment) is introduced. BAUEN is a graphical user interface system that is intended to help the designer author a computer tutorial system with maximum efficiency, focusing the developer's efforts on domain content and not on system implementation.

This work presents the development of a multi-user virtual environment in support of crisis management planning activities. Coupling ongoing work in activity representation with analysis of the domain, an extensible domain ontology was developed. Studying the domain, several challenges became apparent: (1) the need to support extensive dynamic and distributed collaboration, (2) the need for a flexible, open architecture, (3) the need for views of the information tailored to the activities of the planning team, and (4) the opportunity to leverage past and current related efforts in the domain. This project developed a Java-based collaboration architecture around a multi-user domain (MUD) to provide presence and access to collaborative services. The collaborative infrastructure layer provides persistence, user authentication, and access control. Built upon this substrate are collaborative services. The target use of MAST is to represent and track workflow in the crisis action planning done by the Operations Planning Team of the US Pacific Command. The design approach was to base MAST on a reusable class library which implements a rich ontology structure and represents key elements of the domain. Our ontology draws on the SPOT project and the Shared Planning and Activity Representation (SPAR) project. The resulting ontology, implemented in Java, is compliant with the SPAR Reference Object Model Specification. A strength of this work is the extensibility, flexibility and openness both of the representation and the system architecture.