A Juxtopia research team collaborated with the Maryland Fire & Rescue Institute (MFRI) to test how the Juxtopia® wearable augmented reality (AR) intervention may better deliver a hands-on clinical training intervention to firefighter Emergency Medical Technicians (EMT) and prepare them for effective response to HAZMAT (hazardous materials) incidences. During a controlled study, human subjects participated in a minimal risk research study (i.e., both as victims or caregivers) in which firefighter EMTs participated in a simulated training exercise that mimicked their real-world operations. This study consisted of two testing days. Day one included 10 participants that completed a full day of training and familiarity with wearable augmented reality (AR) goggles and a Juxtopia® Virtual Tutor (JVT) software application, which is derived from the Juxtopia® Intelligent Virtual Instructor (JiVi) platform. Day two included the completion of four psychomotor clinical skills performed by EMTs. It also included the application of moulage injuries for victims, using the Juxtopia® CAMMRAD PREPARE system to administer clinical skills on patient actors. This paper discusses the research methods and results.

One major cause of preventable death is a lack of proper skills for providing critical care. Conventional training for advanced emergency medical procedures is often limited to a verbal block of instructions and/or an instructional video. In this study, we evaluate the benefits of using an optical see-through head-mounted display (OST-HMD) for training of caregivers in an emergency medical environment. A rich user interface was implemented that provides 3D visual aids including images, text and tracked 3D overlays for each task. A user study with 20 participants was conducted for two medical tasks, where each subject received conventional training for one task and HMD training for the other task. Our results indicate that using a mixed reality HMD is more engaging, improves the time-on-task, and increases the confidence level of users.

Precise tumor identification is crucial in image-guided neurosurgical procedures. With existing navigation systems, the surgeon must turn away from the patient to view the imaging data on a separate monitor. In this study, an innovative system is introduced that illustrates the tumor boundaries precisely augmented on the spot where the tumor is located with regard to the patient. Additionally, it allows the surgeon to track the distal end of the tools contextually, where direct visualization is not possible. In this approach, the tracking system is compact and worn by the surgeon, eliminating the need for additional devices that are bulky and typically limited by line of sight constraints.

Scaffolding is a well-established instructional approach that facilitates learning by incrementally removing training aids as the learner progresses. By combining multiple training aids (i.e. multimodal interfaces), a trainer, either human or virtual, must make real-time decisions about which aids to remove throughout the training scenario. A significant problem occurs in implementing scaffolding techniques since the speed and choice of removing training aids must be strongly correlated to the individual traits of a specific trainee. We detail an agent-based infrastructure that supports the customization of scaffolding routines per individual user. We describe the integration of this agent-based approach into a simulated augmented reality (AR) environment

Virtual instructor enabled mobile augmented reality systems (MARS) have the potential to provide continuous and autonomous instruction to human learners anytime, anyplace, and at any-pace. MARS based learning provides the advantage of a natural human-computer interface, flexible mobility, and context-aware instruction allowing learners to interact with their natural environment with augmented perceptual cues. These perceptual cues combining multi-modal animation, graphics, text, video, and voice along with empirical pedagogical techniques can elegantly orchestrate a mobile instructional tool that facilitates life-long learning. The challenge, however, is building a mobile instructional tool with capabilities for adapting to various learning environments ranging from traditional schools and outdoor learning to the workplace while also considering the cultural, geographical, and other contexts about the learner. This paper discusses a novel system/software architecture, CAARS, for developing context-aware mobile augmented reality instructional systems

A pervasive virtual instructor is an artificially intelligent instructor that may appear transparent to the learner or appear in the form of a three-dimensional graphical character, digital toy, or robot with capabilities to inhabit mixed reality environments, and provide personalized instruction anytime, anywhere, and at an-pace. Similarly to pedagogical agents, or traditional virtual instructors, pervasive virtual instructors are expected to behave autonomously, respond to human verbal/non-verbal input, and deliver information to human users. Unique to pervasive virtual instructors in this paper are capabilities to provide instruction across distributed networks, interact with human learners using context-aware intelligence, and apply empirically researched pedagogical/andragogical techniques. Technology challenges remain for building pervasive virtual instructors to achieve aforementioned capabilities. This paper summarizes technical challenges and an approach for building pervasive instructors that provide life long instructional services

Mobile augmented reality system (MARS) based e-learning environments equip a learner with a mobile wearable see-through display that interacts with training/learning software. MARS has the potential to adapt to individual learner needs and dynamically distribute tailored instruction to improve learning performance for a life time. While using MARS, learners may interact with their natural environment while MARS digitally annotates real-world objects with digital content. This digital content may combine multi-modal animation, graphics, text, and video as well as voice used to augment instruction based on empirical pedagogical models. The challenge, however, is building a MARS software architecture that fulfills this potential and is also reusable, interoperable, and adaptive to individual augmented reality (AR) "heads-up displays" while, at the same time, capable of delivering personalized instruction to the learner

Problem solving using LEGOS is a 3-credit hour course offered to any student at Pace University of Pleasantville, New York. The class uses a project-based learning environment, which consists of four design projects. This paper discusses the pedagogical advantages of teaching robotics in a mixed-reality environment with a virtual instructor as opposed to teaching robotics in a traditional laboratory setting. The implication of this learning intervention is the personalized and immersive learning experience that has the potential for strengthening problem solving skills using science, technology, engineering, and math (STEM) for life-long STEM proficiency

Problem solving using LEGOS is a 3-credit hour course offered to any student at Pace University of Pleasantville, New York. The class uses a project-based learning environment, which consists of four design projects. This paper discusses the pedagogical advantages of teaching robotics in a mixed-reality environment with a virtual instructor as opposed to teaching robotics in a traditional laboratory setting

A virtual reality instructor that autonomously trains a human learner in network virtual environments, respond to multi-modal input across computer networks, and applies proven pedagogical techniques during instruction has the potential to improve human learning performance anytime, anywhere, and at any pace. Building virtual instructors, however, have challenged researchers because of multidisciplinary expertise required in areas such as education, cognitive science, sociology, artificial intelligence, 3D computer graphics, linguistics, and more. This paper discusses a model for building human computer interactive virtual instructor systems using an innovative, system/software architecture.