This article provides a systematic review of research related to Human–Computer Interaction techniques supporting training and learning in various domains including medicine, healthcare, and engineering. The focus is on HCI techniques involving Extended Reality (XR) technology which encompasses Virtual Reality, Augmented Reality, and Mixed Reality. HCI-based research is assuming more importance with the rapid adoption of XR tools and techniques in various training and learning contexts including education. There are many challenges in the adoption of HCI approaches, which results in a need to have a comprehensive and systematic review of such HCI methods in various domains. This article addresses this need by providing a systematic literature review of a cross-section of HCI approaches involving proposed so far. The PRISMA-guided search strategy identified 1156 articles for abstract review. Irrelevant abstracts were discarded. The whole body of each article was reviewed for the remaining articles, and those that were not linked to the scope of our specific issue were also eliminated. Following the application of inclusion/exclusion criteria, 69 publications were chosen for review. This article has been divided into the following sections: Introduction; Research methodology; Literature review; Threats of validity; Future research and Conclusion. Detailed classifications (pertaining to HCI criteria and concepts, such as affordance; training, and learning techniques) have also been included based on different parameters based on the analysis of research techniques adopted by various investigators. The article concludes with a discussion of the key challenges for this HCI area along with future research directions. A review of the research outcomes from these publications underscores the potential for greater success when such HCI-based approaches are adopted during such 3D-based training interactions. Such a higher degree of success may be due to the emphasis on the design of user-friendly (and user-centric) training environments, interactions, and processes that positively impact the cognitive abilities of users and their respective learning/training experiences. We discovered data validating XR-HCI as an ascending method that brings a new paradigm by enhancing skills and safety while reducing costs and learning time through replies to three exploratory study questions. We believe that the findings of this study will aid academics in developing new research avenues that will assist XR-HCI applications to mature and become more widely adopted.

In the past decade, Extended Reality (XR), through rapid technological advancement, has managed to make modest changes in surgical training approaches. Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR)-based simulators have been adopted for training in various surgical domains such as brain, eye, laparoscopic, and orthopedic surgery among others. However, there is a lack of effort in exploring Human Extended Reality Interaction (HXRI)-based concepts to design effective training environments that corroborate the actual training procedure. In this paper, an HXRI-based multi-level approach has been proposed for the creation of XR-based training environments. The multi-level approach takes advantage of both Virtual and Mixed Reality to create a holistic training environment while eliminating the disadvantage arising from using stand-alone VR and MR-based training environments. The approach was based on the results of an exhaustive study in which medical personnel interacted with VR and MR environments to train in an orthopedic surgical procedure. The approach, design, and development of the XR environments, the study, and the results from the study which underscore the importance of such a multi-level approach are presented in the paper.

The potential of adopting cyber learning approaches to improve student learning and engagement is an important area of educational research. The advent of Virtual Reality (VR) based learning environments or Virtual Learning Environments (VLEs) has ushered in the cyber-based educational era which involves smart technologies supporting experiential learning and other approaches to enrich the learning experiences in Science, Technology, Engineering, and Mathematics + Computing (STEM+C) topics. In this paper, the discussion of such experiential activities to teach advanced concepts in cyber-physical systems using process contexts from NASA's Moon Mission is discussed. There are two thrusts: (1) studying the potential of experiential activities where engineering and computer science students are taught to design and build such cyber-physical environments using Mixed Reality (MR) involving astronaut training scenarios (2) assessing the impact of immersive VLEs using low-cost platforms such as the Vive in learning science and engineering concepts. The results of such learning experiences in terms of engagement and learning on students are discussed. The overall conclusion is that such experiential activities involving VR/MR technologies have the potential to impact student learning in a positive manner.

This paper discusses the design of Virtual Learning Environments (VLEs) in helping students with Autism learn Science and Engineering concepts. The design of the environments was based on information centric principles. Further, human centered computing principles (HCC) were explored during the development of the VLEs. HCC principles such as affordance, visual density and cognitive load were taken into consideration during the design process. The VLEs were created for middle and high school students. An information-centric model was created to understand the process of designing and building the VLEs. Such information models based on engineering Enterprise Modeling Language (eEML) provided a structural foundation for the design and development of the VLEs. The learning environments were created using various interfaces and immersion levels; these included haptic based interfaces, fully immersive 3D environments and Augmented Reality (AR) based environments. These VLEs introduced students to concepts in assembly in the medical context and path-planning and navigation in the context of NASA's moon mission. Assessment activities were conducted to gain a better understanding of the impact of such VLEs on the learning of science and engineering concepts to students with Autism. The preliminary results of the assessment activities demonstrated the positive impact of such cyberlearning techniques and environments on the learning of students with Autism.

This paper discusses the design of Virtual Learning Environments (VLEs) in helping students with autism learn Science and Engineering concepts. The VLEs were created for middle and high school students; subsequently assessment activities were conducted to gain a better understanding of the impact of such VLEs. The learning environments were created using various interfaces and immersion levels; these included haptic based interfaces and fully immersive 3D environments. These VLEs introduced students to concepts in Robotics and Manufacturing. Initial assessment findings have demonstrated the positive impact of such cyber learning techniques and environments.

This paper focuses on the creation of information centric Cyber-Human Learning Frameworks involving Virtual Reality based mediums. A generalized framework is proposed, which is adapted for two educational domains: one to support education and training of residents in orthopedic surgery and the other focusing on science learning for children with autism. Users, experts and technology based mediums play a key role in the design of such a Cyber-Human framework. Virtual Reality based immersive and haptic mediums were two of the technologies explored in the implementation of the framework for these learning domains. The proposed framework emphasizes the importance of Information-Centric Systems Engineering (ICSE) principles which emphasizes a user centric approach along with formalizing understanding of target subjects or processes for which the learning environments are being created.

Simulation-Based Design (SBD) approaches can help conceptualize engineering plans and design activities with the adoption of Virtual / Augmented Reality based digital mockup techniques and technologies. This paper outlines preliminary activities based on SBD principles aimed at supporting some of the material transport activities for NASA's Lunar Mission. Immersive Virtual Reality (VR) based simulation environments were created to support some of the simulation based design activities. The engineering design activities included (i) planning and conceptualizing an autonomous system for transporting payloads from the Gateway to the Lunar surface, (ii) exploring design of cable-based robotic system for transporting parts from one location to another on the surface of the Moon and (iii) studying path planning approaches to transport materials on the Lunar surface. The initial conceptualization of design ideas for these three thrusts are presented along with a discussion of the SBD implementation using the Unity engine on the immersive Vive platform.

The design and implementation of an Internet of Things (IoT) based cyber physical framework in the context of Industry 4.0 is discussed for the field of micro devices assembly. Such frameworks hold the potential to facilitate rapid and agile collaborations among distributed engineering partners. This paper outlines the key cyber and physical components which collaborate using cloud-based principles and emerging next generation global environment for network innovation Internet technologies. An information centric systems engineering approach is proposed to help design the cyber physical interactions, which provide a foundation for implementing this cyber physical framework. The cyber modules are capable of assembly planning, path planning, virtual reality-based assembly simulation, and physical command generation. The physical assembly activities are accomplished using micro assembly work cells. An IoT-based cyber physical test bed has been created to test and validate the design and implementation aspects of the proposed framework.

The objective of this research is to build and demonstrate a design tool in a Virtual Reality (VR) environment. The goal of the tool is to streamline the assembly planning of complex systems. As a case study, we have generated a virtual reality environment for an assembly simulation of a zero-G treadmill for use on spacecraft. This study will help to understand the assembly of the VR model that might lead an assembler to optimize the design related costs. The VR environment has been created using Unity 3D along with Solidworks for creating models of Treadmill parts. A series of controlled user studies has been done to investigate the role of different visual cues (image and text) on user performance while performing manual assembly in an immersive VR setting (VIVE headset) and a non-immersive environment (desktop). To calculate the groundtruth, an optimized path sequences is computed using a genetic algorithm for a collision free layout.

This paper discusses the adoption of Simulation Based Design (SBD) approaches involving 3D Virtual Reality (VR) based digital mockup models to support the study of transportation and assembly alternatives in the context of Deep Space Missions as well as landing on the Moon. An initial set of designs have been proposed using immersive VR environments. The modeling, planning and simulation scope includes the transfer of equipment from the lunar lander to the moon surface, design of astronaut habitats and assembly of such habitats using robots. Using this SBD approach, the project team proposed, compared and analyzed transport and assembly alternatives for these design contexts. The simulation environments were built using the immersive Vive VR platform and Unity 3D software tools.

Simulation systems in medical training are designed for residents and others to become more skilled in complex procedures such as surgery. The surgical context for training discussed in this paper is Condylar plating surgery which deals with treating fractures of the femur bone. This fully immersive environment has been developed using the Vive VR platform. Learning studies conducted with orthopedic residents and medical students underscored the potential of adopting such emerging platforms for simulation based training activities.

The focus of this article is on the adoption of immersive and haptic simulators for training of medical residents in a surgical process called Less Invasive Stabilization System (LISS) plating surgery . LISS surgery is an orthopedic surgical procedure to treat fractures of the femur bone. Development of such simulators is a complex task which involves multiple systems, technologies, and human experts. Emerging Next Generation Internet technologies were used to develop the standalone on-line haptic-based simulator accessible to the students 24/7. A standalone immersive surgical simulator was also developed using HTC Vive. Expert surgeons played an important role in developing the simulator system; use cases of the target surgical processes were built using a modeling language called the engineering Enterprise Modeling Language (eEML) . A detailed study presenting the comparison between the haptic-based simulator and the immersive simulator has been also presented. The outcomes of this study underscore the potential of using such simulators in surgical training.

Autism and Autism Spectrum Disorders (ASD) are general terms for a group of complex disorders of brain development. Autistic children exhibit certain characteristics in varying degrees including difficulties in verbal/non-verbal communication, social interaction and repetitive behaviors. This paper discusses the role of Virtual Learning Environments (VLEs) in helping autistic children learn science and engineering concepts. VLEs are a type cyber learning environments created using Virtual Reality technology; as part of a learning activities, a set of VLEs to teach autistic students concepts in related to the solar system, robotics and density has been developed; assessment results underscore the potential of such VLEs to support science and engineering learning.

The emergencelof Virtual Reality (VR) based technologies holds the potential to facilitate global collaboration in various fields of engineering. Micro Devices Assembly (MDA) is an emerging domain involving the assembly of micron sized objects and devices. In this paper, the focus of the discussion is the design of a VR based Internet-of-Things (IoT) based framework to support collaborative assembly of micro devices using both cyber and physical resources.

Advances in the field of cloud computing and networking have led to rapid development and market growth in areas such as online retail, gaming and healthcare. In the field of advanced manufacturing however, the impact has been significantly lesser than expected due to limitations in cloud platforms for fostering community engagement. To address this problem, we study a new cloud-based architecture that provides Platform-asa-Service (PaaS) management capabilities to the manufacturing community for delivering Software-as-a-Service (SaaS) “Apps” to their customers. Our architecture aims at supporting an “App Marketplace” that thrives on agile development, organic collaboration and scalable sales of next generation manufacturing Apps requiring high-performance simulation and modeling. Towards realizing the vision of the above architecture, our paper involves investigation and implementation of an Ontology Service that interoperates with other common web services related to resource brokering and accounting. Our Ontology Service uses principles of mapping and merging to translate a manufacturing App's collaboration requirements to suitable resource specifications on public cloud platforms. Integrated resultant ontology can be queried to provision the required resource parameters such as amount of memory/storage, number of processing units, and network protocol configurations needed for deployment of an App. We validate the effectiveness of our Ontology Service using the Protégé framework in a pilot testbed of a real-world “WheelSim” App in the NSF GENI Cloud platform. Our ontology integration results show benefits to an App developer in terms of: optimal user experience, lower design time and lower cost/simulation.

This paper discusses an innovative approach to teach engineering concepts using Virtual Reality based Learning Environments (VLEs). New learning modules have been created using Virtual Reality technology and introduced in interdisciplinary senior level and graduate level courses targeting mechanical, industrial and electrical engineering students. These Virtual Reality based learning environments have been used to teach micro systems related topics as part of overall efforts to enhance the learning experiences of students. The learning outcomes including student performance are discussed. The process undertaken to design and develop these VLEs are elaborated along with the technologies used to develop such environments. A brief discussion of next generation Internet technologies which hold the potential to impact engineering and K-12 education is also provided.