This Work-in-Progress Research paper explores how to design and implement Collaborative Virtual Reality Learning Environments (CVRLE) taking Teacher-Student Dynamics (TSD) into account. CVRLEs engage students in a virtual representation of a learning space. However, they do not incorporate TSDs, which are key elements of in-person teaching. The current work explores how to recreate one specific TSD: the watchfulness teachers have over the students' location during a field trip. We propose and compare seamless approaches to recreate this TSD inside CVRLEs via virtual safety boundaries around the teacher. These boundaries keep students in proximity to their instructor as they explore the CVRLE. The current work explores five approaches: 1) an invisible wall (used as baseline), 2) sound-based feedback, 3) a virtual companion, 4) a translucent rope connected to the teacher and 5) a translucent dome mesh around the teacher. The approaches were validated by five middle school teachers, who provided feedback about them from an educator's point of view. The interviews highlighted the importance of subtle and simple approaches that provide a sense of freedom to the students and minimize cognitive load while maintaining the TSDs. Overall, the vision of this Work-in-Progress is to solidify CVRLEs into a plausible method to perform teaching and learning when co-presence between teacher and students cannot be guaranteed. The design guidelines from this project will inform the creation of future interactive CVRLEs.

Proposed here is a come-and-go demo of the tool used in the classroom as discussed in the paper “Towards an Intelligent Tutoring System for Virtual Reality Learning Environments”. The workshop/demo will introduce participants to the immersive VRLE (Virtual Reality Learning Environment) activity with several VR (Virtual Reality) headsets available for participant engagement in the VR environment. Developers and users will be available for discussion and questions. The workshop team will be discussing findings from the literature related to the tool and exploring participant thoughts and observations relative to their experiences with the immersive VR tool. Participant engagement may be tailored to their appetite for exploration. They may work through all elements of the tools, watch developers/users work through particular areas, or any mix in between. They can participate for the full session or depart as their interest-level is satisfied. Existing literature will be made available and potential for bringing the activity to their classrooms can be discussed.

This Work-in-Progress Research paper explores the development of an Intelligent Tutoring System (ITS) inside a Virtual Reality Learning Environment (VRLE). As technology becomes more pervasive in teaching, platforms that afford immersive learning experiences, such as VRLEs, are receiving increased attention. VRLEs engage students in a virtual representation of a learning space; however, understanding how to use and interact with a VRLE usually represents a steep learning curve, particularly for the instructor. Our work addresses this problem by integrating an ITS into the VRLE. In doing so, timely guidance can be provided to the students, and the instructor can adapt the ITS support to best guide student learning across multiple skill levels. Our approach immerses students in an engineering-based VRLE that resembles a space launch mission control. Inside the environment, students interact with the system to acquire data, compute, and evaluate options to attempt the successful landing of a spacecraft. The VRLEs' ITS provides the students with feedback as they interact with the environment to address challenges. The feedback is provided in various modalities, such as virtual floating panels with general suggestions, hints on which parameters from the calculations should be changed, and visual demonstrations of the calculations' outputs. To perform an initial validation of our approach, we compare its effectiveness against a physical laboratory version of the same experiment. The results of the interviews validated the ITS as a successful approach in providing students with prompt feedback while they are immersed in the VRLE. Overall, the work strengthens the validity of VRLEs as platforms that support more engaging and immersive learning experiences, and informs how to effectively integrate ITS within them.

This Work-in-Progress Research paper provides an exploration of viewpoints that are feasible candidates for visualizing the crystal structures in a Virtual Reality Learning Environment (VRLE). Visualization of crystal structures in Materials Science and Engineering is an important topic for various STEM disciplines. Traditional approaches to visualize these structures are limited to two-dimensional representations via paper sheets or computer screens. Beyond traditional methods, VRLEs are a promising approach for teaching about crystal structures, as their visualization is inherently three-dimensional and can afford a more immersive user experience. However, the optimal selection of viewpoints and perspectives that promote effective exploration and enhanced interactions with the crystal structures inside the VRLEs remains unclear. Our work addresses this gap by comparing three distinct viewing configurations in a VRLE: (A) from the bottom base of the structure, (B) hovering on top of the stacked crystal structure, and (C) from the center of the structure. To compare the viewpoints, we analyze the user preferences between the visualization approaches. Overall, this paper provides guidelines on the design of experiences related to crystalline structures in VRLEs for better student engagement and enjoyment with Materials Science content.

Virtual reality (VR) technology has emerged as a promising tool in education, offering new possibilities for learning in immersive environments. This paper explores the application of VR in biology education, specifically focusing on teaching the functionality of the cell membrane. The traditional methods of textbook reading and microscopic observation lack interactivity and engagement, motivating the development of a VR-based learning approach designed as a game-like experience. The study compares the knowledge gain of participants who played the VR game with a control group who watched instructional videos. Surprisingly, the control group exhibited a higher knowledge gain compared to the VR group. Analysis suggests that the VR participants were more distracted by the immersive nature of the VR environment and focused less on the instructional content. Future research involving larger and more motivated participant groups, such as freshmen or high school students, may yield different results. Despite the observed limitations, this study highlights the importance of balancing immersive experiences with focused instructional content for better learning outcomes.

Recent studies have shown that gestures are useful indicators of understanding, learning, and memory retention. However, and specially in collaborative settings, current metrics that estimate task understanding often neglect the information expressed through gestures. This work introduces the physical instruction assimilation (PIA) metric, a novel approach to estimate task understanding by analyzing the way in which collaborators use gestures to convey, assimilate, and execute physical instructions. PIA estimates task understanding by inspecting the number of necessary gestures required to complete a shared task. PIA is calculated based on the multiagent gestural instruction comparer (MAGIC) architecture, a previously proposed framework to represent, assess, and compare gestures. To evaluate our metric, we collected gestures from collaborators remotely completing the following three tasks: block assembly, origami, and ultrasound training. The PIA scores of these individuals are compared against two other metrics used to estimate task understanding: number of errors and amount of idle time during the task. Statistically significant correlations between PIA and these metrics are found. Additionally, a Taguchi design is used to evaluate PIA's sensitivity to changes in the MAGIC architecture. The factors evaluated the effect of changes in time, order, and motion trajectories of the collaborators' gestures. PIA is shown to be robust to these changes, having an average mean change of 0.45. These results hint that gestures, in the form of the assimilation of physical instructions, can reveal insights of task understanding and complement other commonly used metrics.

Gestures are a key aspect of communication during collaboration: through gestures we can express ideas, inquires and formalize instructions as we collaborate. Nevertheless, gesture analysis is not currently used to assess quality of task collaboration. One possible reason for this is that there is no consensus on how to represent and compare gestures from the semantic standpoint. To address this, this paper introduces three novel approaches to compare gestures performed by individuals as they collaborate to complete a physical task. Our approach relies on solving three variations of an integer optimization assignment problem, i.e. based on gesture similarity, based on temporal synchrony, and based on a combination of both. We collected the gestures of 40 participants (divided into 20 pairs) as they performed two collaborative tasks, and generated a human baseline that compared and matched their gestures. Afterwards, our gesture comparison approach was evaluated against other gestures comparison approaches based on how well they replicated the human baseline. Our approach outperformed the other approaches, agreeing with the human baseline over 85% of the times. Thus, the obtained results support the proposed technique for gesture comparison. This in turn can lead to the development of better methods to evaluate collaborative physical tasks.

This work presents an approach to estimate task understanding and performance during a remote ultrasound training task via gestures. These task understanding insights are obtained through the PIA metric, a score that represents how well are gestures being used to complete a shared task. To evaluate our hypothesis, 20 participants performed a remote ultrasound training task consisting of three subtasks: vessel detection, blood extraction, and foreign body detection. Afterwards, their task understanding and performance was estimated using our PIA metric and three other metrics: error rate, idle time rate, and task completion percentage. After performing a correlation analysis, we found significant correlations between the PIA metric and all the other metrics for task understanding estimation. In addition, the insights generated from our PIA score explained inconsistencies in the participants' scores that were not expressed using the other metrics. Finally, we used two post-experiment questionnaires to subjectively evaluate the participants' perceived understanding and performance, and found that the PIA score was significantly correlated with the participants' overall task understanding. All these results indicate that a gesture-based metric can be used to estimate task understanding, which can have a positive impact in the way remote ultrasound tasks are performed and assessed.

Telementoring generalist surgeons as they treat patients can be essential when in situ expertise is not readily available. However, adverse cyber-attacks, unreliable network conditions, and remote mentors' predisposition can significantly jeopardize the remote intervention. To provide medical practitioners with guidance when mentors are unavailable, we present the AI-Medic, the initial steps towards the development of a multimodal intelligent artificial system for autonomous medical mentoring. The system uses a tablet device to acquire the view of an operating field. This imagery is provided to an encoder-decoder neural network trained to predict medical instructions from the current view of a surgery. The network was training using DAISI, a dataset including images and instructions providing step-by-step demonstrations of surgical procedures. The predicted medical instructions are conveyed to the user via visual and auditory modalities.

Augmented Reality (AR) benefits telementoring by enhancing the communication between the mentee and the remote mentor with mentor authored graphical annotations that are directly integrated into the mentee’s view of the workspace. An important problem is conveying the workspace to the mentor effectively, such that they can provide adequate guidance. AR headsets now incorporate a frontfacing video camera, which can be used to acquire the workspace. However, simply providing to the mentor this video acquired from the mentee’s first-person view is inadequate. As the mentee moves their head, the mentor’s visualization of the workspace changes frequently, unexpectedly, and substantially. This paper presents a method for robust high-level stabilization of a mentee first-person video to provide effective workspace visualization to a remote mentor. The visualization is stable, complete, up to date, continuous, distortion free, and rendered from the mentee’s typical viewpoint, as needed to best inform the mentor of the current state of the workspace. In one study, the stabilized visualization had significant advantages over unstabilized visualization, in the context of three number matching tasks. In a second study, stabilization showed good results, in the context of surgical telementoring, specifically for cricothyroidotomy training in austere settings.

Gestures play a fundamental role in instructional processes between agents. However, effectively transferring this non-verbal information becomes complex when the agents are not physically co-located. Recently, remote collaboration systems that transfer gestural information have been developed. Nonetheless, these systems relegate gestures to an illustrative role: only a representation of the gesture is transmitted. We argue that further comparisons between the gestures can provide information of how well the tasks are being understood and performed. While gesture comparison frameworks exist, they only rely on gesture's appearance, leaving semantics and pragmatical aspects aside. This work introduces the Multi-Agent Gestural Instructions Comparer (MAGIC), an architecture that represents and compares gestures at the morphological, semantical and pragmatical levels. MAGIC abstracts gestures via a three-stage pipeline based on a taxonomy classification, a dynamic semantics framework and a constituency parsing; and utilizes a comparison scheme based on subtrees intersections to describe gesture similarity. This work shows the feasibility of the framework by assessing MAGIC's gesture matching accuracy against other gesture comparison frameworks during a mentor-mentee remote collaborative physical task scenario.

This poster presents the design, implementation, and evaluation of a method for robust high-level stabilization of mentees first-person video in augmented reality (AR) telementoring. This video is captured by the front-facing built-in camera of an AR headset and stabilized by rendering from a stationary view a planar proxy of the workspace projectively texture mapped with the video feed. The result is stable, complete, up to date, continuous, distortion free, and rendered from the mentee's default viewpoint. The stabilization method was evaluated in two user studies, in the context of number matching and for cricothyroidotomy training, respectively. Both showed a significant advantage of our method compared with unstabilized visualization.