Rehabilitation exercises are essential for individuals who have lost their ability to function normally. Typically, a functional rehabilitation program consists of two parts: supervised exercises in a rehabilitation center and additional exercises performed independently at home. Home exercises are as important as in-center exercises and can significantly improve recovery time. However, at home, the patient is unsupervised, with a significant risk of performing exercises incorrectly. This article presents a web application designed for hand rehabilitation exercises that uses a pretrained hand landmark model and an artificial neural network (ANN) classifier to ensure correct exercise performance. The application employs transfer learning to extract features from the user's hand movements, with the ANN classifier, designed using the ML5 library, determining if the positions are performed accurately. The ML5 library, a high-level interface to TensorFlow.js, makes the application suitable for web deployment. Physicians can create new exercises and retrain the ANN classifier without programming skills. An evaluation with 12 participants demonstrated an overall classification accuracy of 90.25% and achieved a system usability scale score of 75.30, indicating high usability and potential for real-world rehabilitation scenarios. This lightweight web application is highly accessible and user friendly and runs seamlessly on any smartphone or tablet. This approach shows great promise in enhancing exercise accuracy and safety, thereby improving the rehabilitation process at home and in clinical settings. In addition, it is adaptable for other models for full-body and face tracking, making it versatile for various rehabilitation applications.

This paper presents interfaces for reporting emotion in real-time during VR stimuli. Self-reported emotional responses are critical for developing emotion recognition systems. Such responses can vary throughout a stimulus such as 360° video, but most interfaces for reporting emotion are designed to be used after the experience. This reduces the entire experience to a single data point and raises concerns about validity when multiple emotions can be elicited across the stimulus. We introduce and compare user interfaces that allow for real-time emotion reporting throughout the length of the stimulus. Each interface varies on how emotion is physically input by the user and displayed back to them for confirmation. A preliminary study compared five such interfaces, gathering initial impressions, comparing control schemes, and rating intuitiveness. A primary study considered four refined interface designs and compared reporting precision and subjective opinions. Results suggest that a single interface face icon responding to arousal and valence reports and a gradiating color wheel are intuitive, precise, and unobtrusive. More broadly, results indicate the type of rating interface has a significant effect on the given ratings.

Emotion recognition models require datasets of physiological responses to stimuli designed to elicit targeted emotions, preferably stimuli similar to the experience during which the models will be used. Many libraries of such stimuli have been created to ease this data collection process, most of which involve passive media such as images or videos. Virtual Reality, however, offers an opportunity to investigate uniquely active emotion elicitation stimuli that directly center the user in the experience with an increased feeling of presence and potential to elicit stronger emotions. We leverage this to introduce a set of four active affective tasks in VR designed to quickly elicit targeted emotions without need for narrative understanding common to passive stimuli. We compare our tasks with selections from an existing affective library of passive 360° videos and validate our approach by comparing self-reported emotional responses to the stimuli. Results indicate that these types of active task stimuli can reliably elicit strong emotions comparable to other passive media and provide the basis for building a larger library of training- and education-relevant tasks.

Virtual reality (VR) can potentially enhance student engagement and memory retention in the classroom. However, distraction among participants in a VR-based classroom is a significant concern. Several factors, including mind wandering, external noise, stress, etc., can cause students to become internally and/or externally distracted while learning. To detect distractions, single or multi-modal features can be used. A single modality is found to be insufficient to detect both internal and external distractions, mainly because of individual variability. In this work, we investigated multi-modal features: eye tracking and EEG data, to classify the internal and external distractions in an educational VR environment. We set up our educational VR environment and equipped it for multi-modal data collection. We implemented different machine learning (ML) methods, including k-nearest-neighbors (kNN), Random Forest (RF), one-dimensional convolutional neural network - long short-term memory (1 D-CNN-LSTM), and two-dimensional convolutional neural networks (2D-CNN) to classify participants' internal and external distraction states using the multi-modal features. We performed cross-subject, cross-session, and gender-based grouping tests to evaluate our models. We found that the RF classifier achieves the highest accuracy over 83% in the cross-subject test, around 68% to 78% in the cross-session test, and around 90% in the gender-based grouping test compared to other models. SHAP analysis of the extracted features illustrated greater contributions from the occipital and prefrontal regions of the brain, as well as gaze angle, gaze origin, and head rotation features from the eye tracking data.

Virtual reality (VR) makes learning more interesting for students and could help them remember what they have learned better than traditional methods. However, a student could get distracted in a VR environment because of stress, wandering thoughts, unwanted noise, outside sounds, etc. Distractions could be classified as either external (due to the environment) or internal (due to internal thoughts). To identify external distractions, previous researchers have used eye-gaze data. Eye-gaze data cannot, however, detect internal distractions because a user may be looking at the educational material in VR while also thinking about something else. We explored the usage of electroencephalogram (EEG) data to detect internal distractions. We designed an educational VR environment and trained three machine learning models: Random Forest (RF), Support Vector Machine (SVM), and k-nearest-neighbors (kNN), to detect internal distractions of students. For data labeling, we considered two window lengths (20 and 30 seconds) starting at 5 seconds after the distraction task started. We did cross-subject and cross-session tests, and our results show that kNN provides a better accuracy (64%) compared to RF and SVM. We also found that the shorter window length of 20 seconds provided a slightly better accuracy then the 30 second window. Our results are not far from such random guessing. Therefore, our contribution lies more in the fostering of ideas for future work that must employ more advanced and sophisticated techniques.

Educational virtual reality (VR) applications are the most recent addition to the learning management tools in this modern age. Due to health concerns, financial concerns, and convenience, people are looking for alternate ways to teach and learn. An efficient VR-based teaching interface could enhance student engagement, learning outcomes, and overall educational experience. Typically, teachers in a VR classroom do not have a way to know what students are doing since students are not visible. An efficient teaching interface should include some mechanism for a teacher to monitor students and alert the teacher if a student is trying to catch the attention of the teacher. An ideal interface would be one, which helps a teacher effectively monitor students while teaching without increasing the cognitive load of the teacher. In this paper, we present a comparative study of two such student monitoring interfaces. In the first interface, the student activity related information is shown using icons near the student avatar (representing a student in the VR environment). While in the second interface, a set of centrally-arranged emoticon-like visual indicators are present in addition to the student avatar, and the student activity related information is shown near the student emoticon. We present a detailed user experiment comparing the two interfaces in terms of teaching management, student monitoring capability, cognitive load, and user preference. Participants preferred and performed better with Indicator-located interface over avatar-located interface.

We present a preliminary study on the design of visual interfaces for users to continuously rate their emotion while viewing VR content. Interfaces consist of two from the literature, a continuous adaptation of the popular SAM interface, and two novel interfaces. Designs were tested to discern what elements are intuitive or distracting. Study phases included initial impressions of interface visuals, tuning the interface control scheme, training by rating a list of emotion labels, and continuous rating of 360° video content. Results suggest that an interactive face icon (Smiley) is a promising design choice and suggest further evaluation of possible benefits.

We assess the effectiveness of attention guidance cues in an educational platform in local high schools with real students. Three eye-tracked visual cues, previously assessed for their ability to guide and restore attention, are compared against a baseline absence of cue in a VR field trip of a virtual solar energy field. Students experienced four presentations on solar energy production including in-world animations and teacher imagery, in three of which the visual cues guided attention to the relevant object or teacher in the scene. Attention guidance using visual cues is commonly studied using “search and selection” style tasks, but has not been studied in the context of maintaining attention in real-world environments.

We present and evaluate methods to redirect desktop inputs such as eye gaze and mouse pointing to a VR-embedded avatar. We use these methods to build a novel interface that allows a desktop user to give presentations in remote VR meetings such as conferences or classrooms. Recent work on such VR meetings suggests a substantial number of users continue to use desktop interfaces due to ergonomic or technical factors. Our approach enables desk-top and immersed users to better share virtual worlds, by allowing desktop-based users to have more engaging or present "cross-reality" avatars. The described redirection methods consider mouse pointing and drawing for a presentation, eye-tracked gaze towards audience members, hand tracking for gesturing, and associated avatar motions such as head and torso movement. A study compared different levels of desktop avatar control and headset-based control. Study results suggest that users consider the enhanced desktop avatar to be human-like and lively and draw more attention than a conventionally animated desktop avatar, implying that our interface and methods could be useful for future cross-reality remote learning tools.

Virtual Reality (VR) has been found useful to improve engagement and retention level of students, for some topics, compared to traditional learning tools such as books, and videos. However, a student could still get distracted and disengaged due to a variety of factors including stress, mind-wandering, unwanted noise, and external alerts. Student eye gaze data could be useful for detecting these distracted students. Gaze data-based visualizations have been proposed in the past to help a teacher monitor distracted students. However, it is not practical for a teacher to monitor a large number of student indicators while teaching. To help filter students based on distraction level, we propose an automated system based on machine learning to classify students based on their distraction level. The key aspects are: (1) we created a labeled eye gaze dataset from an educational VR environment, (2) we propose an automatic system to gauge a student’s distraction level from gaze data, and (3) we apply and compare several classifiers for this purpose. Each classifier classifies distraction, per educational activity section, into one of three levels (low, mid or high). Our results show that Random Forest (RF) classifier had the best accuracy (98.88%) compared to the other models we tested. Additionally, a personalized machine learning model using either RF, kNN, or Extreme Gradient Boosting (XGBoost) model was found to improve the classification accuracy significantly.

Virtual Reality (VR) headsets with embedded eye trackers are appearing as consumer devices (e.g. HTC Vive Eye, FOVE). These devices could be used in VR-based education (e.g., a virtual lab, a virtual field trip) in which a live teacher guides a group of students. The eye tracking could enable better insights into students’ activities and behavior patterns. For real-time insight, a teacher’s VR environment can display student eye gaze. These visualizations would help identify students who are confused/distracted, and the teacher could better guide them to focus on important objects. We present six gaze visualization techniques for a VR-embedded teacher’s view, and we present a user study to compare these techniques. The results suggest that a short particle trail representing eye trajectory is promising. In contrast, 3D heatmaps (an adaptation of traditional 2D heatmaps) for visualizing gaze over a short time span are problematic.

We study experiences of students attending classes remotely from home using a social VR platform, considering both desktop-based and headset-based viewing of remote lectures. Ratings varied widely. Headset viewing produced higher presence overall. Strong negative correlations between headset simulator sickness symptoms and overall experience ratings, and some other ratings, suggest that the headset experience was much better for comfortable users than for others. Reduced sickness symptoms, and no similar correlations, were found for desktop viewing. Desktop viewing appears to be a good alternative for students not comfortable with headsets. Future VR systems are expected to provide more stable and comfortable visuals, providing benefits to more users.

Redirected walking (RDW) techniques provide a way to explore a virtual space that is larger than the available physical space by imperceptibly manipulating the virtual world view or motions. These manipulations may introduce conflicts between real and virtual cues (e.g., visual-vestibular conflicts), which can be disturbing when detectable by users. The empirically established detection thresholds of rotation manipulation for RDW still require a large physical tracking space and are therefore impractical for general-purpose Virtual Reality (VR) applications. We investigate Redirected Jumping (RDJ) as a new locomotion metaphor for redirection to partially address this limitation, and because jumping is a common interaction for environments like games. We investigated the detection rates for different curvature gains during RDJ. The probability of users detecting RDJ appears substantially lower than that of RDW, meaning designers can get away with greater manipulations with RDJ than with RDW. We postulate that the substantial vertical (up/down) movement present when jumping introduces increased vestibular noise compared to normal walking, thereby supporting greater rotational manipulations. Our study suggests that the potential combination of metaphors (e.g., walking and jumping) could further reduce the required physical space for locomotion in VR. We also summarize some differences in user jumping approaches and provide motion sickness measures in our study.

We present an architecture to make a VR pedagogical agent responsive to shifts in user attention monitored by eye tracking. The behavior-based AI includes low-level sensor elements, sensor combiners that compute attention metrics for higher-level sensors called generalized hotspots, an annotation system for arranging scene elements and responses, and its response selection system. We show that the techniques can control the playback of teacher avatar clips that point out and explain objects in a VR oil rig for training.

We present a pilot study of four teacher avatars for an educational virtual field trip. The avatars consist of a depth-video-based mesh of a real person, a game-style human model, a robot model, and the robot with its head replaced by a video feed of the teacher's face. Multiple avatars were developed to consider alternatives to the mesh representation that required high-bandwidth networks and a non-immersive teacher interface. The pilot study presents a random avatar to the participant at each of 4 educational stations, and follows up with a subjective questionnaire. Most notably, we find positive affinity for the plain robot model to be similar to that of the video mesh, which was previously shown to provide high co-presence and good results for education. Results are guiding a larger study that will measure the educational efficacy of revised avatars.

In educational virtual reality, it is important to deal with problems of student inattention to presented content. We are developing attention-restoring visual cues for display when gaze tracking detects that student focus shifts away from critical objects. These cues include novel aspects and variations of standard cues that performed well in prior work on visual guidance. For the longer term, we propose experiments to compare various cues and their parameters to assess effectiveness and tradeoffs, and to assess the impact of eye tracking. Eye tracking is used to both detect inattention and to control the appearance and location of cues.

We present a VR field trip framework, Kvasir-VR, and assess its two approaches to teacher-guided content. In one approach, networked student groups are guided by a live teacher captured as live-streamed depth camera imagery. The second approach is a standalone (non-networked) version allowing students to individually experience the field trip based on depth camera recordings of the same teacher. Both approaches were tested at two high schools using a VR environment that teaches students about solar energy production via tours of a solar plant. We show that our live networked approach can produce promising test score gains and very high ratings of co-presence, affective attraction, overall opinion, etc. Results show a benefit of live networked VR, as the standalone approach had lower performance in terms of gains and most ratings, although its ratings were still positive. We further consider possible differences of school environment (dedicated vs. integrated classroom), and we conclude with tradeoffs and implications to benefit future design of educational VR.

This demo presents an approach to networked educational virtual reality for virtual field trips and guided exploration. It shows an asymmetric collaborative interface in which a remote teacher stands in front of a large display and depth camera (Kinect) while students are immersed with HMDs. The teacher's front-facing mesh is streamed into the environment to assist students and deliver instruction. Our project uses commodity virtual reality hardware and high-performance networks to allow students who are unable to visit a real facility with an alternative that provides similar educational benefits. Virtual facilities can further be augmented with educational content through interactables or small games. We discuss motivation, features, interface challenges, and ongoing testing.

We overview the Virtual Energy Center, a VR environment that models a real energy facility to enable virtual field trips and self-guided exploration. Our goal is to take advantage of emerging low-cost hardware and improved networks to provide students who cannot travel to the real facility with alternatives that provide comparable educational benefit. The virtual facility is augmented by visual guides and educational content to teach students about concentrating solar power technology. A teacher physically near the student can appear in the scene via depth camera imagery, allowing the teacher to walk around in a classroom setting and assist students. Additionally, work-in-progress is streaming the depth images over a network to allow students to virtually meet expert guides from the real facility. We summarize these features, some interaction-related challenges, and ongoing testing.

We present design and impact studies of visual feedback for virtual grasping. The studies suggest new or updated guidelines for feedback. Recent grasping techniques incorporate visual cues to help resolve undesirable visual or performance artifacts encountered after real fingers enter a virtual object. Prior guidelines about such visuals are based largely on other interaction types and provide inconsistent and potentially-misleading information when applied to grasping. We address this with a two-stage study. In the first stage, users adjusted parameters of various feedback types, including some novel aspects, to identify promising settings and to give insight into preferences regarding the parameters. In the next stage, the tuned feedback techniques were evaluated in terms of objective performance (finger penetration, release time, and precision) and subjective rankings (visual quality, perceived behavior impact, and overall preference). Additionally, subjects commented on the techniques while reviewing them in a final session. Performance wise, the most promising techniques directly reveal penetrating hand configuration in some way. Subjectively, subjects appreciated visual cues about interpenetration or grasp force, and color changes are most promising. The results enable selection of the best cues based on understanding the relevant tradeoffs and reasonable parameter values. The results also provide a needed basis for more focused studies of specific visual cues and for choosing conditions in comparisons to other feedback modes, such as haptic, audio, or multimodal. Considering results, we propose that 3D interaction guidelines must be updated to capture the importance of interpenetration cues, possible performance benefits of direct representations, and tradeoffs involved in cue selection.

We tuned and evaluated visual feedback techniques for virtual grasps. To date, development of such feedback has been largely ad-hoc, with minimal work that can guide technique selection. We considered several techniques including both standard and novel aspects. In terms of impact on real hand behavior, the best techniques all directly reveal penetrating hand configuration in some way. Subjectively, color changes are most liked.

Recent virtual grasping approaches involve physical simulation and virtual couplings between tracked and virtual hand configurations. We introduce a nonuniform coupling in which the stiffness of thumb coupling is scaled relative to that of other digits. This shifts the position of grasped objects in the hand, which may impact grasp and release performance. We graphically illustrate the effects on grasped object position, and we experimentally measure impact on object motion during grasp release. In addition to basic nonuniform scaling, we propose adaptive scaling to account for the number and depth of digits involved in multi-finger grasps. We show that one particular choice of adaptive coupling results in a tradeoff of increased object position consistency and decreased release consistency. The knowledge gained from our study will enable researchers to optimize couplings in future work.

We present a new 3D lens rendering technique and a new spatiotemporal lens. Interactive 3D lenses, often called volumetric lenses, provide users with alternative views of data sets within 3D lens boundaries while maintaining the surrounding overview (context). In contrast to previous multipass rendering work, we discuss the strengths, limitations, and performance costs of a single-pass technique especially suited to fragment-level lens effects, such as color mapping, lighting, and clipping. Some object-level effects, such as a data set selection lens, are also incorporated, with each object's geometry being processed once by the graphics pipeline. For a substantial range of effects, our approach supports several composable lenses at interactive frame rates without performance loss during increasing lens intersections or manipulation by a user. Other cases, for which this performance cannot be achieved, are also discussed. We illustrate possible applications of our lens system, including Time Warp lenses for exploring time-varying data sets.

This paper extends 3D lens techniques. Interactive 3D lenses, often called volumetric lenses, provide users with alternative views of datasets within spatially bounded regions of interest (focus) while maintaining the surrounding overview (context). In contrast to previous multi-pass rendering work, we discuss the strengths, limitations, and performance cost of a single-pass technique. For a substantial range of effects, it supports several interactive composable lenses at interactive frame rates without performance loss during increasing lens intersections or manipulations. Other cases, for which this performance cannot be achieved, are also discussed. Finally, we illustrate possible applications of our lens system, especially new Time Warp lenses for exploring time-varying datasets in interactive VR.

We present and evaluate a new approach for real-time rendering of composable 3D lenses for polygonal scenes. Such lenses, usually called "volumetric lenses," are an extension of 2D Magic Lenses to 3D volumes in which effects are applied to scene elements. Although the composition of 2D lenses is well known, 3D composition was long considered infeasible due to both geometric and semantic complexity. Nonetheless, for a scene with multiple interactive 3D lenses, the problem of intersecting lenses must be considered. Intersecting 3D lenses in meaningful ways supports new interfaces such as hierarchical 3D windows, 3D lenses for managing and composing visualization options, or interactive shader development by direct manipulation of lenses providing component effects. Our 3D volumetric lens approach differs from other approaches and is one of the first to address efficient composition of multiple lenses. It is well-suited to head-tracked VR environments because it requires no view-dependent generation of major data structures, allowing caching and reuse of full or partial results. A Composite Shader Factory module composes shader programs for rendering composite visual styles and geometry of intersection regions. Geometry is handled by Boolean combinations of region tests in fragment shaders, which allows both convex and nonconvex CSG volumes for lens shape. Efficiency is further addressed by a Region Analyzer module and by broad-phase culling. Finally, we consider the handling of order effects for composed 3D lenses.

We demonstrate composable volumetric lenses as interpretational tools for geological visualization. The lenses provide a constrained focus region that provides alternative views of datasets to the user while maintaining the context of surrounding features. Our rendering method is based on run-time composition of GPU shader programs that implement per-fragment clipping to lens boundaries and surface shader evaluation. It supports composition of lenses and the user can influence the resulting visualization by interactively changing the order in which the individual lens effects are applied. Multiple shader effects have been created and used for interpretation of high-resolution elevation datasets (like LTDAR and SRTM) in our lab.

The goal of this project is to develop a training system that can simulate the welding process in real-time and give feedback that avoids learning wrong motion patterns for beginning welders and can be used to analyze the process by the teacher afterwards. The system is based mainly on COTS components. A standard PC with a Dual-core CPU and a medium-end nVidia graphics card is sufficient. Input is done with a regular welding gun to allow realistic training. The gun is tracked by an OptiTrack system with 3 FLEX:V100 cameras. The same is also used to track a regular welding helmet to get accurate eye positions for display, which was chosen over glasses for robustness. The display itself is a Zalman Trimon stereo monitor that is laid out horizontally. The software is designed around a main simulation component for solving heat conduction on a grid of simulation points based on local GaussSeidel elimination.

Various virtual and augmented reality systems include volumetric lenses, an extension of 2D magic lenses to 3D volumes in which effects are applied to scene elements. We present a new 3D volumetric lens rendering system that differs fundamentally from other approaches and that is the first to address efficient real-time composition of multiple 3D lenses. A lens factory module composes chainable shader programs for rendering composite visual styles and geometry of intersection regions. Geometry is handled by Boolean combinations of region tests in fragment shaders, which allows both convex and non-convex CSG volumes for lens shape. Efficiency is further addressed by a region analyzer module and by broad-phase culling. Finally, we consider the handling of order effects for composed 3D lenses.

We discuss methods to enable haptic visualization on vibrotactile arrays. Our work is motivated by the potential for a tactile array to provide an additional useful channel for information such as location cues related to dataset features or remote user behaviors. We present a framework for array rendering and several specific techniques. Novel aspects of our work include the example application of a palm-sized tactile array to visualize dataset features or remote user state in a VR system, a generalized haptic glyph mechanism for 2D tactile arrays, and the extension of graphical visualization techniques to haptics (glyphs, fisheye distortion, spatial anti-aliasing, gamma correction).

In this paper, we present our networked virtual tennis game that has been developed as a hybrid framework with head-mounted display and fishtank virtual reality systems. The paper reports the findings of a hybrid collaboration task which compared the two systems based on their egocentric and exocentric features. The focus of the study was on the strengths and weaknesses in each system for the given particular task: How do users perform in each system? And how might each system complement the others for teamwork? We report on localization error and correct hit percentage results that were obtained during trials with the two types of systems. These results suggest that head-mounted displays with egocentric features allow more accurate spatial localization and the fishtank displays with exocentric features provide better cues for time synchronization events.

We present a physically-based approach to grasping and manipulation of virtual objects that produces visually realistic results, addresses the problem of visual interpenetration of hand and object models, and performs force rendering for force-feedback gloves in a single framework. Our approach couples tracked hand configuration to a simulation-controlled articulated hand model using a system of linear and torsional spring-dampers. We discuss an implementation of our approach that uses a widely-available simulation tool for collision detection and response. We illustrate the resulting behavior of the virtual hand model and of grasped objects, and we show that the simulation rate is sufficient for control of current force-feedback glove designs. We also present a prototype of a system we are developing to support natural whole-hand interactions in a desktop-sized workspace.

AERCam is a free-flying camera being developed by NASA to assist in Space Shuttle or Space Station operations. We have developed a system for controlling AERCam remotely from the ground when significant communication delays are involved and have simulated its use. Our telerobotic control system includes a predictive display that is based on a simulation of Shuttle and AERCam dynamics. For a number of physical and numeric reasons, the behavior of the remote AERCam system may drift from what is expected by the predictive simulation on the ground. To correct for this, we have developed a technique for periodically reestablishing correspondence of the ground system with AERCam based on data about AERCam's actual behavior. Our system also provides a basis for development of more advanced autonomous AERCam ground control systems in the future.