Physics governs everyday interaction, yet in Virtual Reality (VR) the fidelity of such interactions can diverge from reality. We investigate how Physical Fidelity (virtual object behavior) and Action Fidelity (virtual hand behavior) of physics-driven interaction shape user experience. In a within-subject study (n = 34), participants performed gamified tasks under three conditions: No-Physics (lower Physical and Action Fidelity), Object-Physics (higher Physical, lower Action Fidelity), and Full-Physics (higher Physical and Action Fidelity). Results show that higher Physical Fidelity reduces task efficiency and increases overall workload, with the No-Physics condition outperforming the others in these metrics. When combined with higher Action Fidelity, although efficiency gets even worse in some cases, the Full-Physics condition enhances body ownership and interaction quality. The hybrid Object-Physics condition consistently ranks lowest across all qualitative measures. Interpreting these results through the Interaction Fidelity Model, we offer design implications for VR applications.

Additive models of interaction performance, such as the Keystroke-Level Model (KLM), are tools that allow designers to compare and optimize the performance of user interfaces by summing the predicted times for the atomic components of a specific interaction to predict the total time it would take to complete that interaction. There has been extensive work in creating such additive models for 2D interfaces, but this approach has rarely been explored for 3D user interfaces. We propose a KLM-style additive model, based on existing atomic task models in the literature, to predict task completion time for 3D interaction tasks. We performed two studies to evaluate the feasibility of this approach across multiple input modalities, with one study using a simple menu selection task and the other a more complex manipulation task. We found that several of the models from the literature predicted actual task performance with less than 20% error in both the menu selection and manipulation study. Overall, we found that additive models can predict both absolute and relative performance of input modalities with reasonable accuracy.

Prior research has developed a number of manipulation techniques that can achieve precise object placement in virtual reality, but studies of these techniques typically use simple objects. We conducted a study comparing two existing techniques, (AMP-IT and WISDOM), during alignment of objects with complex geometry to evaluate the potential influence of geometric complexity on performance, usability, workload and preference. Our findings indicate that participants had faster completion times and higher trial completion rates with AMP-IT on high-precision alignment tasks, contrary to earlier findings that used simple objects. Yet WISDOM is still preferred and considered more usable, despite increased workload and poorer performance, exposing participants' willingness to trade objective performance for comfort during use.

Sensemaking is a complex task that places a heavy cognitive demand on individuals. With the recent surge in data availability, making sense of vast amounts of information has become a significant challenge for many professionals, such as intelligence analysts. Immersive technologies such as mixed reality offer a potential solution by providing virtually unlimited space to organize data. However, the difficulty of processing, filtering relevant information, and synthesizing insights remains. We proposed using eye-tracking data from mixed reality head-worn displays to derive the analyst’s perceived interest in documents and words, and convey that part of the mental model to the analyst. The global interest of the documents is reflected in their color, and their order on the list, while the local interest of the documents is used to generate focused recommendations for a document. To evaluate these recommendation cues, we conducted a user study with two conditions: a gaze-aware system, EyeST, and a “Freestyle” system without gaze-based visual cues. Our findings reveal that the EyeST helped analysts stay on track by reading more essential information while avoiding distractions. However, this came at the cost of reduced focused attention and perceived system performance. The results of our study highlight the need for explainable AI in human-AI collaborative sensemaking to build user trust and encourage the integration of AI outputs into the immersive sensemaking process. Based on our findings, we offer a set of guidelines for designing gaze-driven recommendation cues in an immersive environment.

Collaborative virtual environments allow workers to contribute to team projects across space and time. While much research has closely examined the problem of working in different spaces at the same time, few have investigated the best practices for collaborating in those spaces at different times aside from textual and auditory annotations. We designed a system that allows experts to record a tour inside a virtual inspection space, preserving knowledge and providing later observers with insights through a 3D playback of the expert’s inspection. We also created several interactions to ensure that observers are tracking the tour and remaining engaged. We conducted a user study to evaluate the influence of these interactions on an observing user’s information recall and user experience. Findings indicate that independent viewpoint control during a tour enhances the user experience compared to fully passive playback and that additional interactivity can improve auditory and spatial recall of key information conveyed during the tour.

Virtual displays provided through head-worn displays (HWDs) offer users large screen space for productivity, but managing this space effectively presents challenges. Existing research shows that while increased screen space improves performance, it can also introduce significant window management overhead. This paper explores how to enhance window-switching strategies for virtual displays by leveraging eye tracking provided by HWDs and underutilized spaces around the main display area. We investigate the efficiency and usability of different cursor behaviors and selection modes in a Spatial Bar interface for window-switching tasks in augmented reality environments. Our study involved two primary selection modes: Gaze and Cursor, each tested with two cursor behaviors: Teleport and Stay. We measured objective performance metrics, including task completion time and error rates, and subjective evaluations using the NASA TLX and custom questionnaires. Results show that Cursor Stay, while familiar and comfortable for participants, was laborious and led to longer task completion times, particularly over large distances. Gaze Teleport led to the quickest window switching times, particularly in tasks where the original cursor position or the target window were far from the Spatial Bar.

Locating small features in a large, dense object in virtual reality (VR) poses a significant interaction challenge. While existing multiscale techniques support transitions between various levels of scale, they are not focused on handling dense, homogeneous objects with hidden features. We propose a novel approach that applies the concept of progressive refinement to VR navigation, enabling focused inspections. We conducted a user study where we varied two independent variables in our design, navigation style (STRUCTURED vs. UNSTRUCTURED) and display mode (SELECTION vs. EVERYTHING), to better understand their effects on efficiency and awareness during multiscale navigation. Our results showed that unstructured navigation can be faster than structured and that displaying only the selection can be faster than displaying the entire object. However, using an everything display mode can support better location awareness and object understanding.

Virtual environments (VEs) empower geographically distributed teams to collaborate on a shared project regardless of time. Existing research has separately investigated collaborations within these VEs at the same time (i.e., synchronous) or different times (i.e., asynchronous). In this work, we highlight the often-overlooked concept of bichronous collaboration and define it as the seamless integration of archived information during a real-time collaborative session. We revisit the time-space matrix of computer-supported cooperative work (CSCW) and reclassify the time dimension as a continuum. We describe a system that empowers collaboration across the temporal states of the time continuum within a VE during remote work. We conducted a user study using the system to discover how the bichronous temporal state impacts the user experience during a collaborative inspection. Findings indicate that the bichronous temporal state is beneficial to collaborative activities for information processing, but has drawbacks such as changed interaction and positioning behaviors in the VE.

Fidelity describes how closely a replication resembles the original. It can be helpful to analyze how faithful interactions in virtual reality (VR) are to a reference interaction. In prior research, fidelity has been restricted to the simulation of reality—also called realism. Our definition includes other reference interactions, such as superpowers or fiction. Interaction fidelity is a multilayered concept. Unfortunately, different aspects of fidelity have either not been distinguished in scientific discourse or referred to with inconsistent terminology. Therefore, we present the Interaction Fidelity Model (IntFi Model). Based on the human-computer interaction loop, it systematically covers all stages of VR interactions. The conceptual model establishes a clear structure and precise definitions of eight distinct components. As a communication tool, it helps teams to understand and discuss fidelity in VR. It was reviewed through workshops with fourteen VR experts. We provide guidelines, diverse examples, and educational material to apply the IntFi Model universally to any VR experience and propose foundational research opportunities.

Existing research on sensemaking in immersive analytics systems primarily focuses on understanding how users complete analysis within these systems with quantitative and qualitative datasets. However, these user studies mainly concentrate on understanding analysis styles and methodologies from a predominantly novice user study population. While this approach provides excellent initial insights into what users may do within IA systems, it fails to address how professionals may utilize an immersive analytic system for analysis tasks. In our work, we build upon an existing immersive analytics concept - "Immersive Space to Think" to understand how professional user populations differ from novice users in immersive analytic system usage. We conducted a user study with 11 professional intelligence analysts who completed three analysis sessions each. Using our results from this study, we provide deep analysis into how professional users complete sensemaking within immersive analytic systems, compare our findings to previously published findings with a novice user population, and provide insights into how to develop better IA systems to support the professional analyst's strategies within these systems.

Performance models of interaction, such as Fitts' law, are important tools for predicting and explaining human motor performance and for designing high-performance user interfaces. Extensive prior work has proposed such models for the 3D interaction task of distal pointing, in which the user points their hand or a device at a distant target in order to select it. However, there is no consensus on how to compute the index of difficulty for distal pointing tasks. We present a preliminary study suggesting that existing models may not be sufficient to model distal pointing performance with current virtual reality technologies. Based on these results, we hypothesized that both the form of the model and the standard method for collecting empirical data for pointing tasks might need to change in order to achieve a more accurate and valid distal pointing model. In our main study, we used a new methodology to collect distal pointing data and evaluated traditional models, purely ballistic models, and two-part models. Ultimately, we found that the best model used a simple Fitts'-law-style index of difficulty with angular measures of amplitude and width.

Future augmented reality (AR) glasses empower users to view personal applications and services anytime and anywhere without being restricted by physical locations and the availability of physical screens. In typical everyday activities, people move around to carry out different tasks and need a variety of information on the go. Existing interfaces in AR do not support these use cases well, especially when the number of applications increases. We explore the usability of three world-referenced approaches that move AR applications with users as they transition among different locations, featuring different levels of AR app availability: (1) always using a menu to manually open an app when needed; (2) automatically suggesting a relevant subset of all apps; and (3) carrying all apps with the users to the new location. Through a controlled study and a relatively more ecologically-valid study in AR, we reached better understandings on the performance trade-offs and observed the impact of various everyday contextual factors on these interfaces in more realistic AR settings. Our results shed light on how to better support the mobile information needs of users in everyday life in future AR interfaces.

Although access to sufficient screen space is crucial to knowledge work, workers often find themselves with limited access to display infrastructure in remote or public settings. While virtual displays can be used to extend the available screen space through extended reality (XR) head-worn displays (HWD), we must better understand the implications of working with them in public settings from both users' and bystanders' viewpoints. To this end, we conducted two user studies. We first explored the usage of a hybrid AR display across real-world settings and tasks. We focused on how users take advantage of virtual displays and what social and environmental factors impact their usage of the system. A second study investigated the differences between working with a laptop, an AR system, or a VR system in public. We focused on a single location and participants performed a predefined task to enable direct comparisons between the conditions while also gathering data from bystanders. The combined results suggest a positive acceptance of XR technology in public settings and show that virtual displays can be used to accompany existing devices. We highlighted some environmental and social factors. We saw that previous XR experience and personality can influence how people perceive the use of XR in public. In addition, we confirmed that using XR in public still makes users stand out and that bystanders are curious about the devices, yet have no clear understanding of how they can be used.

Virtual displays enabled through head-worn augmented reality have unique characteristics that can yield extensive amounts of screen space. Existing research has shown that increasing the space on a computer screen can enhance usability. Since virtual displays offer the unique ability to present content without rigid physical space constraints, they provide various new design possibilities. Therefore, we must understand the trade-offs of layout choices when structuring that space. We propose a single Canvas approach that eliminates boundaries from traditional multi-monitor approaches and instead places windows in one large, unified space. Our user study compared this approach against a multi-monitor setup, and we considered both purely virtual systems and hybrid systems that included a physical monitor. We looked into usability factors such as performance, accuracy, and overall window management. Results show that Canvas displays can cause users to compact window layouts more than multiple monitors with snapping behavior, even though such optimizations may not lead to longer window management times. We did not find conclusive evidence of either setup providing a better user experience. Multi-Monitor displays offer quick window management with snapping and a structured layout through subdivisions. However, Canvas displays allow for more control in placement and size, lowering the amount of space used and, thus, head rotation. Multi-Monitor benefits were more prominent in the hybrid configuration, while the Canvas display was more beneficial in the purely virtual configuration.

Virtual travel in Virtual Reality experiences is common, offering users the ability to explore expansive virtual spaces. Various interfaces exist for virtual travel, with speed playing a crucial role in user experience and spatial awareness. Teleportation-based interfaces provide instantaneous transitions, whereas continuous and semi-continuous methods vary in speed and control. Prior research by Bowman et al. highlighted the impact of travel speed on spatial awareness demonstrating that instantaneous travel can lead to user disorientation. However, additional cues, such as visual target selection, can aid in reorientation. This study replicates and extends Bowman’s experiment, investigating the influence of travel speed and visual target cues on spatial orientation.

Gaze-adaptive interfaces can enable intuitive hands-free augmented reality (AR) interaction but unintentional selection (i.e. “Midas Touch”) can have serious consequences during high-stakes real-world AR use. In the present study, we assessed how simulated gaze-adaptive AR interfaces, implementing single and dual gaze inputs, influence Soldiers’ human performance and user experience (UX) in a fast-paced virtual reality marksmanship task. In Experiment 1, we investigated 1- and 2-stage dwell-based interfaces, finding confirmatory dual gaze dwell input effectively reduced Midas Touch but also reduced task performance and UX compared to an always-on (AO) interface. In Experiment 2, we investigated gaze depth-based interfaces, finding similar negative impacts of confirmatory dwell on Midas Touch, task performance, and UX. Overall, compared to the AO interface, single gaze input interfaces (e.g. single dwell or gaze depth threshold) reduced viewing of task-irrelevant information and yielded similar task performance and UX despite being prone to Midas Touch. Broadly, our findings demonstrate that AR users performing fast-paced dynamic tasks can tolerate some unintentional activation of AR displays if reliable and rapid information access is maintained and point to the need to develop and refine gaze depth estimation algorithms and novel gaze depth-based interfaces that provide rapid access to AR display content.

Handheld devices have become an inclusive alternative to head-mounted displays in virtual reality (VR) environments, enhancing accessibility and allowing cross-device collaboration. Object manipulation techniques in 3D space with handheld devices, such as those in handheld augmented reality (AR), have been typically evaluated in table-top scale and we currently lack an understanding of how these techniques perform in larger scale environments. We conducted two studies, each with 30 participants, to investigate how different techniques impact usability and performance for room-scale handheld VR object translations. We compared three translation techniques that are similar to commonly studied techniques in handheld AR: 3DSlide, VirtualGrasp, and Joystick. We also examined the effects of target size, target distance, and user mobility conditions (stationary vs. moving). Results indicated that the Joystick technique, which allowed translation in relation to the user's perspective, was the fastest and most preferred, without difference in precision. Our findings provide insights for designing room-scale handheld VR systems, with potential implications for mixed reality systems involving handheld devices.

As immersive analytics research becomes more popular, user studies have been aimed at evaluating the strategies and layouts of users’ sensemaking during a single focused analysis task. However, approaches to sensemaking strategies and layouts are likely to change as users become more familiar/proficient with the immersive analytics tool. In our work, we build upon an existing immersive analytics approach-Immersive Space to Think-to understand how schemas and strategies for sensemaking change across multiple analysis tasks. We conducted a user study with 14 participants who completed three different sensemaking tasks during three separate sessions. We found significant differences in the use of space and strategies for sensemaking across these sessions and correlations between participants’ strategies and the quality of their sensemaking. Using these findings, we propose guidelines for effective analysis approaches within immersive analytics systems for document-based sensemaking.

Analysts need to process large amounts of data in order to extract concepts, themes, and plans of action based upon their findings. Different display technologies offer varying levels of space and interaction methods that change the way users can process data using them. In a comparative study, we investigated how the use of single traditional monitor, a large, high-resolution two-dimensional monitor, and immersive three-dimensional space using the Immersive Space to Think approach impact the sensemaking process. We found that user satisfaction grows and frustration decreases as available space increases. We observed specific strategies users employ in the various conditions to assist with the processing of datasets. We also found an increased usage of spatial memory as space increased, which increases performance in artifact position recall tasks. In future systems supporting sensemaking, we recommend using display technologies that provide users with large amounts of space to organize information and analysis artifacts.

Precise 3D manipulation in virtual reality (VR) is essential for effectively aligning virtual objects. However, state-of-the-art VR manipulation techniques have limitations when high levels of precision are required, including the unnaturalness caused by scaled rotations and the increase in time due to degree-of-freedom (DoF) separation in complex tasks. We designed two novel techniques to address these issues: AMP-IT, which offers direct manipulation with an adaptive scaled mapping for implicit DoF separation, and WISDOM, which offers a combination of Simple Virtual Hand and scaled indirect manipulation with explicit DoF separation. We compared these two techniques against baseline and state-of-the-art manipulation techniques in a controlled experiment. Results indicate that WISDOM and AMP-IT have significant advantages over best-practice techniques regarding task performance, usability, and user preference.

Eye gaze patterns vary based on reading purpose and complexity, and can provide insights into a reader’s perception of the content. We hypothesize that during a complex sensemaking task with many text-based documents, we will be able to use eye-tracking data to predict the importance of documents and words, which could be the basis for intelligent suggestions made by the system to an analyst. We introduce a novel eye-gaze metric called ‘GazeScore’ that predicts an analyst’s perception of the relevance of each document and word when they perform a sensemaking task. We conducted a user study to assess the effectiveness of this metric and found strong evidence that documents and words with high GazeScores are perceived as more relevant, while those with low GazeScores were considered less relevant. We explore potential real-time applications of this metric to facilitate immersive sensemaking tasks by offering relevant suggestions.

This paper provided empirical knowledge of the user experience for using collaborative visualization in a distributed asymmetrical setting through controlled user studies. With the ability to access various computing devices, such as Virtual Reality (VR) head-mounted displays, scenarios emerge when collaborators have to or prefer to use different computing environments in different places. However, we still lack an understanding of using VR in an asymmetric setting for collaborative visualization. To get an initial understanding and better inform the designs for asymmetric systems, we first conducted a formative study with 12 pairs of participants. All participants collaborated in asymmetric (PC-VR) and symmetric settings (PC-PC and VR-VR). We then improved our asymmetric design based on the key findings and observations from the first study. Another ten pairs of participants collaborated with enhanced PC-VR and PC-PC conditions in a follow-up study. We found that a well-designed asymmetric collaboration system could be as effective as a symmetric system. Surprisingly, participants using PC perceived less mental demand and effort in the asymmetric setting (PC-VR) compared to the symmetric setting (PC-PC). We provided fine-grained discussions about the trade-offs between different collaboration settings.

Existing research on immersive analytics to support the sensemaking process focuses on single-session sensemaking tasks. However, in the wild, sensemaking can take days or months to complete. In order to understand the full benefits of immersive analytic systems, we need to understand how immersive analytic systems provide flexibility for the dynamic nature of the sensemaking process. In our work, we build upon an existing immersive analytic system - Immersive Space to Think, to evaluate how immersive analytic systems can support sensemaking tasks over time. We conducted a user study with eight participants with three separate analysis sessions each. We found significant differences between analysis strategies between sessions one, two, and three, which suggest that immersive space to think can benefit analysts during multiple stages in the sensemaking process.

Collaborative virtual environments afford new capabilities in telepresence applications, allowing participants to co-inhabit an environment to interact while being embodied via avatars. However, shared content within these environments often takes away the attention of collaborators from observing the non-verbal cues conveyed by their peers, resulting in less effective communication. Exaggerated gestures, abstract visuals, as well as a combination of the two, have the potential to improve the effectiveness of communication within these environments in comparison to familiar, natural non-verbal visualizations. We designed and conducted a user study where we evaluated the impact of these different non-verbal visualizations on users' identification time, understanding, and perception. We found that exaggerated gestures generally perform better than non-exaggerated gestures, abstract visuals are an effective means to convey intentional reactions, and the combination of gestures with abstract visuals provides some benefits compared to their standalone counterparts.

For touch typists, using a physical keyboard ensures optimal text entry task performance in immersive virtual environments. However, successful typing depends on the user’s ability to accurately position their hands on the keyboard after performing other, non-keyboard tasks. Finding the correct hand position depends on sensory feedback, including visual information. We designed and conducted a user study where we investigated the impact of visual representations of the keyboard and users’ hands on the time required to place hands on the homing bars of a keyboard after performing other tasks. We found that this keyboard homing time decreased as the fidelity of visual representations of the keyboard and hands increased, with a video pass-through condition providing the best performance. We discuss additional impacts of visual representations of a user’s hands and the keyboard on typing performance and user experience in virtual reality.

Immersive spaces have great potential to support analysts in complex sensemaking tasks, but the use of only manual interactions for organizing data elements can become tedious. We analyzed the user interactions to support cluster formation in an immersive sensemaking system, and we designed a semi-automated cluster creation technique that determines the user’s intent to create a cluster based on object proximity. We present the results of a user study comparing this proximity-based technique with a manual clustering technique and a baseline immersive workspace with no explicit clustering support. We found that semi-automated clustering was faster and preferred, while manual clustering gave greater control to users. These results provide support for the approach of adding intelligent semantic interactions to aid the users of immersive analytics systems.

Immersive and interactive storytelling in virtual reality (VR) is an emerging creative practice that has been thriving in recent years. Educational applications using immersive VR storytelling to explain complex science concepts have very promising pedagogical benefits because on the one hand, storytelling breaks down the complexity of science concepts by bridging them to people's everyday experiences and familiar cognitive models, and on the other hand, the learning process is further reinforced through rich interactivity afforded by the VR experiences. However, it is unclear how different amounts of storytelling in an interactive VR storytelling experience may affect learning outcomes due to a paucity of literature on educational VR storytelling research. This preliminary study aims to add to the literature through an exploration of variations in the designs of essential storytelling elements in educational VR storytelling experiences and their impact on the learning of complex immunology concepts.

Glanceable Augmented Reality interfaces have the potential to provide fast and efficient information access for the user. However, where to place the virtual content and how to access them depend on the user context. We designed a Context-Aware AR interface that can intelligently adapt for two different contexts: solo and social. We evaluated information access using Context-Aware AR compared to current mobile phones and non-adaptive Glanceable AR interfaces. We found that in a solo scenario, compared to a mobile phone, the Context-Aware AR interface was preferred, easier, and significantly faster; it improved the user experience; and it allowed the user to better focus on their primary task. In the social scenario, we discovered that the mobile phone was slower, more intrusive, and perceived as the most difficult. Meanwhile, Context-Aware AR was faster for responding to information needs triggered by the conversation; it was preferred and perceived as the easiest for resuming conversation after information access; and it improved the user’s awareness of the other person’s facial expressions.

Welcome to the November 2022 issue of the IEEE Transactions on Visualization and Computer Graphics (TVCG). This issue contains selected papers accepted at the IEEE International Symposium on Mixed and Augmented Reality (ISMAR). The conference took place in Singapore from October 17-22,2022 in hybrid mode.

Referencing objects of interest is a common requirement in many collaborative tasks. Nonetheless, accurate object referencing at a distance can be challenging due to the reduced visibility of the objects or the collaborator and limited communication medium. Augmented Reality (AR) may help address the issues by providing virtual pointing rays to the target of common interest. However, such pointing ray techniques can face critical limitations in large outdoor spaces, especially when the environment model is unavailable. In this work, we evaluated two pointing ray techniques for distant object referencing in model-free AR from the literature: the Double Ray technique enhancing visual matching between rays and targets, and the Parallel Bars technique providing artificial orientation cues. Our experiment in outdoor AR involving participants as pointers and observers partially replicated results from a previous study that only evaluated observers in simulated AR. We found that while the effectiveness of the Double Ray technique is reduced with the additional workload for the pointer and human pointing errors, it is still beneficial for distant object referencing.

Welcome to the 11th IEEE Transactions on Visualization and Computer Graphics (TVCG)special issue on IEEE Virtual Reality and 3D User Interfaces. This volume contains a total of 29 full papers selected for and presented at the IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2022), held fully virtual in Christchurch, New Zealand from March 12 to 16, 2022.

As virtual reality (VR) technology becomes cheaper, higher-quality, and more widely available, it is seeing increasing use in a variety of applications including cultural heritage, real estate, and architecture. A common goal for all these applications is a compelling virtual recreation of a real place. Despite this, there has been very little research into how users perceive and experience such replicated spaces. This article reports the results from a series of three user studies investigating this topic. Results include that the scale of the room and large objects in it are most important for users to perceive the room as real and that non-physical behaviors such as objects floating in air are readily noticeable and have a negative effect even when the errors are small in scale.

The process of sensemaking involves foraging through and extracting information from large sets of documents, and it can be a cognitively intensive task. A recent approach, the Immersive Space to Think (IST), allows analysts to browse, read, mark up documents, and use immersive 3D space to organize and label collections of documents. In this study, we observed seventeen novice analysts perform a historical analysis task in order to understand how users utilize the features of IST to extract meaning from large text-based datasets. We found three different layout strategies they employed to create meaning with the documents we provided. We further found patterns of interaction and organization that can inform future improvements to the IST approach.

In the near future, augmented reality (AR) glasses are envisioned to become the next-generation personal computing platform. They could be always on and worn all day, delivering continuous and pervasive AR experiences for general-purpose everyday use cases. However, it remains unclear how we could enable unobtrusive and easy information access without distracting users, while being acceptable to use at the same time. To address this question, we implemented two prototypes based on the Glanceable AR paradigm, a promising way of managing and acquiring information through glancing at the periphery of AR head-worn displays (HWDs). We conducted two separate studies to evaluate our designs. In the first study, we obtained feedback from a large sample of participants of varied age and background about a video prototype that showcased some envisioned scenarios of using Glanceable AR for everyday tasks. In the second study, we asked participants to use a working prototype during authentic real-world activities for three days. We found that users appreciated the Glanceable AR approach. They found it less distracting or intrusive than existing devices in authentic everyday use cases, and would like to use the interface on a daily basis if the form factor of the AR headset was more like eyeglasses.

Physical monitors require space, lack flexibility, and can become expensive and less portable in large setups. Virtual monitors, on the other hand, can minimize those problems, but may be subject to technological limitations such as lower resolution and field of view. We investigate the impacts of using virtual monitors displayed on a current state-of-the-art augmented reality headset for conducting productivity work. We conducted a user study that compared physical monitors, virtual monitors, and a hybrid combination of both in terms of performance, accuracy, comfort, focus, preference, and confidence. Results show that virtual monitors are a feasible approach for performing serious productivity work, albeit currently constrained by technical limitations that lead to inferior usability and performance compared to physical monitors. We also discovered that, with current technology, the hybrid condition was a better tradeoff between the familiarity and trustworthiness of physical monitors and the extra space provided by virtual monitors. We conclude by expressing the opportunity for designing strategies for mixing virtual and physical monitors into novel hybrid interfaces.

Welcome to the 10thIEEE Transactions on Visualization and Computer Graphics (TVCG)special issue on IEEE Virtual Reality and 3D User Interfaces. This volume contains a total of 25 full papers selected for and presented at the IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2021), held fully virtual from March 27 to April 3, 2021.Founded in 1993, IEEE VR has a long tradition as the premier venue where new research results in the field of Virtual Reality (VR) are presented. With the emergence of VR as a major technology in a diverse set of fields, such as entertainment, education, data analytics, artificial intelligence, medicine, construction, training, and many others, the papers presented at IEEE VR and published in theIEEE TVCGVR special issue mark a major highlight of the year.

Welcome to the November 2021 issue of theIEEE Transactions on Visualization and Computer Graphics (TVCG). This issue contains selected papers accepted at the IEEE International Symposium on Mixed and Augmented Reality (ISMAR). The conference took place in Bari, Italy from October 4–8, 2021 in virtual mode due to the COVID-19 pandemic.

Augmented reality head-worn displays (AR HWDs) have the potential to assist personal computing and the acquisition of everyday information. In this research, we propose Glanceable AR, an interaction paradigm for accessing information in AR HWDs. In Glanceable AR, secondary information resides at the periphery of vision to stay unobtrusive and can be accessed by a quick glance whenever needed. We propose two novel hands-free interfaces: "head-glance", in which virtual contents are fixed to the user’s body and can be accessed by head rotation, and "gaze-summon" in which contents can be "summoned" into central vision by eye-tracked gazing at the periphery. We compared these techniques with a baseline heads-up display (HUD), which we call "eye-glance" interface in two dual-task scenarios. We found that the head-glance and eye-glance interfaces are more preferred and more efficient than the gaze-summon interface for discretionary information access. For a continuous monitoring task, the eye-glance interface was preferred. We discuss the implications of our findings for designing Glanceable AR interfaces in AR HWDs.

Welcome to the 9thIEEE Transactions on Visualization and Computer Graphics (TVCG) special issue on IEEE Virtual Reality and 3D User Interfaces. This volume contains a total of 29 full papers selected for and presented at the IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2020) held in Atlanta, United States on March 22–26, 2020. Founded in 1993, IEEE VR has a long tradition as the premier venue where new research results in the field of Virtual Reality (VR) are presented. With the emergence of VR as a major technology in a diverse set of fields, such as entertainment, education, data analytics, artificial intelligence, medicine, construction, training, and many others, the papers presented at IEEE VR and published in theIEEE TVCG VRspecial issue mark a major highlight of the year.

We propose and evaluate novel pseudo-haptic techniques to display mass and mass distribution for proxy-based object manipulation in virtual reality. These techniques are specifically designed to generate haptic effects during the object's rotation. They rely on manipulating the mapping between visual cues of motion and kinesthetic cues of force to generate a sense of heaviness, which alters the perception of the object's mass-related properties without changing the physical proxy. First we present a technique to display an object's mass by scaling its rotational motion relative to its mass. A psycho-physical experiment demonstrates that this technique effectively generates correct perceptions of relative mass between two virtual objects. We then present two pseudo-haptic techniques designed to display an object's mass distribution. One of them relies on manipulating the pivot point of rotation, while the other adjusts rotational motion based on the real-time dynamics of the moving object. An empirical study shows that both techniques can influence perception of mass distribution, with the second technique being significantly more effective.

Specifying points in three-dimensional (3D) space is an essential function in many augmented reality (AR) applications. When an environment model is not available, a straightforward solution is to perform geometric triangulation using two rays. However, naïve implementations suffer from low accuracy caused by technical limitations of AR devices and human motor constraints. To overcome these issues, we designed and evaluated two enhanced geometric techniques for 3D point marking. VectorCloud uses multiple rays to reduce the effects of pointing jitter, and ImageRefinement improves the accuracy by allowing users to refine the 3D direction of the two rays. We conducted studies to understand the characteristics of these techniques in both ecologically valid outdoor settings using a mobile AR display and in more controlled setting using virtual reality simulation. Our experiments demonstrate that both techniques improve the precision of 3D point marking, and that ImageRefinement is superior to VectorCloud overall. These results are particularly relevant in the design of mobile AR systems intended for use in large outdoor areas.

Mobile augmented reality may eventually replace our smartphones as the primary way of accessing information on the go. However, current interfaces provide little support to walking and to the variety of actions we perform in the real world. To achieve its full potential, augmented reality interfaces must support the fluid way we move and interact in the physical world. We explored how different adaptation strategies can contribute towards this goal. We evaluated design alternatives through contextual studies and identified the key interaction patterns that interfaces for walking should support. We also identified desirable properties of adaptation-based interface techniques, which can be used to guide the design of the next-generation walking-centered augmented reality workspaces.

This paper presents our solution to the 2019 3DUI Contest challenge, which focuses on passive haptics. We aimed to provide a compelling user experience in virtual reality while overcoming the limitations of physical space and current tracking devices. To meet these goals, we designed a time travel scenario that incorporates several novel features. A time machine increases efficiency in the use of physical space. A passive haptic camera prop provides a help system that is integrated into the storyline. Finally, the concept of a “temporal stabilizer” provides a plausible way to reuse a single tracking device to track multiple passive haptic props.

Specifying points in three-dimensional space is essential in AR applications. Geometric triangulation is a straightforward way to specify points, but its naïve implementation has low precision. We designed two enhanced geometric techniques for 3D point marking: VectorCloud, which uses multiple rays to reduce jittering, and ImageRefinement, which allows 3D ray refinement to improve precision. Our experiments, conducted in both simulated and real AR, demonstrate that both techniques improve the precision of 3D point marking, and that ImageRefinement is superior to VectorCloud overall. These results are particularly relevant in the design of mobile AR systems for large outdoor areas.

Welcome to the November 2019 issue of the IEEE Transactions on Visualization and Computer Graphics (TVCG). This issue contains selected papers accepted at the IEEE International Symposium on Mixed and Augmented Reality (ISMAR), held this year in Beijing, China from October 14 to October 18, 2019.

Museums have been long recognized as legitimate places for out-of-school learning, offering unique experiences for large student groups. However, the challenges of assessing learning and engagement outcomes using technological interventions for large audiences, often undermine the value of these practices. To address the need for understanding how learning unfolds within the context of a museum visit as an interplay between the people, the interactives, and the mediators, we conducted a design evaluation in a science museum. Our focus was understanding the impact of audience interaction on learning and engagement during collaborative games for large student groups. For the purposes of this case study we also designed a physical board game having the dual purpose of a learning activity and an assessment tool. We present the results of our observations and discourse analysis, and discuss them in the light of our previous study in public schools.

Augmented reality (AR) head-up displays (HUD) have previously been explored as a potential information delivery system for drivers. In driving scenarios, correct perception of virtual object distance assists with effective use of AR HUDs in safety-critical applications (e.g., collision warnings). AR volumetric displays purportedly offer increased accuracy of distance perception through consistent presentation of oculomotor cues such as vergence and accommodation over traditional displays. For this paper, we investigated volumetric AR displays as a mean of enhancing perception of virtual objects registered to the real world, specifically in terms of distance perception as a result of binocular cues. We designed and ran an experiment where participants controlled and placed virtual objects next to real-world counterparts at ranges between 7-12 meters. We found that the volumetric AR display outperformed a traditional fixed focal plane AR HUD when considering distances 5 meters greater than the traditional display's focal depth.

Specifying 3D positions in the real world is an essential step to create augmented reality content. However, this task can be challenging when information about the depth or geometry of the target location is not available. In this paper, we evaluate alternative techniques for 3D pointing at a distance without knowledge about the environment. We present the results of two studies evaluating the accuracy of techniques based on geometry and human depth perception. We find that geometric methods provide higher accuracy but may suffer from low precision due to pointing errors. We propose a solution that increases precision by combining multiple samples to obtain a better estimate of the target position.

In this paper, we present multiple 3D interaction techniques to address the three tasks in the 2018 3DUI contest: ladder climbing, drone flying and tower stacking. The proposed techniques provide smart interfaces to aid efficiency and error prevention. We discuss the principles and rationale used in the design.

We present Force Push, a hyper-natural gesture-to-action mapping for object manipulation in virtual reality (VR). It maps hand gestures derived from human-human interaction to physics-driven movement of an object and uses expressive features of gestures to enhance controllability. An initial user study shows both the performance and broader user experience qualities of Force Push as compared to a traditional direct control mapping.

Redirected walking techniques have the potential to provide natural locomotion while users experience large virtual environments. However, when using redirected walking in small physical workspaces, disruptive overt resets are often required. We describe the design of an educational virtual reality experience in which users physically walk through virtual tunnels representative of the World War I battle of Vauquois. Walking in only a 15- by 5-foot tracked space, users are redirected through subtle, narrative-driven resets to walk through a tunnel nearly 50 feet in length. This work contributes approaches and lessons that can be used to provide a seamless and natural virtual reality walking experience in highly constrained physical spaces.

Many types of virtual reality (VR) systems allow users to use natural, physical head movements to view a 3D environment. In some situations, such as when using systems that lack a fully surrounding display or when opting for convenient low-effort interaction, view control can be enabled through a combination of physical and virtual turns to view the environment, but the reduced realism could potentially interfere with the ability to maintain spatial orientation. One solution to this problem is to amplify head rotations such that smaller physical turns are mapped to larger virtual turns, allowing trainees to view the entire surrounding environment with small head movements. This solution is attractive because it allows semi-natural physical view control rather than requiring complete physical rotations or a fully-surrounding display. However, the effects of amplified head rotations on spatial orientation and many practical tasks are not well understood. In this paper, we present an experiment that evaluates the influence of amplified head rotation on 3D search, spatial orientation, and cybersickness. In the study, we varied the amount of amplification and also varied the type of display used (head-mounted display or surround-screen CAVE) for the VR search task. By evaluating participants first with amplification and then without, we were also able to study training transfer effects. The findings demonstrate the feasibility of using amplified head rotation to view 360 degrees of virtual space, but noticeable problems were identified when using high amplification with a head-mounted display. In addition, participants were able to more easily maintain a sense of spatial orientation when using the CAVE version of the application, which suggests that visibility of the user's body and awareness of the CAVE's physical environment may have contributed to the ability to use the amplification technique while keeping track of orientation.

One of the most popular audiences of museums is classroom-size groups of students, in the context of school field trips. However, students do not get adequate involvement during interactive group experiences, which might affect their impression and learning gained from the visit. In this paper, we present our findings from a recent study in middle schools, where 507 students were engaged with their class in a learning game about olive oil production. We had two players directly control the game and varied the level of involvement of the audience (the rest of the class), using iPads. We found that higher involvement in the game afforded greater retention of information after two days, while there was no difference after one day. Also, students with direct agency in the game revealed greater learning gains than the audience members. Results about the impact of socioeconomic status and social interactions on learning are reported, along with the most important design implications.

In this work, we present a depth-based solution to multi-level menus for selection and manipulation of virtual objects using freehand gestures. Navigation between and through menus is performed using three gesture states that utilize X, Y translations of the finger with boundary crossing. Although presented in a single context, this menu structure can be applied to a myriad of domains requiring several levels of menu data, and serves to supplement existing and emerging menu design for augmented, virtual, and mixed-reality applications.

We report the findings of a study designed to evaluate the effect of stereopsis and field of regard (FOR) in two different mixed reality (MR) simulation platforms: a head-mounted display (HMD) and a CAVE. We compared the performance of participants on two levels of stereopsis (mono and stereo) and two levels of FOR (90 degrees and 270 degrees) using a variety of scientific visualization tasks. Among the findings, we observed that not all the effects were consistent between the platforms. Stereo alone or in combination with higher FOR improved completion time on both platforms. However, adding stereo solely reduced the accuracy of the participants on the CAVE and improved on the HMD. Our findings extend prior knowledge on the contribution of visual fidelity components and suggests potential limits on MR simulation between platforms.

Museums are rich and complex learning experiences, using a variety of interactive approaches to engage their audiences. However, the largely unstructured nature of free-choice learning calls for alternative approaches that can effectively engage groups of school age students with diverse cultural backgrounds. In this paper, we present our findings from a recent study in a museum in Greece, where triads of students had to learn about olive oil production using a game enabling different levels of interactivity and collaboration. We found that facilitation by an expert guide led to greater learning gains as compared to students playing alone, with one or three simultaneous game controllers. We also compared these results with a previous controlled experiment conducted in the US with middle school students, using the same game but without the ecologically valid facilitation. Drawing ideas from sociocultural and cognitive theories we interpret the contradictory findings, identifying the impact of culture on their (social) interactions, their subjective game experience, and eventually learning, in these spaces.

One of the goals of much virtual reality (VR) research is to increase realism. In particular, many techniques for locomotion in VR attempt to approximate real-world walking. However, it is not yet fully understood how the design of more realistic locomotion techniques affects user task performance. We performed an experiment to compare a semi-natural locomotion technique (based on the Virtusphere device) with a traditional, non-natural technique (based on a game controller) and a fully natural technique (real walking). We found that the Virtusphere technique was significantly slower and less accurate than both of the other techniques. Based on this result and others in the literature, we speculate that locomotion techniques with moderate interaction fidelity will often have performance inferior to both high-fidelity techniques and well-designed low-fidelity techniques. We argue that our experimental results are an effect of interaction fidelity, and perform an analysis of the fidelity of the three locomotion techniques to support this argument.

Virtual reality training systems are commonly used in a variety of domains, and it is important to understand how the realism of a training simulation influences training effectiveness. We conducted a controlled experiment to test the effects of display and scenario properties on training effectiveness for a visual scanning task in a simulated urban environment. The experiment varied the levels of field of view and visual complexity during a training phase and then evaluated scanning performance with the simulator's highest levels of fidelity and scene complexity. To assess scanning performance, we measured target detection and adherence to a prescribed strategy. The results show that both field of view and visual complexity significantly affected target detection during training; higher field of view led to better performance and higher visual complexity worsened performance. Additionally, adherence to the prescribed visual scanning strategy during assessment was best when the level of visual complexity during training matched that of the assessment conditions, providing evidence that similar visual complexity was important for learning the technique. The results also demonstrate that task performance during training was not always a sufficient measure of mastery of an instructed technique. That is, if learning a prescribed strategy or skill is the goal of a training exercise, performance in a simulation may not be an appropriate indicator of effectiveness outside of training-evaluation in a more realistic setting may be necessary.

In informal learning spaces employing digital content, such as museums, visitors often do not get adequate exposure to content, or they passively receive instruction offered by a museum docent to the whole group. This research aims to identify which elements of co-located group collaboration, virtual environments, and serious games can be leveraged for an enhanced museum learning and entertaining experience. We developed C-OLiVE, an interactive virtual environment supporting tripartite group collaboration, which we used to explore our hypothesis that synchronous, co-located, group collaboration will afford greater learning compared to conventional approaches. In an empirical study, we found some evidence supporting this hypothesis, taking into consideration other factors such as game experience and social presence. Students participating in the three-player condition demonstrated a better understanding of the collaborative tasks compared to their single-player counterparts. We discuss these results and outline future studies using the same virtual environment.

In informal learning spaces employing digital content, such as museums, visitors either do not get adequate exposure to content or get information through passive instruction offered by a museum docent to the whole group. This research aims to identify which elements of co-located group collaboration, virtual environments, and serious games can be leveraged for an enhanced learning experience. Our hypothesis is that synchronous, co-located, group collaboration will afford greater learning compared to conventional approaches. We developed C-OLiVE, an interactive virtual learning environment supporting tripartite group collaboration, which we will use as a testbed to respond to our research questions. In this paper, we discuss our proposed research, which involves exploring some benefits of the involved technologies and proposing a list of design guidelines for anyone interested to exploit them in developing virtual environments for informal learning spaces.

In this special issue of IEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present the long papers from the IEEE Virtual Reality Conference 2014 (IEEE VR 2014), held March 29–April 2, 2014 in Minneapolis, Minnesota, USA.

Volume visualization is an important technique for analyzing datasets from a variety of different scientific domains. Volume data analysis is inherently difficult because volumes are three-dimensional, dense, and unfamiliar, requiring scientists to precisely control the viewpoint and to make precise spatial judgments. Researchers have proposed that more immersive (higher fidelity) VR systems might improve task performance with volume datasets, and significant results tied to different components of display fidelity have been reported. However, more information is needed to generalize these results to different task types, domains, and rendering styles. We visualized isosurfaces extracted from synchrotron microscopic computed tomography (SR-μCT) scans of beetles, in a CAVE-like display. We ran a controlled experiment evaluating the effects of three components of system fidelity (field of regard, stereoscopy, and head tracking) on a variety of abstract task categories that are applicable to various scientific domains, and also compared our results with those from our prior experiment using 3D texture-based rendering. We report many significant findings. For example, for search and spatial judgment tasks with isosurface visualization, a stereoscopic display provides better performance, but for tasks with 3D texture-based rendering, displays with higher field of regard were more effective, independent of the levels of the other display components. We also found that systems with high field of regard and head tracking improve performance in spatial judgment tasks. Our results extend existing knowledge and produce new guidelines for designing VR systems to improve the effectiveness of volume data analysis.

Virtual reality (VR) has been successfully applied to a broad range of training domains; however, to date there is little research investigating its benefits for sport psychology training. We hypothesized that using high-fidelity VR systems to display realistic 3D sport environments could trigger anxiety, allowing resilience-training systems to prepare athletes for real-world, highpressure situations. In this work we investigated the feasibility and usefulness of using VR for sport psychology training. We developed a virtual soccer goalkeeping application for the Virginia Tech Visionarium VisCube (a CAVE-like display system), in which users defend against simulated penalty kicks using their own bodies. Using the application, we ran a controlled, within-subjects experiment with three independent variables: known anxiety triggers, field of regard, and simulation fidelity. The results demonstrate that a VR sport-oriented system can induce increased anxiety (physiological and subjective measures) compared to a baseline condition. There were a number of main effects and interaction effects for all three independent variables in terms of the subjective measures of anxiety. Both known anxiety triggers and simulation fidelity had a direct relationship to anxiety, while field of regard had an inverse relationship. Overall, the results demonstrate great potential for VR sport psychology training systems; however, further research is needed to determine if training in a VR environment can lead to long-term reduction in sport-induced anxiety.

In our research agenda to study the effects of immersion (level of fidelity) on various tasks in virtual reality (VR) systems, we have found that the most generalizable findings come not from direct comparisons of different technologies, but from controlled simulations of those technologies. We call this the mixed reality (MR) simulation approach. However, the validity of MR simulation, especially when different simulator platforms are used, can be questioned. In this paper, we report the results of an experiment examining the effects of field of regard (FOR) and head tracking on the analysis of volume visualized micro-CT datasets, and compare them with those from a previous study. The original study used a CAVE-like display as the MR simulator platform, while the present study used a high-end head-mounted display (HMD). Out of the 24 combinations of system characteristics and tasks tested on the two platforms, we found that the results produced by the two different MR simulators were similar in 20 cases. However, only one of the significant effects found in the original experiment for quantitative tasks was reproduced in the present study. Our observations provide evidence both for and against the validity of MR simulation, and give insight into the differences caused by different MR simulator platforms. The present experiment also examined new conditions not present in the original study, and produced new significant results, which confirm and extend previous existing knowledge on the effects of FOR and head tracking. We provide design guidelines for choosing display systems that can improve the effectiveness of volume visualization applications.

In this paper, we investigate the validity of Mixed Reality (MR) Simulation by conducting an experiment studying the effects of the visual realism of the simulated environment on various search tasks in Augmented Reality (AR). MR Simulation is a practical approach to conducting controlled and repeatable user experiments in MR, including AR. This approach uses a high-fidelity Virtual Reality (VR) display system to simulate a wide range of equal or lower fidelity displays from the MR continuum, for the express purpose of conducting user experiments. For the experiment, we created three virtual models of a real-world location, each with a different perceived level of visual realism. We designed and executed an AR experiment using the real-world location and repeated the experiment within VR using the three virtual models we created. The experiment looked into how fast users could search for both physical and virtual information that was present in the scene. Our experiment demonstrates the usefulness of MR Simulation and provides early evidence for the validity of MR Simulation with respect to AR search tasks performed in immersive VR.

Spatial judgments are important for many real-world tasks in engineering and scientific visualization. While existing research provides evidence that higher levels of display and interaction fidelity in virtual reality systems offer advantages for spatial understanding, few investigations have focused on small-scale spatial judgments or employed experimental tasks similar to those used in real-world applications. After an earlier study that considered a broad analysis of various spatial understanding tasks, we present the results of a follow-up study focusing on small-scale spatial judgments. In this research, we independently controlled field of regard, stereoscopy, and head-tracked rendering to study their effects on the performance of a task involving precise spatial inspections of complex 3D structures. Measuring time and errors, we asked participants to distinguish between structural gaps and intersections between components of 3D models designed to be similar to real underground cave systems. The overall results suggest that the addition of the higher fidelity system features support performance improvements in making small-scale spatial judgments. Through analyses of the effects of individual system components, the experiment shows that participants made significantly fewer errors with either an increased field of regard or with the addition of head-tracked rendering. The results also indicate that participants performed significantly faster when the system provided the combination of stereo and head-tracked rendering.

Visualizations often use spatial representations to aid understanding, but it is unclear what properties of a spatial information presentation are most important to effectively support cognitive processing. This research explores how spatial layout and view control impact learning and investigates the role of persistent visibility when working with large displays. We performed a controlled experiment with a learning activity involving memory and comprehension of a visually represented story. We compared performance between a slideshow-type presentation on a single monitor and a spatially distributed presentation among multiple monitors. We also varied the method of view control (automatic vs. interactive). Additionally, to separate effects due to location or persistent visibility with a spatially distributed layout, we controlled whether all story images could always be seen or if only one image could be viewed at a time. With the distributed layouts, participants maintained better memory of the associated locations where information was presented. However, learning scores were significantly better for the slideshow presentation than for the distributed layout when only one image could be viewed at a time.

3D interaction has made a big splash in the gaming community with the Nintendo Wii, Sony Move and Microsoft Kinect; however, most of the interaction is based on natural imprecise motions. By using the wealth of 3D interaction research, we can use these low cost commercial systems to bring not only richer interaction to gaming but also commodity trackers to Virtual Reality research. One of the first steps in understanding how these devices can be used is to determine their limitations and strengths compared to traditional 3D tracking systems.

In this paper we present a user study evaluating the benefits of geometrically correct user-perspective rendering using an Augmented Reality (AR) magic lens. In simulation we compared a user-perspective magic lens against the common device-perspective magic lens on both phone-sized and tablet-sized displays. Our results indicate that a tablet-sized display allows for significantly faster performance of a selection task and that a user-perspective lens has benefits over a device-perspective lens for a selection task. Based on these promising results, we created a proof-of-concept prototype, engineered with current off-the-shelf devices and software. To our knowledge, this is the first geometrically correct user-perspective magic lens.

Mixed Reality (MR) simulation, in which a Virtual Reality (VR) system is used to simulate both the real and virtual components of an Augmented Reality (AR) system, has been proposed as a method for evaluating AR systems with greater levels of experimental control. However, factors such as the latency of the MR simulator may impact the validity of experimental results obtained with MR simulation. We present a study evaluating the effects of simulator latency on the equivalence of results from an MR simulator and a real AR system. We designed an AR experiment which required the participants to visually follow a virtual pipe around a small room filled with real targets and to find and identify the targets which were intersected by the pipe. We show that, with a 95% confidence interval, the results from all three simulated AR conditions fall well within one standard deviation of the real AR case.

Studying what design features are necessary and effective for educational virtual environments (VEs), we focused on two design issues: level of environmental detail and method of navigation. In a controlled experiment, participants studied animal facts distributed among different locations in an immersive VE. Participants viewed the information as either an automated tour through the environment or with full navigational control. The experiment also compared two levels of environmental detail: a sparse environment with only the animal fact cards and a detailed version that also included landmark items and ground textures. The experiment tested memory and understanding of the animal information. Though neither environmental detail nor navigation type significantly affected learning outcomes, the results suggest that manual navigation may have negatively affected the learning activity. Also, learning scores were correlated with both spatial ability and video game usage, suggesting that educational VEs may not be an appropriate presentation method for some learners.

Inaccurate perception of distances is a known problem within virtual environments. We hypothesize that the inclusion of virtual characters within these environments can improve an observer's ability to judge distances and achieve accurate spatial understanding. We have conducted an empirical study using a desktop display of a small scale environment to evaluate the validity of this concept. We investigated whether the presence and quantity of virtual human characters, as well as the naturalness of their locomotion animations, could improve egocentric and exocentric distance estimations. Preliminary results suggest that static or properly animated characters could improve exocentric estimations, and properly animated characters could reduce egocentric distance compression errors.

Volume visualization has been widely used for decades for analyzing datasets ranging from 3D medical images to seismic data to paleontological data. Many have proposed using immersive virtual reality (VR) systems to view volume visualizations, and there is anecdotal evidence of the benefits of VR for this purpose. However, there has been very little empirical research exploring the effects of higher levels of immersion for volume visualization, and it is not known how various components of immersion influence the effectiveness of visualization in VR. We conducted a controlled experiment in which we studied the independent and combined effects of three components of immersion (head tracking, field of regard, and stereoscopic rendering) on the effectiveness of visualization tasks with two x-ray microscopic computed tomography datasets. We report significant benefits of analyzing volume data in an environment involving those components of immersion. We find that the benefits do not necessarily require all three components simultaneously, and that the components have variable influence on different task categories. The results of our study improve our understanding of the effects of immersion on perceived and actual task performance, and provide guidance on the choice of display systems to designers seeking to maximize the effectiveness of volume visualization applications.

In recent years, consumers have witnessed a technological revolution that has delivered more-realistic experiences in their own homes through high-definition, stereoscopic televisions and natural, gesture-based video game consoles. Although these experiences are more realistic, offering higher levels of fidelity, it is not clear how the increased display and interaction aspects of fidelity impact the user experience. Since immersive virtual reality (VR) allows us to achieve very high levels of fidelity, we designed and conducted a study that used a six-sided CAVE to evaluate display fidelity and interaction fidelity independently, at extremely high and low levels, for a VR first-person shooter (FPS) game. Our goal was to gain a better understanding of the effects of fidelity on the user in a complex, performance-intensive context. The results of our study indicate that both display and interaction fidelity significantly affect strategy and performance, as well as subjective judgments of presence, engagement, and usability. In particular, performance results were strongly in favor of two conditions: low-display, low-interaction fidelity (representative of traditional FPS games) and high-display, high-interaction fidelity (similar to the real world).

In this paper, we explore the opportunity of bringing unistroke text entry to freehand gesture interfaces. Using existing text entry methods directly in such interfaces is impractical because of the differences between freehand gestures and traditional forms of input. To address this problem, we consider the design constraints of text entry methods using freehand gestures, and present AirStroke, a new technique based on a reengineering of the well-known unistroke technique Graffiti. Using Graffiti's alphabet, AirStroke takes advantage of the richer input capabilities of two-handed freehand gestures by providing combined mode selection and character entry with one hand, as well as word completion with the other hand. A longitudinal study suggests that AirStroke has competitive speed and accuracy to unistroke methods based on stylus input.

It is extremely challenging to run controlled studies comparing multiple Augmented Reality (AR) systems. We use an AR simulation approach, in which a Virtual Reality (VR) system is used to simulate multiple AR systems. To investigate the validity of this approach, in our first experiment we carefully replicated a well-known study by Ellis et al. using our simulator, obtaining comparable results. We include a discussion on general issues we encountered with replicating a prior study. In our second experiment further exploring the validity of AR simulation, we investigated the effects of simulator latency on the results from experiments conducted in an AR simulator. We found simulator latency to have a significant effect on 3D tracing, however there was no interaction between simulator latency and artificial latency. Based on the results from these two experiments, we conclude that simulator latency is not inconsequential in determining task performance. Simulating visual registration is not sufficient to simulate the overall perception of registration errors in an AR system. We also need to keep simulator latency at a minimum. We discuss the impact of these results on the use of the AR simulation approach.

It is extremely challenging to run controlled studies comparing multiple augmented reality (AR) systems. We use an ldquoAR simulationrdquo approach, in which a virtual reality (VR) system is used to simulate multiple AR systems. In order to validate this approach, we carefully replicated a well-known study by Ellis et al. using our simulator, obtaining comparable results.

We propose the use of virtual environments to simulate augmented reality (AR) systems for the purposes of experimentation and usability evaluation. This method allows complete control in the AR environment, providing many advantages over testing with true AR systems. We also discuss some of the limitations to the simulation approach. We have demonstrated the use of such a simulation in a proof of concept experiment controlling the levels of registration error in the AR scenario. In this experiment, we used the simulation method to investigate the effects of registration error on task performance for a generic task involving precise motor control for AR object manipulation. Isolating jitter and latency errors, we provide empirical evidence of the relationship between accurate registration and task performance.

We examine the effect of varying levels of immersion on the performance of a target following task in augmented reality (AR) X-ray vision. We do this using virtual reality (VR) based simulation. We analyze participant performance while varying the field of view of the AR display, as well as the reliability of the head tracking sensor as our components of immersion. In low reliability conditions, we simulate sensor dropouts by disabling the augmented view of the scene for brief time periods. Our study gives insight into the effect of tracking sensor reliability, as well as the relationship between sensor reliability and field of view on user performance in a target following task in a simulated AR system.

3D User Interface (3DUI) design and development requires practitioners (designers and developers) to represent their ideas in representations designed for machine execution rather than natural representations, hampering development of effective 3DUIs. As such, Concept-Oriented Design (COD) was created as a theory of software development for both natural and executable design and development. Instantiated in the toolkit Chasm, Chasm is a natural, tiered, executable User Interface Description Language (UIDL) for 3DUIs resulting in improved understandability, as well as reduced complexity and reuse. Chasm's utility is shown through evaluations by domain experts, case studies of long-term use, and an analysis of spaces .

In our experience, novel ideas for 3D interaction techniques greatly outpace developers' ability to implement them, despite the potential benefit of these ideas. We believe this is due to the inherent implementation complexity of 3D interfaces, without sufficient support from methods and tools. Believing a developer-centric representation could overcome this problem, we investigated developer practices, artifacts and language. This resulted in the theory of concept-oriented design and Chasm, a prototype realization of the theory. The key feature of concept-oriented design is its use of developer-centric representations to create a multi-tiered implementation, ranging from an envisioned behavior expressed in conversational language to low-level code. Evaluation of Chasm by domain experts and its use in multiple case studies has demonstrated that concept-oriented design in Chasm enabled developers to represent an exponential growth in 3D interface complexity with only a linear growth in implementation complexity. Positive comments by developers further support the developer-centric representation.

Immersive virtual environments (VEs) have the potential to offer rich three-dimensional interaction to users. In many instances, however, 3D interaction tasks are difficult due to both the imprecision of tracking devices and the inability of users to achieve and maintain precise hand positions in 3D space. One way to improve upon existing interaction techniques is to dynamically change the sensitivity of the interaction technique based on user input. Previous research has applied this principle to virtual hand-based manipulation techniques; when the user slows down the movement of her physical hand, the virtual hand slows down even more to allow precise manipulation. In this study we extend the prior research by applying the velocity-based scaling principle to HOMER, an existing at-a-distance manipulation technique based on ray-casting. The scaled HOMER technique offers the user the freedom to accomplish both long- and short-distance manipulation tasks with higher levels of precision without compromising speed. We present results from a user study that shows that the addition of scaling to HOMER significantly improves user performance on 3D manipulation tasks.

Many interesting and promising prototypes for visualizing video data have been proposed, including those that combine videos with their spatial context (contextualized videos). However, relatively little work has investigated the fundamental design factors behind these prototypes in order to provide general design guidance. Focusing on real-time video data visualization, we evaluated two important design factors--video placement method and spatial context presentation method--through a user study. In addition, we evaluated the effect of spatial knowledge of the environment. Participants' performance was measured through path reconstruction tasks, where the participants followed a target through simulated surveillance videos and marked the target paths on the environment model. We found that embedding videos inside the model enabled realtime strategies and led to faster performance. With the help of contextualized videos, participants not familiar with the real environment achieved similar task performance to participants that worked in that environment. We discuss design implications and provide general design recommendations for traffic and security surveillance system interfaces.

In navigating large information spaces, previous work indicates potential advantages of physical navigation (moving eyes, head, body) over virtual navigation (zooming, panning, flying). However, there is also indication of users preferring or settling into the less efficient virtual navigation. We present a study that examines these issues in the context of large, high resolution displays. The study identifies specific relationships between display size, amount of physical and virtual navigation, and user task performance. Increased physical navigation on larger displays correlates with reduced virtual navigation and improved user performance. Analyzing the differences between this study and previous results helps to identify design factors that afford and promote the use of physical navigation in the user interface.

Although 3D interaction techniques (3DITs) such as the Go-Go technique for object manipulation can be conceptually very simple, implementing these techniques can be a difficult task. Hidden complexities are often revealed at the low-level implementation stage. VR toolkits, which are commonly used to implement 3DITs, solve the problem of allowing applications to run in any hardware environment, but rarely provide support for the technique development process or technique reuse. Because VR toolkits are not interoperable, 3DIT developers cannot share their working techniques with others. In this paper, we describe IFFI, a toolkit that has been designed specifically to support 3DIT development and to allow for reuse of techniques in different VR toolkits. We are using IFFI to move towards the goal of implementing a library of reusable 3DITs to help increase their usage, increase consistency, and provide a foundation for future technique development.

A common reason for using immersive virtual environments (IVEs) in visualization is the hypothesis that IVEs should provide a higher level of spatial understanding for complex 3D structures, such as those found in underground cave systems. Therefore, we aimed to explore the use of IVEs for visualization of underground caves, and to determine the benefits of immersion for viewing such models. We ran an experiment in which domain experts answered questions with two different levels of immersion. The results show that for certain tasks the more immersive system significantly improved accuracy, speed, and comprehension over the non-immersive environment, and that 3D visualization overall is a good match for the underground cave data.

Abstract This paper describes our recent experimental evaluation of Information‐Rich Virtual Environment (IRVE) interfaces. To explore the depth cue/visibility tradeoff between annotation schemes, we design and evaluate two information layout techniques to support search and comparison tasks. The techniques provide different depth and association cues between objects and their labels: labels were displayed either in the virtual world relative to their referent (Object Space) or on an image plane workspace (Viewport Space). The Software Field of View (SFOV) was controlled to 60° or 100° of vertical angle and two groups were tested: those running on a single monitor and those on a tiled nine‐panel display. Users were timed, tracked for correctness, and gave ratings for both difficulty and satisfaction on each task. Significant advantages were found for the Viewport interface, and for high SFOV. The interactions between these variables suggest special design considerations to effectively support search and comparison performance across monitor configurations and projection distortions. Copyright © 2006 John Wiley & Sons, Ltd.

Multiple spatially-related videos are increasingly used in security, communication, and other applications. Since it can be difficult to understand the spatial relationships between multiple videos in complex environments (e.g. to predict a person's path through a building), some visualization techniques, such as video texture projection, have been used to aid spatial understanding. In this paper, we identify and begin to characterize an overall class of visualization techniques that combine video with 3D spatial context. This set of techniques, which we call contextualized videos, forms a design palette which must be well understood so that designers can select and use appropriate techniques that address the requirements of particular spatial video tasks. In this paper, we first identify user tasks in video surveillance that are likely to benefit from contextualized videos and discuss the video, model, and navigation related dimensions of the contextualized video design space. We then describe our contextualized video testbed which allows us to explore this design space and compose various video visualizations for evaluation. Finally, we describe the results of our process to identify promising design patterns through user selection of visualization features from the design space, followed by user interviews.

Most current VE applications make use of well-known 3D interaction techniques, such as ray-casting or the Go-Go technique. Such techniques, however, were designed generically, without considering the context of use. We argue that the design of 3D interaction techniques should take the application domain into account. In our work, we have developed interfaces for a VE application for structural engineering, including interfaces for cloning, a domain-specific task. We designed and compared six cloning interfaces. The techniques designed using domain characteristics resulted in significant performance gains and a better workflow. In addition, those techniques designed from scratch using domain characteristics were generally superior to generic techniques that were modified to consider domain characteristics. Finally, we found that the transition time between actions in these complex interfaces was a major bottleneck for task performance.

The design of virtual environments for applications that have several levels of scale has not been deeply addressed. In particular, navigation in such environments is a significant problem. This paper describes the design and evaluation of two navigation techniques for multiscale virtual environments (MSVEs). Issues such as spatial orientation and understanding were addressed in the design process of the navigation techniques. The evaluation of the techniques was done with two experimental and two control groups. The results show that the techniques we designed were significantly better than the control conditions with respect to the time for task completion and accuracy.

Visual analytics is the science of analytical reasoning facilitated by interactive visual interfaces. It has grown out of and is strongly related to information visualization. Visual analytic tools assist analysts in detecting the expected and discovering the unexpected from complex, noisy, incomplete, heterogeneous, and sometimes deliberately deceptive data. VR/AR research has claimed for years to provide the potential for more effective environments to understand and explore information spaces. This would seem to make VR technology a natural for application to visual analytics. However, visual analytics is focused on the analytical process not the tools and technology. As such, any new methods, techniques, and technologies need to show benefit to analysts working on real problems. Additionally, the majority of visual analytics research funding is not focused on disruptive physical interface technologies. Generally speaking, new technologies and techniques for visual analytics need to function within the current analytical environment. The purpose of this panel is to introduce the domain of visual analytics to the audience and explore how and where VR/AR research can be adapted for use in visual analytics. The panelists have been selected based on topic areas of research that have potential for near term insertion or impact on visual analytics. This panel will focus on the hard issues of defining what it will take to get VR technology introduced into the visual analytics research funding stream.

Empirical evidence of the benefits of immersion is an important goal for the virtual environment (VE) community. Direct comparison of immersive systems and non-immersive systems is insufficient because differences between such systems may be due not only to the level of immersion, but also to other factors, such as the input devices and interaction techniques used. In this paper, a study is presented that separates the effects of level of immersion and 3D interaction technique for a six-degree-of-freedom manipulation task. In the study, two components of immersion -- stereoscopy and field of regard -- were varied and three 3D interaction techniques -- HOMER, Go-Go, and DO-IT (a new keyboard- and mouse-based technique) -- were tested. The results of the experiment show that the interaction technique had a significant effect on object manipulation time, while the two components of immersion did not. The implications of these results are discussed for VE application developers.

Although many virtual environment (VE) technologies such as the four-screen CAVE™ are described as immersive, users can still perceive distractions from the real world. This exposure to real-world distraction may reduce users' sense of presence, and if presence is correlated with performance as some have claimed, the real-world distractions may also hinder performance. Thus, VE designers may want to consider ways to reduce real-world distraction. This paper presents an experiment to investigate the effect of reduced visual stimulus in the peripheral area on user performance and the usability of an immersive VE. We carefully designed three tasks that cause different levels of awareness of the real-world distraction. Using these tasks, we evaluated users' performance and preference in two conditions. The low-stimulus condition was created by hanging a black cloth across the missing back wall of a CAVE. The high-stimulus condition was created by projected animations and real human motion outside the CAVE. The experiments show that reduced distraction may have a positive or negative effect on user performance, depending on the specific tasks and environments.

Spatial collaboration is a specialized form of collaboration where the discussion relates to a physical space. This work investigates how to support distributed spatial collaboration activities. It presents a novel prototype that integrates both two-dimensional and three-dimensional representations. This collaborative software is examined in a qualitative study as a group virtually rearranges their lab furniture. The results describe the group's collaboration and their use of the combined representations. The findings highlight the usefulness of multiple representations and the need for additional features to support collaboration across representations.

User-centered design is often performed without regard to individual user differences in aptitude and experience. The methodology of this study is an anthropological and observational approach observing users performing a selection task using common virtual environment raybased techniques and analyzes the interaction through psychology aptitude tests, questionnaires and observation. The results of this study show the approach yields useful information about users even in a simple task. The study indicates correlations between performance and aptitude test and user behavior performed to overcome difficulties in the task.

Collaborative Virtual Environments allow multiple users to interact collaboratively while taking advantage of the perceptual richness that Virtual Environments (VEs) provide. In this paper, we demonstrate empirically that increasing the level of immersion in a VE can have a beneficial effect on the usability of that environment in a collaborative context. We present the results of a study in which we varied two immersive factors, stereo and head tracking, within the context of a two person collaborative task. Our results indicate that stereo can have a positive effect on task performance; that different levels of immersion have effects that vary with gender; and that varying the level of immersion has a pronounced effect on communication between users. These results show that the level of immersion can play an important role in determining user performance on some collaborative tasks.

This paper describes our recent experimental evaluation of Information-Rich Virtual Environment (IRVE) interfaces. To explore the depth cue/visibility tradeoff between annotation schemes, we design and evaluate two information layout techniques to support search and comparison tasks. The techniques provide different depth and association cues between objects and their labels: labels were displayed either in the virtual world relative to their referent (Object Space) or on an image plane workspace (Viewport Space). The Software Field of View (SFOV) was controlled to 60 or 100 degrees of vertical angle and two groups were tested: those running on a single monitor and those on a tiled nine-panel display. Users were timed, tracked for correctness, and gave ratings for both difficulty and satisfaction on each task. Significant advantages were found for the Viewport interface, and for high SFOV. The interactions between these variables suggest special design considerations to effectively support search and comparison performance across monitor configurations and projection distortions.

Three-dimensional objects in many applications domains, such as architecture and construction, can be extremely complex and can consist of a large number of components. However, many of these complex objects also contain a great deal of repetition. Therefore, cloning techniques, which generate multiple spatially distributed copies of an object to form a repeated pattern, can be used to model these objects more efficiently. Such techniques are important and useful in desktop three-dimensional modeling systems, but we are not aware of any cloning techniques designed for immersive virtual environments (VEs). In this paper, we present an initial effort toward the design and development of such interfaces. We define the design space of the cloning task, and present five novel VE interfaces for cloning, then articulate the design rationale. We have also performed a usability study intended to elicit subjective responses with regard to affordances, feedback, attention, perceived usefulness, ease of use, and ease of learning in these interfaces. The study resulted in four major conclusions. First, slider widgets are better suited for discrete than for continuous numeric input. Second, the attentional requirements of the interface increase with increased degrees-of-freedom associated with widgets. Third, users prefer constrained widget movement, although more degrees-of-freedom allow more efficient parameter setting. Finally, appropriate feedback can reduce the cognitive load. The lessons we learned will influence our continuing design of cloning techniques, and these techniques will ultimately be applied to VE applications for design, construction, and prototyping

Start of the above-titled section of the conference proceedings record.

Immersive virtual environments (VEs) offer interactive, real-time visualization capabilities for engineers, architects, and scientists. This paper presents Virtual-SAP, an immersive VE application allowing users to assess building structures and their response to various environmental conditions through an interactive design-build-test-redesign cycle. Virtual-SAP has three distinct user interfaces that support its use with a high-end, multi-tracker VE system, with a low-cost portable VE system, or on the desktop. Two of these interfaces have been proven highly usable in user testing, and the third will be tested soon. Virtual-SAP is being used for both research and education.

Virtual Reality (VR), Augmented Reality (AR) and other realities along the Mixed Reality(MR) continuum use multi-sensory displays and spatial interaction to enable a number of usefuland novel applications. These virtual worlds are able to train, explore and entertain in addition toaugment the real world with information for our everyday lives. To perform tasks such as traveland other interaction, researchers in these worlds have focused heavily on issues of three-dimensionalinteraction, including interaction techniques, user interface metaphors and inputdevices but always with an outlook centric to their particular position along the MR continuum.This has resulted in interfaces tuned to certain MR instances that are not possible in them allbecause of factors such as the display types, tracking issues or interference and limits from thereal world.This panel brings together diverse researchers across the MR continuum to discuss theirresearch and dissect the similarities and differences of each other's fields. The result is anunderstanding of interfaces across these realities and a merger of developing terminology,techniques and tools for future collaborations.Panel members will give a description of their research followed by a short vision of futureinterface research. Audience questions submitted via index cards are heavily encouraged andexpected to shape the panel discussion and direction.

Virtual environments (VEs) allow users to experience and interact with a rich sensory environment, but most virtual worlds contain only sensory information similar to that which we experience in the physical world. Information-rich virtual environments (IRVEs) combine the power of VEs and information visualization, augmenting VEs with additional abstract information such as text, numbers, or graphs. IRVEs can be useful in many contexts, such as education, medicine, or construction. In our work, we are developing a theoretical foundation for the study of IRVEs and tools for their development and evaluation. We present a working definition of IRVEs, a discussion of information display and interaction in IRVEs. We also describe a software framework for IRVE development and a testbed enabling evaluation of text display techniques for IRVEs. Finally, we present a research agenda for this area.

Cooperative manipulation refers to the simultaneous manipulation of a virtual object by multiple users in an immersive virtual environment. This paper describes a framework supporting the development of collaborative manipulation techniques, and example techniques we have tested within this framework. We describe the modeling of cooperative interaction techniques, methods of combining simultaneous user actions, and the awareness tools used to provide the necessary knowledge of partner activities during the cooperative interaction process. Our framework is based on a Collaborative Metaphor concept that defines rules to combine user interaction techniques. The combination is based on the separation of degrees of freedom between two users. Finally, we present novel combinations of two interaction techniques (Simple Virtual Hand and Ray-casting).

Interfaces for system control tasks in virtual environments (VEs) have not been extensively studied. The paper focuses on various types of menu systems to be used in such environments. We describe the design of the TULIP menu, a menu system using Pinch Gloves/sup TM/, and compare it to two common alternatives: floating menus and pen and tablet menus. These three menus were compared in an empirical evaluation. The pen and tablet menu was found to be significantly faster, while users had a preference for TULIP. Subjective discomfort levels were also higher with the floating menus and pen and tablet.

As immersive virtual environment (VE) applications become more complex, it is clear that we need a firm understanding of the principles of VE interaction. In particular, designers need guidance in choosing three-dimensional interaction techniques. In this paper, we present a systematic approach, testbed evaluation, for the assessment of interaction techniques for VEs. Testbed evaluation uses formal frameworks and formal experiments with multiple independent and dependent variables in order to obtain a wide range of performance data for VE interaction techniques. We present two testbed experiments, covering techniques for the common VE tasks of travel and object selection/manipulation. The results of these experiments allow us to form general guidelines for VE interaction, and to provide an empirical basis for choosing interaction techniques in VE applications. This has been shown to produce measurable usability gains in a real-world VE application.

Grabbing and manipulating virtual objects is an important user interaction for immersive virtual environments. We present implementations and discussion of six techniques which allow manipulation of remote objects. A user study of these techniques was performed which revealed their characteristics and deficiencies, and led to the development of a new class of techniques. These hybrid techniques provide distinct advantages in terms of ease of use and efficiency because they consider the tasks of grabbing and manipulation separately. CR Categories and Subject Descriptors: I.3.7 [Computer Graphics]:Three-Dimensional Graphics and Realism - Virtual Reality; I.3.6 [Computer Graphics]:Methodology and Techniques - Interaction Techniques. 1 INTRODUCTION A defining feature of virtual reality (VR) is the ability to manipulate virtual objects interactively, rather than simply viewing a passive environment. This capability is desirable in many VR applications and is typically accomplished using ...