A replication attempts to confirm outcomes of earlier research, which is critical in validating and generalizing scientific findings. Yet, its prevalence and practices remain underexplored in Augmented and Virtual Reality (AR/VR) research. To address this, we present a scoping literature review of replication studies within IEEE ISMAR and IEEE VR, spanning 15 years from 2010 to 2024. Our analysis revealed that replication in AR/VR research is rare. Of 2167 total papers reviewed, less than 4 % were identified as replication studies. Of these, conceptual replications were the predominant type (57 %). Most of those were studies in VR (67 %) and within-subject designs (66 %). Complementing the literature survey results, we conducted an online survey with 61 participants about their experiences with replication studies and found that 39 % of them had conducted a replication study. However, limited resources and external motivation hamper the execution of replication studies among these AR/VR researchers. Combining the findings from our literature review and online survey, we discuss the current state of replication research and the factors contributing to its infrequency. We provide recommendations to improve AR/VR research replication practices, focusing on research culture and reporting, and discuss ongoing challenges.

In an optical see-through augmented reality system, virtual and real information can be viewed at different distances. This requires users to frequently change their eye focus from one distance to another, and only one piece of information is in sharp focus while the other is out of focus. Previous studies have found that due to out-of-focus virtual information, when integrating information between the distances, users suffer fatigue and miss important information. Therefore, this paper introduces a novel font, termed a SharpView font, which looks sharper and more legible than standard fonts when seen out of focus. Our method models out-of-focus blur with Zernike polynomials and coefficients, develops a focus correction algorithm based on constrained total variation optimization, and proposes a novel gradient-based algorithm to quantify the sharpness of textual information. We have evaluated the SharpView font through simulation and optically viewed camera-based measurement. When seen out of focus, our proposed font are significantly sharper than standard fonts, as assessed both visually and quantitatively through simulation (40%-44%), as well as the optics of an augmented reality display (24%-32%).

The IEEE VR Steering Committee congratulates and offers gratitude to the enormous efforts of the IEEE VR Conference Organizing Committees—with special recognition for the multiple years of hard work of the General Chairs, Carolina Cruz-Neira, Greg Welch, and Xubo Yang. To realize the conference requires a sizeable, motivated team of volunteers to work with the IEEE conference management team. Maintaining the conference’s status as the premier international virtual reality conference is a testament to the commitment of all the organizers and participants that make up the IEEE Virtual Reality community.

The IEEE VR Steering Committee congratulates and offers gratitude to the enormous efforts of the IEEE VR Conference Organizing Committees — with special recognition for the multiple years of hard work of the General Chairs, Xubo Yang, Kun Zhou, Tobias Langlotz, and Stephan Lukosch. To realize the conference requires a sizeable, motivated team of volunteers to work with the IEEE conference management team. Maintaining the conference's status as the premier international virtual reality conference is a testament to the commitment of all the organizers and participants that make up the IEEE Virtual Reality community.

The 2023 IEEE VGTC Virtual Reality Service Award goes to J. Edward Swan II of Mississippi State University in recognition of his dedication, support, and many years of service contributions to the VR/AR academic community in operational capacity and in numerous leadership roles.Specifically, Prof. Swan serves as chair of the IEEE VR Steering Committee.He also served as a general chair of IEEE VR conferences, and as a chair of the international program committees for IEEE VR and ISMAR.The IEEE VGTC is pleased to award J. Edward Swan II the 2023 Virtual Reality Service Award.J. Edward Swan II started his research in computer graphics and human-computer interaction after receiving a B.S. (1989) degree in Computer Science from Auburn University and M.S. (1992) and Ph.D. (1997) degrees in Computer Science from Ohio State University.His work at the Naval Research Laboratory (NRL) in Washington D.C. allowed him to expand his expertise in augmented and virtual reality, perception, data science, empirical methods, human-computer interaction, human factors, and visualization.His early involvement with the world of conference volunteer activity started as a graduate student and scientist at the NRL.The first duties were the positions of Student Volunteers Co-Chair for IEEE Visualization in 1995 and 1996, followed by serving as a chair for panels, mini-workshops and birds-of-a-feather, and technical program scheduling for the same conference.His first IEEE 1999 conference was VR 1999 in Houston Texas, after which that event became his primary conference and community.As his research focus shifted to augmented reality, he became involved with the IEEE VR conference as a Workshop Co-Chair (2006, 2012) and Posters Co-Chair (2010, 2011).His engagement continued with a stream of roles that included Program Co-Chair for IEEE VR (2014, 2015), IEEE VR Steering Committee (member 2018-2020, chair 2020-present), General Co-Chair for IEEE VR (2020, 2021), IEEE ISMAR Program Co-Chair (2016, 2017), and a member of the IEEE ISMAR Steering Committee (2017)(2018)(2019)(2020)(2021)(2022).A variety of formal roles that Dr. Swan had in all conferences included not only the work that those roles assumed but also new efforts that he initiated.His genuine interest in the technical topics and goals of the VR community led to several significant contributions to the VR domain.In 2020, together with several other colleagues, he led an effort to produce the conference proceedings while dramatically shortening the period between conference paper presentations and the papers being publicly available in the IEEE Digital Library.Taking an idea from ISMAR, they produced three conference proceedings: the TVCG special issue, the conference

Virtual Reality (VR) technology has advanced to include eye-tracking, allowing novel research, such as investigating how our visual system coordinates eye movements with changes in perceptual depth. The purpose of this study was to examine whether eye tracking could track perceptual depth changes during a visual discrimination task. We derived two depth-dependent variables from eye tracker data: eye vergence angle (EVA) and interpupillary distance (IPD). As hypothesized, our results revealed that shifting gaze from near-to-far depth significantly decreased EVA and increased IPD, while the opposite pattern was observed while shifting from far-to-near. Importantly, the amount of change in these variables tracked closely with relative changes in perceptual depth, and supported the hypothesis that eye tracker data may be used to infer real-time changes in perceptual depth in VR. Our method could be used as a new tool to adaptively render information based on depth and improve the VR user experience.

Modern machine learning (ML) is based to a great extent on supervised deep learning models that require large amounts of labeled training data. While image data sets with annotations exist, the annotations are produced manually and possess relatively simple descriptions. To date, none of the freely available labeled image data sets incorporate spatial reasoning, one of Gardner's nine human intelligences. This article presents a new process with open source tools provided to label imagery based on spatial interactions between image objects and auto-mated reasoning under uncertainty. The resulting annotated data can be used to train new ML/AI algorithms and/or help us better understand existing methodologies.

In optical see-through augmented reality (AR), information is often distributed between real and virtual contexts, and often appears at different distances from the user. To integrate information, users must repeatedly switch context and change focal distance. If the user's task is conducted under time pressure, they may attempt to integrate information while their eye is still changing focal distance, a phenomenon we term transient focal blur. Previously, Gabbard, Mehra, and Swan (2018) examined these issues, using a text-based visual search task on a one-eye optical see-through AR display. This paper reports an experiment that partially replicates and extends this task on a custom-built AR Haploscope. The experiment examined the effects of context switching, focal switching distance, binocular and monocular viewing, and transient focal blur on task performance and eye fatigue. Context switching increased eye fatigue but did not decrease performance. Increasing focal switching distance increased eye fatigue and decreased performance. Monocular viewing also increased eye fatigue and decreased performance. The transient focal blur effect resulted in additional performance decrements, and is an addition to knowledge about AR user interface design issues.

For optical see-through augmented reality (AR), a new method for measuring the perceived three-dimensional location of virtual objects is presented, where participants verbally report a virtual object’s location relative to both a vertical and horizontal grid. The method is tested with a small (1.95 × 1.95 × 1.95 cm) virtual object at distances of 50 to 80 cm, viewed through a Microsoft HoloLens 1stgeneration AR display. Two experiments examine two different virtual object designs, whether turning in a circle between reported object locations disrupts HoloLens tracking, and whether accuracy errors, including a rightward bias and underestimated depth, might be due to systematic errors that are restricted to a particular display. Turning in a circle did not disrupt HoloLens tracking, and testing with a second display did not suggest systematic errors restricted to a particular display. Instead, the experiments are consistent with the hypothesis that, when looking downwards at a horizontal plane, HoloLens 1stgeneration displays exhibit a systematic rightward perceptual bias. Precision analysis suggests that the method could measure the perceived location of a virtual object within an accuracy of less than 1 mm.

Many augmented reality (AR) applications operate within near-field reaching distances, and require matching the depth of a virtual object with a real object. The accuracy of this matching was measured in three experiments, which examined the effect of focal distance, age, and brightness, within distances of 33.3 to 50 cm, using a custom-built AR haploscope. Experiment I examined the effect of focal demand, at the levels of collimated (infinite focal distance), consistent with other depth cues, and at the midpoint of reaching distance. Observers were too young to exhibit age-related reductions in accommodative ability. The depth matches of collimated targets were increasingly overestimated with increasing distance, consistent targets were slightly underestimated, and midpoint targets were accurately estimated. Experiment II replicated Experiment I, with older observers. Results were similar to Experiment I. Experiment III replicated Experiment I with dimmer targets, using young observers. Results were again consistent with Experiment I, except that both consistent and midpoint targets were accurately estimated. In all cases, collimated results were explained by a model, where the collimation biases the eyes' vergence angle outwards by a constant amount. Focal demand and brightness affect near-field AR depth matching, while age-related reductions in accommodative ability have no effect.

A haploscope is an optical system which produces a carefully controlled virtual image. Since the development of Wheatstone's original stereoscope in 1838, haploscopes have been used to measure perceptual properties of human stereoscopic vision. This paper presents an augmented reality (AR) haploscope, which allows the viewing of virtual objects superimposed against the real world. Our lab has used generations of this device to make a careful series of perceptual measurements of AR phenomena, which have been described in publications over the previous 8 years. This paper systematically describes the design, assembly, calibration, and measurement of our AR haploscope. These methods have been developed and improved in our lab over the past 10 years. Despite the fact that 180 years have elapsed since the original report of Wheatstone's stereoscope, we have not previously found a paper that describes these kinds of details.

In augmented reality (AR) environments, information is often distributed between real world and virtual contexts, and often appears at different distances from the user. Therefore, to integrate the information, users must repeatedly switch context and refocus the eyes. To focus at different distances, the user's eyes must accommodate, which when done repeatedly can cause eyestrain and degrade task performance. An experiment was conducted that examined switching context and focal distance between a real and an AR environment, using a text-based visual search task and a monocular optical see-through AR display. Both context switching and focal distance switching resulted in significantly reduced performance. In addition, repeatedly performing the task caused visual fatigue to steadily increase. Performance was particularly poor for virtual text presented at optical infinity, and for target letters that participants tried to read before their eyes had completely accommodated to a new focal distance. The results show that context switching and focal distance switching are important AR user interface design issues.

The use of Optical See-Through (OST) technology for presenting Augmented Reality (AR) experiences is becoming more common. However, OST-AR displays require a calibration procedure, in order to determine the location of the users eyes. Currently, the predominantly cited manual calibration technique is the Single Point Active Alignment Method (SPAAM). However, with the SPAAM technique, there remains uncertainty about the causes of poor calibration results. This paper reports an experiment which examined the influence of two factors on SPAAM accuracy and precision: alignment point distribution, and user posture. Alignment point distribution is examined at user-centered reaching distances, 0.15 to 0.3 meters, as well as environment-centered room-scale distances, 0.5 to 2.0 meters. User posture likely contributes to misalignment error, and is examined at the levels of sitting and standing. In addition, a control condition replaces the user with a rigidly-mounted camera, and mounts the OST display on a precisely-adjustable tripod. The experiment finds that user-centric distributions are more accurate than environment-centric distributions, and, somewhat surprisingly, that the users posture has no effect. The control condition replicates these findings. The implication is that alignment point distribution is the predominant mode for induction of calibration error for SPAAM calibration procedures.

The use of Optical See-Through Head-Mounted Displays (OST-HMDs) for presenting Augmented Reality experiences has become more common, due to the increasing availability of lower cost head-worn device options. Despite this growth, commercially available OST hardware remains devoid of the integrated eye-tracking cameras necessary for automatically calibrating user-specific view parameters, leaving manual calibration methods as the most consistently viable option across display types. The Single Point Active Alignment Method (SPAAM) is currently the most-cited manual calibration technique, due to the relaxation of user constraints with respect to allowable motion during the calibration process. This work presents the first formal study directly investigating the effects that alignment point distribution imposes on SPAAM calibration accuracy and precision. A user experiment, employing a single expert user, is presented, in which SPAAM calibrations are performed under each of five conditions. Four of the conditions cross alignment distance (arm length, room scale) with user pose (sitting, standing). The fifth condition is a control condition, in which the user is replaced with a rigidly mounted camera; the control condition removes the effect of noise from uncontrollable postural sway. The final experimental results show no significant impact on calibration due to user pose (sitting, standing). The control condition also did not differ from the user produced calibration results, suggesting that posture sway was not a significant factor. However, both the user and control conditions show significant improvement using arm's length alignment points over room scale alignments, with an order of magnitude difference in eye location estimate error between conditions.

Presents the introductory welcome message from the conference proceedings. May include the conference officers' congratulations to all involved with the conference event and publication of the proceedings record.

Presents the introductory welcome message from the conference proceedings. May include the conference officers' congratulations to all involved with the conference event and publication of the proceedings record.

Optical see-through head-mounted displays (OST HMDs) are a major output medium for Augmented Reality, which have seen significant growth in popularity and usage among the general public due to the growing release of consumer-oriented models, such as the Microsoft Hololens. Unlike Virtual Reality headsets, OST HMDs inherently support the addition of computer-generated graphics directly into the light path between a user's eyes and their view of the physical world. As with most Augmented and Virtual Reality systems, the physical position of an OST HMD is typically determined by an external or embedded 6-Degree-of-Freedom tracking system. However, in order to properly render virtual objects, which are perceived as spatially aligned with the physical environment, it is also necessary to accurately measure the position of the user's eyes within the tracking system's coordinate frame. For over 20 years, researchers have proposed various calibration methods to determine this needed eye position. However, to date, there has not been a comprehensive overview of these procedures and their requirements. Hence, this paper surveys the field of calibration methods for OST HMDs. Specifically, it provides insights into the fundamentals of calibration techniques, and presents an overview of both manual and automatic approaches, as well as evaluation methods and metrics. Finally, it also identifies opportunities for future research.

Does visually perceived distance differ when objects are viewed in augmented reality (AR), as opposed to the real world? What are the differences? These questions are theoretically interesting, and the answers are important for the development of many tablet- and phone-based AR applications, including mobile AR navigation systems. This article presents a thorough literature review of distance judgment experimental protocols, and results from several areas of perceptual psychology. In addition to distance judgments of real and virtual objects, this section also discusses previous work in measuring the geometry of virtual picture space and considers how this work might be relevant to tablet AR. Then, the article presents the results of two experiments. In each experiment, observers bisected egocentric distances of 15 and 30 m in tablet-based AR and in the real world, in both indoor corridor and outdoor field environments. In AR, observers bisected the distances to virtual humans, while in the real world, they bisected the distances to real humans. This is the first reported research that directly compares distance judgments of real and virtual objects in a tablet AR system. Four key findings were: (1) In AR, observers expanded midpoint intervals at 15 m, but compressed midpoints at 30 m. (2) Observers were accurate in the real world. (3) The environmental setting—corridor or open field—had no effect. (4) The picture perception literature is important in understanding how distances are likely judged in tablet-based AR. Taken together, these findings suggest the depth distortions that AR application developers should expect with mobile and especially tablet-based AR.

The growing prevalence of hand and gesture tracking technology has led to an increased availability of consumer level devices, such as the Leap Motion controller, and also facilitated the inclusion of similar hardware into forth coming head mounted display offerings, including the Microsoft HoloLens and Moverio Pro BT-2000. In this video, we demonstrate the utility of the Leap Motion for calibrating optical see-through augmented reality systems by employing a variation on Tuceryan and Navab's Single Point Active Alignment Method [3]. We also showcase a straightforward method for calibrating the coordinate frame of the Leap Motion to a secondary tracking system by employing absolute orientation algorithms [2, 1, 4], allowing us to properly transform and visualize hand and finger tracking data from the user's viewpoint. Our combined display and coordinate frame calibration techniques produce a viable mechanism for not only intuitive interaction with virtual objects but also the creation of natural occlusion between computer generated content and the user themselves. We believe that these techniques will be pivotal in the development of novel consumer applications for next generation head mounted display hardware.

Next generation OST HMDs promise the inclusion of a variety of integrated and on-board sensors. In particular, hand tracking cameras, such as the Leap Motion, show potential for facilitating intuitive OST calibration procedures accessible to researchers, developers, and novice users alike. In this work, we evaluate hand and stylus based OST calibration utilizing tracking data from a Leap Motion. Our findings show that performance of both methods is comparable to results from prior studies using standard environment-centric methods. Also, while our hand based calibration improved through the use of more contextual reticle designs, calibrations performed with a stylus yielded the most accurate and precise results over all.

A common factor among current generation optical see-through augmented reality systems is fixed focal distance to virtual content. In this work, we investigate the issue of focus blur, in particular, the blurring caused by simultaneously viewing virtual content and physical objects in the environment at differing focal distances. We examine the application of dynamic sharpening filters as a straight forward, system independent, means for mitigating this effect improving the clarity of defocused AR content. We assess the utility of this method, termed SharpView, by employing an adjustment experiment in which users actively apply varying amounts of sharpening to reduce the perception of blur in AR content. Our experimental results validate the ability of our SharpView model to improve the visual clarity of focus blurred content, with optimal performance at focal differences well suited for near field AR applications.

Correct spatial alignment is an essential requirement for convincing augmented reality experiences. Registration error, caused by a variety of systematic, environmental, and user influences decreases the realism and utility of head mounted display AR applications. Focus is often given to rigorous calibration and prediction methods seeking to entirely remove misalignment error between virtual and real content. Unfortunately, producing perfect registration is often simply not possible. Our goal is to quantify the sensitivity of users to registration error in these systems, and identify acceptability thresholds at which users can no longer distinguish between the spatial positioning of virtual and real objects. We simulate both video see-through and optical see-through environments using a projector system and experimentally measure user perception of virtual content misalignment. Our results indicate that users are less perceptive to rotational errors over all and that translational accuracy is less important in optical see-through systems than in video see-through.

We are investigating methods for improving the robustness and consistency of the Single Point Active Alignment Method (SPAAM) optical see-through (OST) head-mounted display (HMD) calibration procedure. Our investigation focuses on two variants of SPAAM. The first utilizes a standard monocular alignment strategy to calibrate the left and right eye separately, while the second leverages stereoscopic cues available from binocular HMDs to calibrate both eyes simultaneously. We compare results from repeated calibrations between methods using eye location estimates and inter pupillary distance (IPD) measures. Our findings indicate that the stereo SPAAM method produces more accurate and consistent results during calibration compared to the monocular variant.

The current advent of consumer level optical see-through (OST) head-mounted displays (HMD's) has greatly broadened the accessibility of Augmented Reality (AR) to not only researchers but also the general public as well. This increased user base heightens the need for robust automatic calibration mechanisms suited for nontechnical users. We are developing a fully automated calibration system for two stereo OST HMD's, a consumer level and prototype model, based on the recently introduced interaction free display calibration (INDICA) method. Our current efforts are also focused on the development of an evaluation process to assess the performance of the system during use by non-expert subjects.

Summary form only given. Effectively evaluating optical see-through (OST) head-mounted display (HMD) calibration is problematic and largely relies on feedback from the user. Studies evaluating OST HMD calibration, such as those by McGarrity, Tang, and Navab et al. [2, 3, 1], utilize user interaction methods, such as touch pads, to facilitate on-line evaluation and correction of calibration results. In all of these studies, however, only the users themselves receive any visual feedback related to the calibration quality or the corrective actions taken to improve it. In this video, we present the use of standard frustum visualization to provide calibration quality information to the researcher in real time. We use a standard Single Point Active Alignment Method (SPAAM) calibration, [4], after which both the eye location estimate and resulting intrinsic values are displayed superimposed onto the user. Presenting the eye position relative to the user's head benefits studies on system error sources, and rendering on-screen visuals also allows outside observers to identify calibration issues and offer corrective suggestions. We believe that techniques, such as frustum visualization, will expand the amount of information available for evaluating calibration results, and will greatly aid those investigating new and improved calibration procedures.

In previous papers, a novel haploscope-based AR environment was implemented [1, 3]. In that system, a participant looks through a set of reflective lenses onto a real-world environment. However, at the same time, there are monitors to the side displaying a virtual object. This object is reflected onto the lenses and is thus, from the viewpoint of the participant, overlaid onto the real environment. In Hua [1], some initial work was done designing a calibration procedure for this haploscope-based AR environment. The current work seeks to modify and expand Hua's original calibration procedure to make it both more effective and more efficient. As part of developing this new calibration procedure, this paper examines potential sources of error and recommends processes and steps for reducing or eliminating these potential error sources.

With the growing availability of optical see-through (OST) head-mounted displays (HMDs) there is a present need for robust, uncomplicated, and automatic calibration methods suited for non-expert users. This work presents the results of a user study which both objectively and subjectively examines registration accuracy produced by three OST HMD calibration methods: (1) SPAAM, (2) Degraded SPAAM, and (3) Recycled INDICA, a recently developed semi-automatic calibration method. Accuracy metrics used for evaluation include subject provided quality values and error between perceived and absolute registration coordinates. Our results show all three calibration methods produce very accurate registration in the horizontal direction but caused subjects to perceive the distance of virtual objects to be closer than intended. Surprisingly, the semi-automatic calibration method produced more accurate registration vertically and in perceived object distance overall. User assessed quality values were also the highest for Recycled INDICA, particularly when objects were shown at distance. The results of this study confirm that Recycled INDICA is capable of producing equal or superior on-screen registration compared to common OST HMD calibration methods. We also identify a potential hazard in using reprojection error as a quantitative analysis technique to predict registration accuracy. We conclude with discussing the further need for examining INDICA calibration in binocular HMD systems, and the present possibility for creation of a closed-loop continuous calibration method for OST Augmented Reality.

Many compelling augmented reality (AR) applications require users to correctly perceive the location of virtual objects, some with accuracies as tight as 1 mm. However, measuring the perceived depth of AR objects at these accuracies has not yet been demonstrated. In this paper, we address this challenge by employing two different depth judgment methods, perceptual matching and blind reaching, in a series of three experiments, where observers judged the depth of real and AR target objects presented at reaching distances. Our experiments found that observers can accurately match the distance of a real target, but when viewing an AR target through collimating optics, their matches systematically overestimate the distance by 0.5 to 4.0 cm. However, these results can be explained by a model where the collimation causes the eyes' vergence angle to rotate outward by a constant angular amount. These findings give error bounds for using collimating AR displays at reaching distances, and suggest that for these applications, AR displays need to provide an adjustable focus. Our experiments further found that observers initially reach ∼4 cm too short, but reaching accuracy improves with both consistent proprioception and corrective visual feedback, and eventually becomes nearly as accurate as matching.

In many medical Augmented Reality (AR) applications, doctors want the ability to place a physical object, such as a needle or other medical device, at the depth indicated by a virtual object. Often, this object will be located behind an occluding surface, such as the patient's skin. In this poster we describe efforts to determine how accurately this can be done. In particular, we used a table-mounted AR haploscope to conduct two depth-matching experiments.

We have conducted an experiment to study the effect of an occluding surface on the accuracy of near field depth matching in augmented reality (AR). Our experiment was based on replicating a similar experiment conducted by Edwards et al. [2]. We used an AR haploscope [1], which allows us to independently manipulate accommodative demand and vergence angle of the visible image. Fifteen observers matched the perceived depth of an AR-presented virtual object with a physical pointer. Overall, observers overestimated depth by 5 mm or less in the presence of the occluder, while in the absence of an occluder they overestimated depth by 5 to 10 mm. The data from Edwards et al. [2] is normalized, and when we performed the same normalization procedure on our own data, our results do not agree with Edwards et al. [2]. We suspect that eye vergence explains these results.

We have conducted an experiment to study the effect of an occluding surface on the accuracy of near field depth matching in augmented reality (AR). Our experiment was based on replicating a similar experiment conducted by Edwards et al. [2]. We used an AR haploscope, which allows us to independently manipulate accommodative demand and vergence angle. Sixteen observers matched the perceived depth of an AR-presented virtual object with a physical pointer. Overall, observers overestimated depth by 6 mm or less with or without the presence of the occluder. The data from Edwards et al. [2] is normalized, and when we performed the same normalization procedure on our own data, our results do not agree with Edwards et al. [2]. We suspect that eye vergence explains these results.

We conducted an experiment in an attempt to generate baseline accuracy and precision values for optical see-through (OST) head mounted display (HMD) calibration without the inclusion of human postural sway error. This preliminary work will act as a control condition for future studies into postural error reduction. An experimental apparatus was constructed to allow performance of a SPAAM calibration using 25 alignments taken using one of three distance distribution patterns: static, sequential, and magic square. The accuracy of the calibrations were determined by calculating the extrinsic X, Y, Z translation values from the resulting projection matrix. The standard deviation for each translation component was also calculated. The results show that the magic square distribution resulted in the most accurate parameter estimation and also resulted in the smallest standard deviation for each extrinsic translation component.

This tutorial is for researchers and engineers, working in the field of Virtual and Augmented Reality, who wish to conduct user-based experiments and/or evaluations for assessing usability. We propose a full-day tutorial presenting both quantitative and qualitative approaches to conducting human-subject experiments. It will cover (1) the basic principles of experimental design and analysis, with an emphasis on human-subject experiments in AR (Swan), and (2) qualitative studies (e.g., formative evaluation methods) for assessing and improving AR user interfaces and user interaction along with lessons learned from conducting many user-based studies (Gabbard). Swan, Gabbard, and other co-presenters have taught pre-cursor versions of this tutorial 11 previous times at IEEE Virtual Reality, IEEE Visualization, and ISMAR. This tutorial was most recently given at ISMAR 2012, where we included updated examples from our research and further expanded upon qualitative approaches for assessing usability and lessons learned from conducting studies. We both have current, active AR human-subject research projects, and if this tutorial is accepted to be presented at VR 2014, we will discuss some of these projects as case studies.

In this special issue of IEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present the long papers from the IEEE Virtual Reality Conference 2014 (IEEE VR 2014), held March 29–April 2, 2014 in Minneapolis, Minnesota, USA.

Current augmented reality (AR) systems often fail to indicate the distance between the user and points of interest in the environment. Empirical evaluations of human depth perception in AR settings compared to real world settings are needed. Our goal in this study was to understand tablet-based AR depth perception by comparing it with real-world depth perception.

When viewing augmented reality (AR) through an optical see-through head-mounted display (oHMD), the colors of AR elements become washed out as the luminance of the background increases. In addition, the colors of AR elements can shift and change depending on the color of the background that the AR elements are seen against. Both of these phenomena result from the way the color of an AR element blends with the color of the real-world background that the AR element is seen against. Although the fact that this color blending occurs is generally known, to date it has not been systematically studied or quantified. In this work, we sought to quantify this color blending by building an apparatus that allows precise and repeatable color measurements, and then conducting an empirical engineering study.

The following series of experiments explore the effect of static peripheral stimulation on the perception of distance and spatial scale in a typical head-mounted virtual environment. It was found that applying constant white light in an observer's far periphery enabled the observer to more accurately judge distances using blind walking. An effect of similar magnitude was also found when observers estimated the size of a virtual space using a visual scale task. The presence of the effect across multiple psychophysical tasks provided confidence that a perceptual change was, in fact, being invoked by the addition of the peripheral stimulation. These results were also compared to observer performance in a very large field of view virtual environment and in the real world. The subsequent findings raise the possibility that distance judgments in virtual environments might be considerably more similar to those in the real world than previous work has suggested.

A previous study indicated that peripheral visual information strongly affects the judgment of egocentric distances for users of immersive virtual environments. The experiment described in this document aimed to investigate if these effects could be explained in terms of changes in gait caused by visual information in the extreme periphery. Three conditions with varying degrees of peripheral occlusion were tested and participants' walking characteristics measured. The results indicate that the improvements in distance judgments, as peripheral information increases, can only partially be explained in terms of gait modification, but likely involve both changes in the characteristics of gait and other spatial or movement parameters.

This tutorial is for researchers and engineers, working in the field of Virtual Reality (VR) and Augmented Reality (AR), who wish to conduct user-based experiments with a specific aim of promoting both traditional quantitative human-subject experiments and qualitative methods for assessing usability. This tutorial is for a full-day tutorial presenting both quantitative and qualitative approaches to conducting human-subject experiments. It covers (1) the basic principles of experimental design and analysis, with an emphasis on human-subject experiments in VR/AR; (2) qualitative studies (e.g., formative evaluation methods) for assessing and improving VR/AR user interfaces and user interaction along with lessons learned from conducting many user-based studies; and (3) a “journalistic approach” to measuring human performance that organizes the activity around questions such as “Who? What? When? Where? How? and Why?”.

In this abstract we describe an experiment that measured depth judgments in optical see-through augmented reality (AR) at near-field reaching distances of ~ 24 to ~ 56 cm. The 2×2 experiment crossed two depth judgment tasks, perceptual matching and blind reaching, with two different environments, a real-world environment and an augmented reality environment. We designed a task that used a direct reaching gesture at constant percentages of each participant's maximum reach; our task was inspired by previous work by Tresilian and Mon-Williams [6] that found very accurate blind reaching results in a real-world environment.

This paper presents a 2D flow visualization user study that we conducted using new methodologies to increase the objectiveness. We evaluated grid-based variable-size arrows, evenly spaced streamlines, and line integral convolution (LIC) variants (basic, oriented, and enhanced versions) coupled with a colorwheel and/or rainbow color map, which are representative of many geometry-based and texture-based techniques. To reduce data-related bias, template-based explicit flow synthesis was used to create a wide variety of symmetric flows with similar topological complexity. To suppress task-related bias, pattern-based implicit task design was employed, addressing critical point recognition, critical point classification, and symmetric pattern categorization. In addition, variable-duration and fixed-duration measurement schemes were utilized for lightweight precision-critical and heavyweight judgment intensive flow analysis tasks, respectively, to record visualization effectiveness. We eliminated outliers and used the Ryan REGWQ post-hoc homogeneous subset tests in statistical analysis to obtain reliable findings. Our study shows that a texture-based dense representation with accentuated flow streaks, such as enhanced LIC, enables intuitive perception of the flow, while a geometry-based integral representation with uniform density control, such as evenly spaced streamlines, may exploit visual interpolation to facilitate mental reconstruction of the flow. It is also shown that inappropriate color mapping (e.g., colorwheel) may add distractions to a flow representation.

A frequently observed problem in virtual environments is the underestimation of egocentric depth. This problem has been described numerous times and with widely varying degrees of severity. Though there has been considerable progress made in modifying observer behavior to compensate for these misperceptions, the question of why these errors exist is still an open issue. The study detailed in this document presents the preliminary findings of a large, between-subjects experiment (N=98) that attempts to identify and quantify the source of a pattern of adaptation and improved accuracy in the absence of explicit feedback found in Jones et al. [1].

In this poster abstract we describe an experiment that measured depth judgments in optical see-through augmented reality at near-field distances of 34 to 50 centimeters. The experiment compared two depth judgment tasks: perceptual matching, a closed-loop task, and blind reaching, a visually open-loop task. The experiment tested each of these tasks in both a real-world environment and an augmented reality environment, and used a between-subjects design that included 40 participants. The experiment found that matching judgments were very accurate in the real world, with errors on the order of millimeters and very little variance. In contrast, matching judgments in augmented reality showed a linear trend of increasing overestimation with increasing distance, with a mean overestimation of ~1 cm. With reaching judgments participants underestimated ~4.5 cm in both augmented reality and the real world. We also discovered and solved a calibration problem that arises at near-field distances.

This paper provides a classification of perceptual issues in augmented reality, created with a visual processing and interpretation pipeline in mind. We organize issues into ones related to the environment, capturing, augmentation, display, and individual user differences. We also illuminate issues associated with more recent platforms such as handhelds or projector-camera systems. Throughout, we describe current approaches to addressing these problems, and suggest directions for future research.

It is well-documented that natural lighting conditions and real-world backgrounds affect the usability of optical see-through augmented reality (AR) displays in outdoor environments. In many cases, outdoor environmental conditions can dramatically alter users' color perception of user interface elements, by for example, washing out text or icon colors. As a result, users' semantic interpretation of interface elements can be compromised, rendering interface designs useless or counter-productive — an especially critical problem in application domains where color encoding is critical, such as military or medical visualization. In this paper, we present our experiences designing and constructing an optical AR testbed that emulates outdoor lighting conditions and allows us to measure the combined color of real-world backgrounds and virtual colors as projected through an optical see-through display. We present a formalization of color blending in AR, which supports further research on perceived color in AR displays. We describe an engineering study where we measure the color of light that reaches an optical see-through display user's eye under systematically varied virtual and real-world conditions. Our results further quantify the effect of lighting and background color on the color of virtual graphics, and specifically quantify how virtual colors change based on different real-world backgrounds.

Abstract A perceptual study of four tensor glyphs for symmetric, real, traceless tensors was performed. Each glyph encodes three properties of the system: Orientation, uniaxiality (alignment along the direction of orientation), and biaxiality (alignment along a vector orthogonal to the orientation). Thirty users over two studies were asked to identify these three properties for each glyph type under a variety of permutations in order to evaluate the effectiveness of visually communicating the properties; response time was also measured. We discuss the significant differences found between the methods as guidance to the use of these glyphs for traceless tensor visualization.

We describe the development of a set of tools for analyzing the textual contents of digital forensic evidence for the purpose of enhancing an investigator's ability to discover information quickly and efficiently. By examining the textual contents of files and unallocated space, relationships between sets of files and clusters can be formed based on the information that they contain. Using the information gathered from the evidence through the analysis tool, the visualization tool can be used to search through the evidence in an organized and efficient manner. The visualization depicts both the frequency of relevant terms and their location on disk. We also discuss a task analysis with forensics officers to motivate the design.

How do users of virtual environments perceive virtual space? Many experiments have explored this question, but most of these have used head-mounted immersive displays. This paper reports an experiment that studied large-screen immersive displays at medium-field distances of 2 to 15 meters. The experiment measured ego-centric depth judgments in a CAVE, a tiled display wall, and a real-world outdoor field as a control condition. We carefully modeled the outdoor field to make the three environments as similar as possible. Measuring egocentric depth judgments in large-screen immersive displays requires adapting new measurement protocols; the experiment used timed imagined walking, verbal estimation, and triangulated blind walking. We found that depth judgments from timed imagined walking and verbal estimation were very similar in all three environments. However, triangulated blind walking was accurate only in the out-door field; in the large-screen immersive displays it showed under-estimation effects that were likely caused by insufficient physical space to perform the technique. These results suggest using timed imagined walking as a primary protocol for assessing depth perception in large-screen immersive displays. We also found that depth judgments in the CAVE were more accurate than in the tiled display wall, which suggests that the peripheral scenery offered by the CAVE is helpful when perceiving virtual space.

We tested users' depth perception of virtual objects in our mobile augmented reality (AR) system in both indoor and outdoor environments using a depth matching task. The indoor environment is characterized by strong linear perspective cues; we attempted to re-create these cues in the outdoor environment. In the indoor environment, we found an overall pattern of underestimation of depth that is typical for virtual environments and AR systems. However, in the outdoor environment, we found that subjects overestimated depth. In addition, our synthetic linear perspective cues met with a measure of success, leading users to reduce their estimate of the depth of distant objects. We describe the experimental procedure, analyze the data, present the results of the study, and discuss the implications for mobile, outdoor AR systems.

A large number of previous studies have shown that egocentric depth perception tends to be underestimated in virtual reality (VR) - objects appear smaller and farther away than they should. Various theories as to why this might occur have been investigated, but to date the cause is not fully understood. A much smaller number of studies have investigated how depth perception operates in augmented reality (AR), and some of these studies have also indicated a similar underestimation effect. In this paper we report an experiment that further investigates these effects. The experiment compared VR and AR conditions to two real-world control conditions, and studied the effect of motion parallax across all conditions. Our combined VR and AR head-mounted display (HMD) allowed us to develop very careful calibration procedures based on real-world calibration widgets, which cannot be replicated with VR-only HMDs. To our knowledge, this is the first study to directly compare VR and AR conditions as part of the same experiment.

Abstract We present the results from a user study looking at the ability of observers to mentally integrate wind direction and magnitude over a vector field. The data set chosen for the study is an MM5 (PSU/NCAR Mesoscale Model) simulation of Hurricane Lili over the Gulf of Mexico as it approaches the southeastern United States. Nine observers participated in the study. This study investigates the effect of layering on the observer's ability to determine the magnitude and direction of a vector field. We found a tendency for observers to underestimate the magnitude of the vectors and a counter‐clockwise bias when determining the average direction of a vector field. We completed an additional study with two observers to try to uncover the source of the counter‐clockwise bias. These results have direct implications to atmospheric scientists, but may also be able to be applied to other fields that use 2D vector fields.

A major challenge, and thus opportunity, in the field of human-computer interaction and specifically usability engineering is designing effective user interfaces for emerging technologies that have no established design guidelines or interaction metaphors or introduce completely new ways for users to perceive and interact with technology and the world around them. Clearly, augmented reality is one such emerging technology. We propose a usability engineering approach that employs user-based studies to inform design, by iteratively inserting a series of user-based studies into a traditional usability engineering lifecycle to better inform initial user interface designs. We present an exemplar user-based study conducted to gain insight into how users perceive text in outdoor augmented reality settings and to derive implications for design in outdoor augmented reality. We also describe lessons learned from our experiences conducting user-based studies as part of the design process.

A challenge in presenting augmenting information in outdoor augmented reality (AR) settings lies in the broad range of uncontrollable environmental conditions that may be present, specifically large-scale fluctuations in natural lighting and wide variations in likely backgrounds or objects in the scene. In this paper, we present a active AR testbed that samples the user's field of view, and collects outdoor illuminance values at the participant's position. The main contribution presented herein is a user-based study (conducted using the testbed) that examined the effects on user performance of four outdoor background textures, four text colors, three text drawing styles, and two text drawing style algorithms for a text identification task using an optical, see-through AR system. We report significant effects for all these variables, and discuss design guidelines and ideas for future work

Abstract-A fundamental problem in optical, see-through augmented reality (AR) is characterizing how it affects the perception of spatial layout and depth. This problem is important because AR system developers need to both place graphics in arbitrary spatial relationships with real-world objects, and to know that users will perceive them in the same relationships. Furthermore, AR makes possible enhanced perceptual techniques that have no real-world equivalent, such as x-ray vision, where AR users are supposed to perceive graphics as being located behind opaque surfaces. This paper reviews and discusses protocols for measuring egocentric depth judgments in both virtual and augmented environments, and discusses the well-known problem of depth underestimation in virtual environments. It then describes two experiments that measured egocentric depth judgments in AR. Experiment I used a perceptual matching protocol to measure AR depth judgments at medium and far-field distances of 5 to 45 meters. The experiment studied the effects of upper versus lower visual field location, the x-ray vision condition, and practice on the task. The experimental findings include evidence for a switch in bias, from underestimating to overestimating the distance of AR-presented graphics, at approximately 23 meters, as well as a quantification of how much more difficult the x-ray vision condition makes the task. Experiment II used blind walking and verbal report protocols to measure AR depth judgments at distances of 3 to 7 meters. The experiment examined real-world objects, real-world objects seen through the AR display, virtual objects, and combined real and virtual objects. The results give evidence that the egocentric depth of AR objects is underestimated at these distances, but to a lesser degree than has previously been found for most virtual reality environments. The results are consistent with previous studies that have implicated a restricted field-of-view, combined with an inability for observers to scan the ground plane in a near-to-far direction, as explanations for the observed depth underestimation.

Summary form only for tutorial. As virtual and augmented reality hardware and software have grown more mature over the past decade, the focus of the field is shifting away from the basic engineering technology, and towards the science and applications of VR and AR techniques. Increasingly, virtual and augmented reality researchers are conducting human-subject experiments, both to understand the way humans perceive, manipulate, and cognate with VR and AR information, and to quantify the utility of VR and AR in different application contexts.

A fundamental problem in optical, see-through augmented reality (AR) is characterizing how it affects the perception of spatial layout and depth. This problem is important because AR system developers need to both place graphics in arbitrary spatial relationships with real-world objects, and to know that users will perceive them in the same relationships. Furthermore, AR makes possible enhanced perceptual techniques that have no real-world equivalent, such as x-ray vision, where AR users are supposed to perceive graphics as being located behind opaque surfaces. This paper reviews and discusses techniques for measuring egocentric depth judgments in both virtual and augmented environments. It then describes a perceptual matching task and experimental design for measuring egocentric AR depth judgments at medium- and far-field distances of 5 to 45 meters. The experiment studied the effect of field of view, the x-ray vision condition, multiple distances, and practice on the task. The paper relates some of the findings to the well-known problem of depth underestimation in virtual environments, and further reports evidence for a switch in bias, from underestimating to overestimating the distance of AR-presented graphics, at 23 meters. It also gives a quantification of how much more difficult the x-ray vision condition makes the task, and then concludes with ideas for improving the experimental methodology.

A challenge in presenting augmenting information in outdoor augmented reality (AR) settings lies in the broad range of uncontrollable environmental conditions that may be present, specifically large-scale fluctuations in natural lighting and wide variations in likely backgrounds or objects in the scene. In this paper, we present a user-based study which examined the effects of outdoor background textures, changing outdoor illuminance values, and text drawing styles on user performance of a text identification task with an optical, see-through augmented reality system. We report significant effects for all of these variables, and discuss design guidelines and ideas for future work.

Augmented reality (AR) systems present a mixture of virtual and real objects. The challenge for AR system evaluators is how to tell whether the virtual world is effective at conveying the sense of reality. It may never be possible or even necessary to determine whether the user is truly fooled in all situations or is merely suspending disbelief, but one can objectively measure the effectiveness of an AR environment with a task-based approach. We present the results of our first such experiment, involving low-level perceptual tasks of recognition and depth matching.

Provides an abstract of the tutorial presentation and a brief professional biography of the presenter. The complete presentation was not made available for publication as part of the conference proceedings.

A useful function of augmented reality (AR) systems is their ability to visualize occluded infrastructure directly in a user's view of the environment. This is especially important for our application context, which utilizes mobile AR for navigation and other operations in an urban environment. A key problem in the AR field is how to best depict occluded objects in such a way that the viewer can correctly infer the depth relationships between different physical and virtual objects. Showing a single occluded object with no depth context presents an ambiguous picture to the user. But showing all occluded objects in the environments leads to the "Superman's X-ray vision" problem, in which the user sees too much information to make sense of the depth relationships of objects. Our efforts differ qualitatively from previous work in AR occlusion, because our application domain involves far-field occluded objects, which are tens of meters distant from the user. Previous work has focused on near-field occluded objects, which are within or just beyond arm's reach, and which use different perceptual cues. We designed and evaluated a number of sets of display attributes. We then conducted a user study to determine which representations best express occlusion relationships among far-field objects. We identify a drawing style and opacity settings that enable the user to accurately interpret three layers of occluded objects, even in the absence of perspective constraints.

We describe two engineering experiments designed to evaluate the effectiveness of Measurand's ShapeTape for wearable, mobile interaction. Our initial results suggest that the ShapeTape is not appropriate for interactions which require a high degree of accuracy. However, ShapeTape is capable of reproducing the qualitative motion the user is performing and thus could be used to support 3D gesture-based interaction.

We present a comparative study of user performance with tasks involving navigation, visual search, and geometric manipulation, in a map-based battlefield visualization virtual environment (VE). Specifically, our experiment compared user performance of the same task across four different VE platforms: desktop, cave, workbench, and wall. Independent variables were platform type, stereopsis (stereo, mono), movement control mode (rate, position), and frame of reference (egocentric, exocentric). Overall results showed that users performed tasks fastest using the desktop and slowest using the workbench. Other results are detailed in the article. Notable is that we designed our task in an application context, with tasking much closer to how users would actually use a real-world battlefield visualization system. This is very uncommon for comparative studies, which are usually designed with abstract tasks to minimize variance. This is, we believe, one of the first and most complex studies to comparatively examine, in an application context, this many key variables affecting VE user interface design.

The paper describes a computer modeling and simulation system that supports computational steering, which is an effort to make the typical simulation workflow more efficient. Our system provides an interface that allows scientists to perform all of the steps in the simulation process in parallel and online. It uses a standard network flow visualization package, which has been extended to display graphical output in an immersive virtual environment such as a CAVE. Our system allows scientists to interactively manipulate simulation parameters and observe the results. It also supports inverse steering, where the user specifies the desired simulation result, and the system searches for the simulation parameters that achieve this result. Taken together, these capabilities allow scientists to more efficiently and effectively understand model behavior, as well as to search through simulation parameter space. The paper is also a case study of applying our system to the problem of simulating microwave interactions with missile bodies. Because these interactions are difficult to study experimentally, and have important effects on missile electronics, there is a strong desire to develop and validate simulation models of this phenomena.

The ever-increasing power of computers and hardware rendering systems has, to date, primarily motivated the creation of visually rich and perceptually realistic virtual environment (VE) applications. Comparatively very little effort has been expended on the user interaction components of VEs. As a result, VE user interfaces are often poorly designed and are rarely evaluated with users. Although usability engineering is a newly emerging facet of VE development, user-centered design and usability evaluation in VEs as a practice still lags far behind what is needed. This paper presents a structured, iterative approach for the user-centered design and evaluation of VE user interaction. This approach consists of the iterative use of expert heuristic evaluation, followed by formative usability evaluation, followed by summative evaluation. We describe our application of this approach to a real-world VE for battlefield visualization, describe the resulting series of design iterations, and present evidence that this approach provides a cost-effective strategy for assessing and iteratively improving user interaction design in VEs. This paper is among the first to report applying an iterative, structured, user-centered design and evaluation approach to VE user interaction design.

In this paper we describe a battlefield visualization system, called Dragon, which we have implemented on a virtual reality responsive workbench. The Dragon system has been successfully deployed as part of two large military exercises: the Hunter Warrior advanced warfighting experiment, in March 1997, and the Joint Counter Mine advanced concept tactical demonstration, in August and September 1997. We describe battlefield visualization, the Dragon system, and the workbench, and we describe our experiences as part of these two real-world deployments, with an emphasis on lessons learned and needed future work.

This case study describes a technique for the three-dimensional analysis of the internal microscopic structure (microstructure) of materials. This technique consists of incrementally polishing through a thin layer (approximately 0.2 /spl mu/m) of material, chemically etching the polished surface, applying reference marks, and performing optical or scanning electron microscopy on selected areas. The series of images are then processed employing AVS and other visualization software to obtain a 3D reconstruction of the material. We describe how we applied this technique to an alloy steel to study the morphology, connectivity, and distribution of cementite precipitates formed during thermal processing. The results showed microstructural features not previously identified with traditional 2D techniques.

The paper describes three new results for volume rendering algorithms utilizing splatting. First, an antialiasing extension to the basic splatting algorithm is introduced that mitigates the spatial aliasing for high resolution volumes. Aliasing can be severe for high resolution volumes or volumes where a high depth of field leads to converging samples along the perspective axis. Next, an analysis of the common approximation errors in the splatting process for perspective viewing is presented. In this context, we give different implementations, distinguished by efficiency and accuracy, for adding the splat contributions to the image plane. We then present new results in controlling the splatting errors and also show their behavior in the framework of our new antialiasing technique. Finally, current work in progress on extensions to splatting for temporal antialiasing is demonstrated. We present a simple but highly effective scheme for adding motion blur to fast moving volumes.

Splatting is a popular direct volume rendering algorithm. However, the algorithm does not correctly render cases where the volume sampling rate is higher than the image sampling rate (e.g. more than one voxel maps into a pixel). This situation arises with orthographic projections of high-resolution volumes, as well as with perspective projections of volumes of any resolution. The result is potentially severe spatial and temporal aliasing artifacts. Some volume ray-casting algorithms avoid these artifacts by employing reconstruction kernels which vary in width as the rays diverge. Unlike ray-casting algorithms, existing splatting algorithms do not have an equivalent mechanism for avoiding these artifacts. The authors propose such a mechanism, which delivers high-quality splatted images and has the potential for a very efficient hardware implementation.