International audience

Augmented Reality (AR) has grown from specialised uses to applications for the common public. One of these developments led to Augmented Vision (AV), which enhances vision beyond traditional methods like glasses or contact lenses. This review aims to compare and categorise AV systems according to the paradigms they implement to enhance the users' vision. Additionally, the review examines whether researchers conduct measurements and analysis on the human visual system (HVS) when evaluating their system. Such an overall view will help future researchers position their work on AV. By understanding AV systems' paradigms and approaches, researchers will be well-equipped to identify gaps, explore novel directions, and leverage existing advancements. We searched Scopus, Web of Science, and PubMed databases for publications until February 26, 2025, exploring citations and references for the selected articles to avoid missing out on relevant articles. We then conducted a two-step screening process that involved LLM-assisted screening of the article's abstracts and an in-depth assessment of the article. This review follows the PRISMA statement, reducing bias risk. We selected 113 of 469 articles, as they improved users' visual performance. We defined three main categories: (1) adding light to the incoming light field, (2) modifying the incoming light field, and (3) intersecting approaches. We found three main application areas: (1) task-specific, (2) vision correction, and (3) visual perception enhancement. The most typical application is task-specific. We identified a gap in the literature since just four of the papers we reviewed measured and analysed the accommodation while utilising the device.

Augmented Reality (AR) teleconferencing allows spatially distributed users to interact with each other in 3D through agents in their own physical environments. Existing methods leveraging volumetric capturing and reconstruction can provide a high-fidelity experience but are often too complex and expensive for everyday use. Other solutions target mobile and effortless-to-setup teleconferencing on AR Head Mounted Displays (HMD). They directly transplant the conventional video conferencing onto an AR-HMD platform or use avatars to represent remote participants. However, they can only support either a high fidelity or a high level of co-presence. Moreover, the limited Field of View (FoV) of HMDs could further degrade users' immersive experience. To achieve a balance between fidelity and co-presence, we explore using life-size 2D video-based avatars (video avatars for short) in AR teleconferencing. Specifically, with the potential effect of FoV on users' perception of proximity, we first conducted a pilot study to explore the local-user-centered optimal placement of video avatars in small-group AR conversations. With the placement results, we then implement a proof-of-concept prototype of video-avatar-based teleconferencing. We conduct user evaluations with our prototype to verify its effectiveness in balancing fidelity and co-presence. Following the indication in the pilot study, we further quantitatively explore the effect of FoV size on the video avatar's optimal placement through a user study involving more FoV conditions in a VR-simulated environment. We regress placement models to serve as references for computationally determining video avatar placements in such teleconferencing applications on various existing AR HMDs and future ones with bigger FoVs.

One of the main targets of criticism of head-mounted displays (HMDs) is the field of view (FOV) size, whether in virtual or augmented reality. This limitation is prominent with optical see-through head-mounted displays (OST-HMD), as those with narrow overlay FOV (OFOV) sizes only provide a small window to view virtual objects. We investigated if restricting this OFOV negatively affects a user’s ability to memorize spatial locations in a simulation of a work environment, and consequently, long-term memory transfer to an equivalent scenario in the real world two days later. To find empirical evidence, we conducted a within-subjects experiment with 18 participants performing in three phases with an OST-HMD, simulated on an immersive HMD. For each phase, they viewed the training scenario with a different OFOV size of the augmentable area (30°, 70°, 110° diagonal). Results from recall tests showed that smaller OFOV size did not significantly affect user’s performance on both short-term and transfer tests, but HMD data revealed that users rotated their heads less with a 110° OFOV. We also found that proximity of objects to memorize had an interaction effect with smaller OFOV sizes. Our findings could have implications on the design and HMD choices of augmented training.

Gesture recognition with miniaturised radar sensors has received increasing attention as a novel interaction medium. The practical use of radar technology, however, often requires sensing through materials. Yet, it is still not well understood how the internal structure of materials impacts recognition performance. To tackle this challenge, we collected a large dataset of 14,090 radar recordings for 6 paradigmatic gesture classes sensed through a variety of everyday materials, performed by humans (6 materials) and a robot system (75 materials). Next, we developed a hybrid CNN+LSTM deep learning model and derived a robust indirect method to measure signal distortions, which we used to compile a comprehensive catalogue of materials for radar-based interaction. Among other findings, our experiments show that it is possible to estimate how different materials would affect gesture recognition performance of arbitrary classifiers by selecting just 3 reference materials. Our catalogue, software, models, data collection platform, and labeled datasets are publicly available.

Avatar-mediated symmetric Augmented Reality (AR) telepresence has emerged with the ability to empower users located in different remote spaces to interact with each other in 3D through avatars. However, different spaces have heterogeneous structures and features, which bring difficulties in synchronizing avatar motions with real user motions and adapting avatar motions to local scenes. To overcome these issues, existing methods generate mutual movable spaces or retarget the placement of avatars. However, these methods limit the telepresence experience in a small sub-area space, fix the positions of users and avatars, or adjust the beginning/ending positions of avatars without presenting smooth transitions. Moreover, the delay between the avatar retargeting and users' real transitions can break the semantic synchronization between users' verbal conversation and perceived avatar motion. In this paper, we first examine the impact of the aforementioned transition delay and explore the preferred transition style with the existence of such delay through user studies. With the results showing a significant negative effect of avatar transition delay and providing the design choice of the transition style, we propose a Predict-and-Drive controller to diminish the delay and present the smooth transition of the telepresence avatar. We also introduce a grouping component as an upgrade to immediately calculate a coarse virtual target once the user initiates a transition, which could further eliminate the avatar transition delay. Once having the coarse virtual target or an exactly predicted target, we find the corresponding target for the avatar according to the pre-constructed mapping of objects of interest between two spaces. The avatar control component maintains an artificial potential field of the space and drives the avatar towards the target while respecting the obstacles in the physical environment. We further conduct ablation studies to evaluate the effectiveness of our proposed components.

Accurately placing virtual objects in a scene is a challenging tasks in handheld augmented reality (HAR). To add and arrange virtual objects in HAR, users must manipulate 6 degrees of freedom (DoFs) of the virtual object, namely: position (3) and orientation (3). However, it is difficult to manipulate all DoFs with the two-dimensional display of the handheld device. We present SlidAR+, a method for controlling the position and orientation of objects in HAR. SlidAR+ is an extension of SlidAR [1], a technique that allows users to control the position of a virtual object by manipulating only 1 DoF. We use the direction of gravity as a constraint to improve the user’s control and reduce the time it takes to adjust the orientation. Upon comparing it with a state-of-the-art object manipulation method, using SlidAR+, user were able to complete the tasks faster under our expected conditions and were also preferred by most participants.

We present a novel method that robustly estimates the reflectance, even in an environment with dynamically changing light. To control the appearance of an object by using a projector-camera system, an appropriate estimate of the object's reflectance is vital to the creation of an appropriate projection image. Most conventional estimation methods assume static light conditions; however, in practice, the appearance is affected by both the reflectance and environmental light. In an environment with dynamically changing light, conventional reflectance estimation methods require calibration every time the conditions change. In contrast, our method requires no additional calibration because it simultaneously estimates both the reflectance and environmental light. Our method is based on the concept of creating two different light conditions by switching the projection at a rate higher than that perceived by the human eye and captures the images of a target object separately under each condition. The reflectance and environmental light are then simultaneously estimated by using the pair of images acquired under these two conditions. We implemented a projector-camera system that switches the projection on and off at 120 Hz. Experiments confirm the robustness of our method when changing the environmental light. Further, our method can robustly estimate the reflectance under practical indoor lighting conditions.

With projection mapping, we can control the appearance of real-world objects by adding illumination. A projector can be used to control light reflected from an object, where the reflected light depends not only on the projection but also on the reflectance and environmental light. Because the resulting colors are affected by the reflectance and environmental light, the presentable color range of a projector is limited. The purpose of this work is to broaden this limited range by focusing on the perceived colors. Although our eyes capture reflected light to perceive the colors of an object, the colors perceived by humans are not always the same as the actual colors, and there are often significant differences between them because of the human visual system. To overcome the limitations of a projector based on human perception, we intentionally generate this difference by inducing a visual illusion, namely, color constancy. In this work, we designed an algorithm to determine the projected colors for presenting the desired colors perceptually by employing a color constancy effect. In addition, we conducted a user study and confirmed that our algorithm can (1) create a misperception regarding the color of illumination, (2) broaden the presentable color range of a projector, and (3) shift the perceptual colors in the desirable direction.

Using projection mapping, we can control the appearance of realworld objects by projecting colored light onto them. Since a projector can only add illumination to the scene, limited color gamut can be presented through projection mapping. However, actual color and perceived color are not always the same, and there is often large difference between them. We intentionally generate this difference by inducing visual illusion for extending the controllable color gamut of a projector. In particular, we induce color constancy. We demonstrate changing object color with and without inducing color constancy. Audiences perceive the controlled colors with and without illusion as different color nevertheless these colors are physically completely same.

Level editors let end-users create custom levels and content within a given video game. In this paper, we explore the concept and design of Augmented reality game level editors. These new types of editors are not only spatial and embodied, but also situated, as they enable users to tailor games to the unique characteristics and emotional value of their own space.

Using projection mapping, we can control the appearance of realworld objects by projecting colored light onto them. Because a projector can only add illumination to the scene, only a limited color gamut can be presented through projection mapping. In this paper we describe how the controllable color gamut can be extended by accounting for human perception and visual illusions. In particular, we induce color constancy to control what color space observers will perceive. In this paper, we explain the concept of our approach, and show first results of our system.

Work space simulations help trainees acquire skills necessary to perform their tasks efficiently without disrupting the workflow, forgetting important steps during a procedure, or the location of important information. This training can be conducted in Augmented and Virtual Reality (AR, VR) to enhance its effectiveness and speed. When the skills are transferred to the actual application, it is referred to as positive training transfer. However, thus far, it is unclear which training, AR or VR, achieves better results in terms of positive training transfer. We compare the effectiveness of AR and VR for spatial memory training in a control-room scenario, where users have to memorize the location of buttons and information displays in their surroundings. We conducted a within-subject study with 16 participants and evaluated the impact the training had on short-term and long-term memory. Results of our study show that VR outperformed AR when tested in the same medium after the training. In a memory transfer test conducted two days later AR outperformed VR. Our findings have implications on the design of future training scenarios and applications.

For optical see-through head-mounted displays, the mismatch between a display's focal length and the real world scene inadvertently prevents users from simultaneously focusing on the presented virtual content and the scene. It has been shown that it is possible to ameliorate the out-of-focus blur for images with a known focus distance, by applying an algorithm called Sharp View. However, it remains unclear if Sharp View also improves the readability and clarity of text rendered on the display. In this study, we investigate whether users reported increased text clarity when Sharp View was applied to a text label, and how the focal demand of the display, the focal distance to real world content, and gaze condition affect the result. Our results indicate that, in non-fixated viewing, there is a significant user preference for Sharp View-enhanced text strings.

Inflammation, stiffness, and swelling are frequently reported symptoms of patellar tendinitis among cyclists; making knee pain a consistently observed overuse injury in cycling. In this paper, we investigate the applicability of a knee trajectory visualization to self-analysis for increasing awareness of movement patterns leading to injuries. We briefly explain overuse injuries and patellar instability, describe the experiments we did with cyclists for gathering requirements, and finally illustrate an augmented reality concept. We also show two different types of visualizations with participant opinions; one being conventional and other being a video-based one and discuss how situated visualizations can be utilized for improving self awareness to injury causes.

Optical See-Through Head-Mounted Displays (OST-HMDs) lose the visibility of virtual contents under bright environment illumination due to their see-through nature. We demonstrate how a liquid crystal (LC) filter attached to an OST-HMD can be used to dynamically increase the perceived brightness of virtual content without impacting the perceived brightness of the real scene. We present a prototype OST-HMD that continuously adjusts the opacity of the LC filter to attenuate the environment light without users becoming aware of the change. Consequently, virtual content appears to be brighter. The proposed approach is evaluated in psychophysical experiments in three scenes, with 16, 31, and 31 participants, respectively. The participants were asked to compare the magnitude of brightness changes of both real and virtual objects, before and after dimming the LC filter over a period of 5, 10, and 20 seconds. The results showed that the participants felt increases in the brightness of virtual objects while they were less conscious of reductions of the real scene luminance. These results provide evidence for the effectiveness of our display design. Our design can be applied to a wide range of OST-HMDs to improve the brightness and hence realism of virtual content in augmented reality applications.

Virtual Reality (VR) Head-Mounted Displays (HMDs) are on the verge of becoming commodity hardware available to the average user and feasible to use as a tool for 3D work. Some HMDs include front-facing cameras, enabling Augmented Reality (AR) functionality. Apart from avoiding collisions with the environment, interaction with virtual objects may also be affected by seeing the real environment. However, whether these effects are positive or negative has not yet been studied extensively. For most tasks it is unknown whether AR has any advantage over VR. In this work we present the results of a user study in which we compared user performance measured in task completion time on a 9 degrees of freedom object selection and transformation task performed either in AR or VR, both with a 3D input device and a mouse. Our results show faster task completion time in AR over VR. When using a 3D input device, a purely VR environment increased task completion time by 22.5 percent on average compared to AR ( ). Surprisingly, a similar effect occurred when using a mouse: users were about 17.3 percent slower in VR than in AR ( ). Mouse and 3D input device produced similar task completion times in each condition (AR or VR) respectively. We further found no differences in reported comfort.

Inspection tasks focus on observation of the environment and are required in many industrial domains. Inspectors usually execute these tasks by using a guide such as a paper manual, and directly observing the environment. The effort required to match the information in a guide with the information in an environment and the constant gaze shifts required between the two can severely lower the work efficiency of inspector in performing his/her tasks. Augmented reality (AR) allows the information in a guide to be overlaid directly on an environment. This can decrease the amount of effort required for information matching, thus increasing work efficiency. AR guides on head-mounted displays (HMDs) have been shown to increase efficiency. Handheld AR (HAR) is not as efficient as HMD-AR in terms of manipulability, but is more practical and features better information input and sharing capabilities. In this study, we compared two handheld guides: an AR interface that shows 3D registered annotations, that is, annotations having a fixed 3D position in the AR environment, and a non-AR picture interface that displays non-registered annotations on static images. We focused on inspection tasks that involve high information density and require the user to move, as well as to perform several viewpoint alignments. The results of our comparative evaluation showed that use of the AR interface resulted in lower task completion times, fewer errors, fewer gaze shifts, and a lower subjective workload. We are the first to present findings of a comparative study of an HAR and a picture interface when used in tasks that require the user to move and execute viewpoint alignments, focusing only on direct observation. Our findings can be useful for AR practitioners and psychology researchers.

Left-right confusion occurs across the entire population and refers to an impeded ability to distinguish between left and right. In medicine this phenomenon is particularly relevant as left and right are always defined with respect to the patient's point of view, i.e. the doctor's right is the patient's left. Traditional anatomy learning resources such as illustrations in textbooks naturally consider this by consistently depicting the anatomy of a patient as seen by an observer standing in front. Augmented Reality Magic Mirrors (MM) are one example of novel anatomy teaching resources and show a user's digital mirror image augmented with virtual anatomy on a large display. As left and right appear to be reversed in such MM setups, similar to real-world physical mirrors, intriguing perceptual questions arise: is a non-reversing MM (NRMM) the more natural choice for the task of anatomy learning and do users even learn anatomy the wrong way with a traditional, reversing MM (RMM)? In this paper, we explore the perceptual differences between an NRMM and RMM design and present the first empirical study comparing these two concepts for the purpose of anatomy learning. Experimental results demonstrate that medical students perform significantly better at identifying anatomically correct placement of virtual organs in an NRMM. However, interaction was significantly more difficult compared to an RMM. We explore the underlying psychological effects and discuss the implications of using an NRMM on user perception, knowledge transfer, and interaction. This study is relevant for the design of future MM systems in the medical domain and lessons-learned can be transferred to other application domains.

Does visually perceived distance differ when objects are viewed in augmented reality (AR), as opposed to the real world? What are the differences? These questions are theoretically interesting, and the answers are important for the development of many tablet- and phone-based AR applications, including mobile AR navigation systems. This article presents a thorough literature review of distance judgment experimental protocols, and results from several areas of perceptual psychology. In addition to distance judgments of real and virtual objects, this section also discusses previous work in measuring the geometry of virtual picture space and considers how this work might be relevant to tablet AR. Then, the article presents the results of two experiments. In each experiment, observers bisected egocentric distances of 15 and 30 m in tablet-based AR and in the real world, in both indoor corridor and outdoor field environments. In AR, observers bisected the distances to virtual humans, while in the real world, they bisected the distances to real humans. This is the first reported research that directly compares distance judgments of real and virtual objects in a tablet AR system. Four key findings were: (1) In AR, observers expanded midpoint intervals at 15 m, but compressed midpoints at 30 m. (2) Observers were accurate in the real world. (3) The environmental setting—corridor or open field—had no effect. (4) The picture perception literature is important in understanding how distances are likely judged in tablet-based AR. Taken together, these findings suggest the depth distortions that AR application developers should expect with mobile and especially tablet-based AR.

Entertainment is challenging to observe, especially with children, due to limited analytical tools. In response, we present a modified framework for entertainment computing, COMPASS -- COmbined Mental, PhysicAl, Social and Spatial factors, which we use to analyze augmented reality activities for cub scouts.

Over the last decades, visual representations of data has been a commonly used medium to bolster human cognition in performance evaluation of professional athletes. However, the current approaches to these visualizations still build upon the paper based principles of initial designs with solid backgrounds. Due to this situation, same visualizations usually fail to provide explicit information about the physical characteristics of the scenario that the data was captured, such as the form of athletes. In this work, we present a data visualization method which combines visual representations of cyclist's pedaling with correlated frames of indoor training videos. We designed a prototype system which allows us to superimpose various pedaling visualizations onto simultaneously captured training videos of cyclists. The results of user studies we conducted with twelve professional cyclists confirmed their interest in new possibilities emerging from intuitive data visualizations. We also received valuable feedback about the feasible benefits of our approach over traditional approaches, such as reduced cognitive overload in understanding visualizations. We conclude by discussing the future implementations and application areas of our approach and further need of adjusting it to distinct training scenarios.

Co-located haptic feedback in mixed and augmented reality environments can improve realism and user performance, but it also requires careful system design and calibration. In this poster, we determine the thresholds for perceiving co-location errors through two psychophysics experiments in a typical fine-motor manipulation task. In these experiments we simulate the two fundamental ways of implementing VHAR systems: first, attaching a real tool; second, augmenting a virtual tool. We determined the just-noticeable co-location errors for position and orientation in both experiments and found that users are significantly more sensitive to co-location errors with virtual tools. Our overall findings are useful for designing visuo-haptic augmented reality workspaces and calibration procedures.

A common factor among current generation optical see-through augmented reality systems is fixed focal distance to virtual content. In this work, we investigate the issue of focus blur, in particular, the blurring caused by simultaneously viewing virtual content and physical objects in the environment at differing focal distances. We examine the application of dynamic sharpening filters as a straight forward, system independent, means for mitigating this effect improving the clarity of defocused AR content. We assess the utility of this method, termed SharpView, by employing an adjustment experiment in which users actively apply varying amounts of sharpening to reduce the perception of blur in AR content. Our experimental results validate the ability of our SharpView model to improve the visual clarity of focus blurred content, with optimal performance at focal differences well suited for near field AR applications.

Handheld Augmented Reality (HAR) has the potential to introduce Augmented Reality (AR) to large audiences due to the widespread use of suitable handheld devices. However, many of the current HAR systems are not considered very practical and they do not fully answer to the needs of the users. One of the challenging areas in HAR is the in-situ AR content creation where the correct and accurate positioning of virtual objects to the real world is fundamental. Due to the hardware limitations of handheld devices and possible restrictions in the environment, the correct 3D positioning of objects can be difficult to achieve we are unable to use AR markers or correctly map the 3D structure of the environment. We present SlidAR, a 3D positioning for Simultaneous Localization And Mapping (SLAM) based HAR systems. SlidAR utilizes 3D ray-casting and epipolar geometry for virtual object positioning. It does not require a perfect 3D reconstruction of the environment nor any virtual depth cues. We have conducted a user experiment to evaluate the efficiency of SlidAR method against an existing device-centric positioning method that we call HoldAR. Results showed that SlidAR was significantly faster, required significantly less device movement, and also got significantly better subjective evaluation from the test participants. SlidAR also had higher positioning accuracy, although not significantly.

Projection-based Augmented Reality commonly projects on rigid objects, while only few systems project on deformable objects. In this paper, we present Pseudo Printed Fabrics (PPF), which enables the projection on a deforming piece of cloth. This can be applied to previewing a cloth design while manipulating its shape. We support challenging manipulations, including heavy occlusions and stretching the cloth. In previous work, we developed a similar system, based on a novel marker pattern; PPF extends it in two important aspects. First, we improved performance by two orders of magnitudes to achieve interactive performance. Second, we developed a new interpolation algorithm to keep registration during challenging manipulations. We believe that PPF can be applied to domains including virtual-try on and fashion design.

Usability evaluations are important to the development of augmented reality systems. However, conducting large-scale longitudinal studies remains challenging because of the lack of inexpensive but appropriate methods. In response, we propose a method for implicitly estimating usability ratings based on readily available sensor logs. To demonstrate our idea, we explored the use of features of accelerometer data in estimating usability ratings in an annotation task. Results show that our implicit method corresponds with explicit usability ratings at 79% and 84%. These results should be investigated further in other use cases, with other sensor logs.

Visuo-haptic augmented reality systems enable users to see and touch digital information that is embedded in the real world. PHANToM haptic devices are often employed to provide haptic feedback. Precise co-location of computer-generated graphics and the haptic stylus is necessary to provide a realistic user experience. Previous work has focused on calibration procedures that compensate the non-linear position error caused by inaccuracies in the joint angle sensors. In this article we present a more complete procedure that additionally compensates for errors in the gimbal sensors and improves position calibration. The proposed procedure further includes software-based temporal alignment of sensor data and a method for the estimation of a reference for position calibration, resulting in increased robustness against haptic device initialization and external tracker noise. We designed our procedure to require minimal user input to maximize usability. We conducted an extensive evaluation with two different PHANToMs, two different optical trackers, and a mechanical tracker. Compared to state-of-the-art calibration procedures, our approach significantly improves the co-location of the haptic stylus. This results in higher fidelity visual and haptic augmentations, which are crucial for fine-motor tasks in areas such as medical training simulators, assembly planning tools, or rapid prototyping applications.

The IEEE International Symposium on Mixed and Augmented Reality (ISMAR) is the leading venue for publishing the latest Mixed and Augmented Reality research, applications, and technologies. This special section presents significantly extended versions of the five best papers from the IEEE ISMAR 2014 proceedings. Within the past few years, Augmented Reality (AR) has reached a critical mass in both research and commercial applications. It is now becoming truly feasible to use augmented reality to place graphics anywhere at any time. However, although the basic capabilities exist, many open research problems continue. This collection of papers considers underlying issues and technologies. IEEE ISMAR 2014 had 89 paper submissions; each paper was reviewed by at least four experts in the field. An international programcommittee of 15 ARexperts invited reviewers, led discussions, invited a rebuttal by the paper authors and prepared a consensus review. To select the final papers for publication, an online two-day PC meeting was held connecting three continents, where each paper was discussed. Thirty-five papers were accepted either as long or short publications, giving an overall acceptance rate of 40%. An independent Award Committee reviewed the highest- ranked submissions again to determine the awards for Best Paper and Honorable Mention. For this special section, the authors of the award papers were invited to submit an extended version of their conference papers, with a clear focus on additional content that expands the scientific contribution of the original conference paper. A standard TVCG reviewing cycle was initiated in which all papers were reviewed, feedback was provided, and papers were revised to suit. Out of all submitted papers, less than 6% appear in this TVCG Special Section.

With the growing availability of optical see-through (OST) head-mounted displays (HMDs) there is a present need for robust, uncomplicated, and automatic calibration methods suited for non-expert users. This work presents the results of a user study which both objectively and subjectively examines registration accuracy produced by three OST HMD calibration methods: (1) SPAAM, (2) Degraded SPAAM, and (3) Recycled INDICA, a recently developed semi-automatic calibration method. Accuracy metrics used for evaluation include subject provided quality values and error between perceived and absolute registration coordinates. Our results show all three calibration methods produce very accurate registration in the horizontal direction but caused subjects to perceive the distance of virtual objects to be closer than intended. Surprisingly, the semi-automatic calibration method produced more accurate registration vertically and in perceived object distance overall. User assessed quality values were also the highest for Recycled INDICA, particularly when objects were shown at distance. The results of this study confirm that Recycled INDICA is capable of producing equal or superior on-screen registration compared to common OST HMD calibration methods. We also identify a potential hazard in using reprojection error as a quantitative analysis technique to predict registration accuracy. We conclude with discussing the further need for examining INDICA calibration in binocular HMD systems, and the present possibility for creation of a closed-loop continuous calibration method for OST Augmented Reality.

Augmented reality (AR) is an emerging technology for communicating learning contents. Several AR systems are designed for learning. However, studies that have investigated instructional strategies for applying AR are few. This investigation requires the implementation of prototypes that use state-of-the-art technology and sound learning theory. In this work, we implemented two prototypes for learning Filipino and German words by first developing a handheld AR platform. These prototypes demonstrate situated vocabulary learning. Using our AR system, students can learn words related to their current environment. We assessed the quality of these prototypes by conducting usability evaluations. For the theoretical grounding, we leveraged on multimedia learning theory to design the content. Through our handheld AR platform, we evaluated situated vocabulary learning by comparing our prototypes to a flash cards application. In the first evaluation, students scored significantly lower when using AR in an immediate post-test. However, this difference disappeared after taking into account the variability in usability scores via analysis of covariance. Taking account usability is fairer when comparing an emerging technology to traditional technology. Test scores were also not significantly different in a delayed post-test. In the second evaluation, although the post-test score and answering time of students did not differ, our results showed that they feel more satisfied and can keep their attention better when using AR. For the first time, we demonstrated situated vocabulary learning by using AR. Moreover, our preliminary study confirms the intuition that students can achieve the same score using AR, but with benefits such as ease in maintaining attention and increased satisfaction.

Augmented reality (AR) is an enabling technology for presenting information in relation to real objects or real environments. AR is situated multimedia or information that is positioned in authentic physical contexts. In this paper, we discuss how we address issues in creating AR content for educational settings. From the learning theory perspective, we explain that AR is a logical extension of multimedia learning theory. From the development perspective, we demonstrate how AR content can be created through our in situ authoring tool and our platform for handheld AR.

Visuo-haptic augmented reality systems enable users to see and touch digital information that is embedded in the real world. Precise co-location of computer graphics and the haptic stylus is necessary to provide a realistic user experience. PHANToM haptic devices are often used in such systems to provide haptic feedback. They consist of two interlinked joints, whose angles define the position of the haptic stylus and three sensors at the gimbal to sense its orientation. Previous work has focused on calibration procedures that align the haptic workspace within a global reference coordinate system and developing algorithms that compensate the non-linear position error, caused by inaccuracies in the joint angle sensors. In this paper, we present an improved workspace calibration that additionally compensates for errors in the gimbal sensors. This enables us to also align the orientation of the haptic stylus with high precision. To reduce the required time for calibration and to increase the sampling coverage, we utilize time-delay estimation to temporally align external sensor readings. This enables users to continuously move the haptic stylus during the calibration process, as opposed to commonly used point and hold processes. We conducted an evaluation of the calibration procedure for visuo-haptic augmented reality setups with two different PHANToMs and two different optical trackers. Our results show a significant improvement of orientation alignment for both setups over the previous state of the art calibration procedure. Improved position and orientation accuracy results in higher fidelity visual and haptic augmentations, which is crucial for fine-motor tasks in areas including medical training simulators, assembly planning tools, or rapid prototyping applications. A user friendly calibration procedure is essential for real-world applications of VHAR.

Visuo-haptic augmented reality systems enable users to see and touch digital information that is embedded in the real world. Precise colocation of computer graphics and the haptic stylus is necessary to provide a realistic user experience. PHANToM haptic devices are often used in such systems to provide haptic feedback. They consist of two interlinked joints, whose angles define the position of the haptic stylus and three sensors at the gimbal to sense its orientation. Previous work has focused on a calibration procedures that align the haptic workspace within a global reference coordinate system and an algorithms that compensate the non-linear position error, which is caused by inaccuracies in the joint angle sensors. In our science and technology paper “Comprehensive Workspace Calibration for Visuo-Haptic Augmented Reality” [1], we present an improved workspace calibration that additionally compensates for errors in the gimbal sensors. This enables us to also align the orientation of the haptic stylus with high precision. To reduce the required time for calibration and to increase the sampling coverage, we utilize time-delay estimation to temporally align external sensor readings. This enables users to continuously move the haptic stylus during the calibration process, as opposed to commonly used point and hold processes. This demonstration showcases the complete workspace calibration procedure as described in our paper including a mixed reality demo scenario, that allows users to experience the calibrated workspace. Additionally, we demonstrate an early stage of our proposed future work in improved user guidance during the calibration procedure using visual guides.

The idea of using Augmented Reality (AR) user interfaces (UIs) to create 3D media content, such as 3D models for movies and games has been repeatedly suggested over the last decade. Even though the concept is intuitively compelling and recent technological advances have made such an application increasingly feasible, very little progress has been made towards an actual real-world application of AR in professional media production. To this day, no immersive 3D UI has been commonly used by professionals for 3D computer graphics (CG) content creation. In this paper, we are first to publish a requirements analysis for our target application in the professional domain. Based on a survey that we conducted with media professionals, the analysis of professional 3D CG software, and professional training tutorials, we identify these requirements and put them into the context of AR UIs. From these findings, we derive several interaction design principles that aim to address the challenges of real-world application of AR to the production pipeline. We implemented these in our own prototype system while receiving feedback from media professionals. The insights gained in the survey, requirements analysis, and user interface design are relevant for research and development aimed at creating production methods for 3D media production.

For this demo, we present an Augmented Reality (AR) User Interface (UI) for the 3D design software Autodesk Maya, aimed at professional media creation. A user wears a head-mounted display (HMD) and thin cotton gloves which allow him to interact with virtual 3D models in the work area. Additional viewers can see the video stream on a projector and thus share the users view. Both head and hand positions are tracked from the HMD video stream, and an inertial measurement unit (IMU) and conductive materials on the gloves allow interaction with virtual objects. This system is built using Autodesk Maya — a professional 3D software package commonly used in the media industry — and aims to fulfill the requirements of professional 3D design work which we identified in our paper of the same title. While still an early prototype, it was already tested with media professionals to evaluate our approach.

The goal of our work is to create highly realistic graphics for Augmented Reality on mobile phones. One of the greatest challenges for this is to provide realistic lighting of the virtual objects that matches the real world lighting. This becomes even more difficult with the limited capabilities of mobile phone GPUs. Our approach differs in the following important aspects compared to previous attempts: (1) most have relied on rasterizer approaches, while our approach is based on raytracing; (2) we perform distributed rendering in order to address the limited mobile GPU capabilities; (3) we use image-based lighting from a pre-captured panorama to incorporate real world lighting. We utilize two markers: one for object tracking and one for registering the panorama. Our initial results are encouraging, as the visual quality resembles real objects and also the reference renderings which were created offline. However, we still need to validate our approach in human subject studies, especially with regards to the trade-off between latency of remote rendering and visual quality.

Handheld augmented reality (HAR) applications must be carefully designed and improved based on user feedback to sustain commercial use. However, no standard questionnaire considers perceptual and ergonomic issues found in HAR. We address this issue by creating a HAR Usability Scale (HARUS).

Current augmented reality (AR) systems often fail to indicate the distance between the user and points of interest in the environment. Empirical evaluations of human depth perception in AR settings compared to real world settings are needed. Our goal in this study was to understand tablet-based AR depth perception by comparing it with real-world depth perception.

During the last decade, Visuo-Haptic Augmented Reality (VHAR) systems have emerged that enable users to see and touch digital information that is embedded in the real world. They pose unique problems to developers, including the need for precise augmentations, accurate colocation of haptic devices, and efficient concurrent processing of multiple, realtime sensor inputs to achieve low latency. We think that this complexity is one of the main reasons, why VHAR technology has only been used in few user interface research projects. The proposed project's main objective is to pioneer the development of a widely applicable VHAR runtime environment, which meets the requirements of realtime, low latency operation with precise co-location, haptic interaction with deformable bodies, and realistic rendering, while reducing the overall cost and complexity for developers. A further objective is to evaluate the benefits of VHAR user interfaces with a focus on medical training applications, so that creators of future medical simulators or other haptic applications recognize the potential of VHAR.

In the last decade, Augmented Reality has become more mature and is widely adopted on mobile devices. Exploring the available information of a user's environment is one of the key applications. However, current mobile Augmented Reality interfaces are very limited compared to the recently emerging big data exploration tools for desktop computers. Our vision is to bring powerful Visual Analytic tools to mobile Augmented Reality.

Education of anatomy is a challenging but crucial element in educating medical professionals, but also for general education of pupils. Our research group has previously developed a prototype of an Augmented Reality (AR) magic mirror which allows intuitive visualization of realistic anatomical information on the user. However, the current overlay is imprecise as the magic mirror depends on the skeleton output from Kinect. These imprecisions affect the quality of education and learning. Hence, together with clinicians we have defined bone landmarks which users can touch easily on their body while standing in front of the sensor. We demonstrate that these landmarks allow the proper deformation of medical data within the magic mirror and onto the human body, resulting in a more precise augmentation.

Visuo-Haptic Augmented Reality (VHAR) is a technology, which enables users to see and touch virtual objects. It poses unique problems to developers, including the need for precise augmentations, accurate colocation of haptic devices, and efficient processing of multiple, realtime sensor inputs to achieve low latency. We have designed and implemented the software framework HARP, which addresses these issues and simplifies the creation of haptic-enabled Augmented Reality (AR) applications. It allows developers to compose their applications on a high level of abstraction and hides most of the complexity of VHAR. Our contribution is the initial design and implementation of the framework, which we have validated in nine VHAR research projects ranging from simple interaction design to psychophysical experiments. We discuss limitations of our approach and future developments. These insights will be helpful to researchers and framework designers in the field of VHAR.

Augmented Reality systems that run interactively and in real time, using high quality graphical displays and sensational cues, can create the illusion of virtual objects appearing to be real. This paper presents the design, implementation, and evaluation of BurnAR, a novel demonstration which enables users to experience the illusion of seeing their own hands burning, which we achieve by overlaying virtual flames and smoke on their hands. Surprisingly, some users reported an involuntary warming sensation of their hands. Based on these comments, we hypothesized that stimulation of multiple sensory modalities presented in this AR environment can induce an involuntary experience in an additional sensory pathway: observation of virtual flames resulting in a heat sensation. This cross-modal transfer, known as virtual synesthe-sia, is a temporary experience which affects some people who are not synesthetes and only lasts for a short time during the illusory experience. To verify our hypothesis, we conducted an exploratory study where participants experienced the BurnAR demonstration under controlled conditions.

ClonAR enables users to rapidly clone and edit real-world objects. First, real-world objects can be scanned using KinectFusion. Second, users can edit the scanned objects in our visuo-haptic Augmented Reality environment. Our whole pipeline does not use mesh representation of objects, but rather Signed Distance Fields, which are the output of KinectFusion. We directly render Signed Distance Fields haptically and visually. We do direct haptic rendering of Signed Distance Fields, which is faster and more flexible than rendering meshes. Visual rendering is performed by our custom-built raymarcher, which facilitates realistic illumination effects like ambient occlusions and soft shadows. Our prototype demonstrates the whole pipeline. We further present several results of redesigned real-world objects.

Augmented Reality (AR) applications on mobile devices like smartphones and tablet computers have become increasingly popular. In this paper, for the first time in the AR domain, we present: (1) the influence of different handheld displays and (2) the exocentric depth perception. Unlike egocentric depth perception, exocentric depth perception has not been investigated in AR.

Augmented Reality systems that run interactively and in real time, using high quality graphical displays and sensational cues, can create the illusion of virtual objects appearing to be real. This paper presents the design and implementation of BurnAR, a demonstration which enables users to experience the illusion of seeing their own hands burning, which we achieve by overlaying virtual flames and smoke on their hands. Surprisingly, some users reported an involuntary warming sensation of their hands.

In Virtual Environments (VE), users are often facing tasks that involve direct manipulation of virtual objects at close distances, such as touching, grabbing, placement. In immersive systems that employ head-mounted displays these tasks could be quite challenging, due to lack of convergence of virtual cameras.

Reliable and unobtrusive eye tracking remains a technical challenge for immersive virtual environment, especially when Head Mounted Displays (HMD) are used for visualization and users are allowed to move freely in the environment. In this work, we provide experimental evidence that gaze direction can be safely approximated by user head rotation, in HMD-based Virtual Reality (VR) applications, where users actively interact with the environment. We discuss the application range of our approach and consider possible extensions.

The calibration of optical see-through head-mounted displays is an important fundament for correct object alignment in augmented reality. Any calibration process for OSTHMDs requires users to align 2D points in screen space with 3D points in the real world and to confirm each alignment. In this poster, we present the results of our empiric evaluation where we compared four confirmation methods: Keyboard, Hand-held, Voice, and Waiting. The Waiting method, designed to reduce head motion during confirmation, showed a significantly higher accuracy than all other methods. Averaging over a time frame for sampling user input before the time of confirmation improved the accuracy of all methods in addition. We conducted a further expert study proving that the results achieved with a video see-through head-mounted display showed valid for optical see-through head-mounted display calibration, too.

Mixed and Augmented Reality displays extend the user's perception with computer generated information. This information is typically registered in three-dimensional space, and related to objects and places in the physical world. While individual annotation of objects has historically been a topic of MR research, visualization incorporating multiple related data points or models provides a variety of new research challenges in systems and techniques. For example, photorealistic augmented reality visualization presents data by adapting additionally presented imagery to the real world condition while illustrative visualization techniques aim at enhancing the understanding of augmented scenarios by carefully combining and mediating real and virtual data. Situated visualization techniques present virtual representations of data in relevant locations in the physical scene. A challenge in many of these techniques is the need to correctly communicate the relationships between physical imagery and virtual data.

In the past, several systems have been presented that enable users to view occluded points of interest using Augmented Reality X-ray visualizations. It is challenging to design a visualization that provides correct occlusions between occluder and occluded objects while maximizing legibility. We have previously published an Augmented Reality X-ray visualization that renders edges of the occluder region over the occluded region to facilitate correct occlusions while providing foreground context. While this approach is simple and works in a wide range of situations, it provides only minimal context of the occluder object. In this paper, we present the background, design, and implementation of our novel visualization technique that aims at providing users with richer context of the occluder object. While our previous visualization only employed one salient feature (edges) to determine which parts of the occluder to display, our novel visualization technique is an initial attempt to explore the design space of employing multiple salient features for this task. The prototype presented in this paper employs three additional salient features: hue, luminosity, and motion. We have conducted two evaluations with human participants to investigate the benefits and limitations of our prototype compared to our previous system. The first evaluation showed that although our novel visualization provides a richer context of the occluder object, it does not impede users to select objects in the occluded area; but, it also indicated problems in our prototype. In the second evaluation, we have investigated these problems through an online survey with systematically varied occluder and occluded scenes, focussing on the qualitative aspects of our visualizations. The results were encouraging, but pointed out that our novel visualization needs a higher level of adaptiveness.

Most of today's mobile internet devices contain facilities to display maps of the user's surroundings with points of interest embedded into the map. Other researchers have already explored complementary, egocentric visualizations of these points of interest using mobile mixed reality. Being able to perceive the point of interest in detail within the user's current context is desirable, however, it is challenging to display off-screen or occluded points of interest. We have designed and implemented space-distorting visualizations to address these situations. While this class of visualizations has been extensively studied in information visualization, we are not aware of any attempts to apply them to augmented or mixed reality. Based on the informal user feedback that we have gathered, we have performed several iterations on our visualizations. We hope that our initial results can inspire other researchers to also investigate space-distorting visualizations for mixed and augmented reality.

Enabling users to accurately perceive the correct depth of occluded objects is one of the major challenges in user interfaces for Mixed Reality (MR). Therefore, several visualization techniques and user evaluations for this area have been published. Our research is fo-cused on photorealistic X-ray type visualizations in outdoor envi-ronments. In this paper, we present an evaluation of depth percep-tion in far-field distances through two photorealistic visualizations of occluded objects (X-ray and Melt) in the presence and absence of a depth cue. Our results show that the distance to occluded ob-jects was underestimated in all tested conditions. This finding is curious, as it contradicts previously published results of other re-searchers. The Melt visualization coupled with a depth cue was the most accurate among all the experimental conditions.

This paper presents a new user interface methodology for Spatial Augmented Reality systems. The methodology is based on a set of physical tools that are overloaded with logical functions. Visual feedback presents the logical mode of the tool to the user by projecting graphics onto the physical tools. This approach makes the tools malleable in their functionality, with this change conveyed to the user by changing the projected information. Our prototype application implements a two handed technique allowing an industrial designer to digitally airbrush onto an augmented physical model, masking the paint using a virtualized stencil.

Throughout the last decade, mobile information browsing has become a widely-adopted practice. Most of today's mobile Internet devices contain facilities to display maps of the user's surroundings with points of interest embedded into the map. Other researchers have already explored complementary, egocentric visualizations of these points of interest using mobile mixed reality. However, it is challenging to display off-screen or occluded points of interest. We have designed and implemented space-distorting visualizations to address these situations. Based on the informal user feedback that we have gathered, we have performed several iterations on our visualizations. We hope that our initial results can inspire other researchers to also investigate space-distorting visualizations for mixed and augmented reality.

Augmented reality x-ray vision allows users to see through walls and view real occluded objects and locations. We present an augmented reality x-ray vision system that employs multiple view modes to support new visualizations that provide depth cues and spatial awareness to users. The edge overlay visualization provides depth cues to make hidden objects appear to be behind walls, rather than floating in front of them. Utilizing this edge overlay, the tunnel cut-out visualization provides details about occluding layers between the user and remote location. Inherent limitations of these visualizations are addressed by our addition of view modes allowing the user to obtain additional detail by zooming in, or an overview of the environment via an overhead exocentric view.

User interfaces for ubiquitous augmented reality incorporate a wide variety of concepts such as multi-modal, multi-user, multi-device aspects and new input/output devices. In this paper we present a twofold approach that consists of an execution engine for ubiquitous augmented reality user interfaces and a runtime development environment that enables rapid prototyping and live system adaption for such advanced user interfaces.

Information in hybrid user interfaces can be spread over a variety of different, but complementary, displays, with which users interact through a potentially equally varied range of interaction devices. Since the exact configuration of these displays and devices may not be known in advance, it is desirable for users to be able to reconfigure at runtime the dataflow between interaction devices and objects on the displays. To make this possible, we present the design and implementation of a prototype mixed reality system that allows users to immersively reconfigure a running hybrid user interface.

With recent advances of head-up display technology in cars, augmented reality becomes interesting in supporting the driving task to guide a driver's attention. We have set up an experiment to compare two different approaches to inform the driver about dangerous situations around the car. One approach used AR to visualize the source of danger in the driver's frame of reference while the other one presented information in an egocentric frame of reference. Both approaches were evaluated in user tests.

We present a workbench to build, evaluate and iteratively develop user interfaces using augmented reality, eye tracking, and visual programming. To test our system, we have developed an attentive user interface (AUI) for automotive environments, which coordinates its activities based on the context and visual attention of the user. Development of AUI requires interdisciplinary teams like psychologists, human-factor engineers, designers and computer scientists to work together. The main problem of interdisciplinary communication is a lack of common language and different notion of the system. We have developed a workbench, which facilitates the communication between the team members and enhances the comprehension of the system by visualizing users' attention and system reactions.

In the past, architectures of augmented reality systems have been widely different and tailored to specific tasks. In this paper, we use the example of the SHEEP game to show how the structural flexibility of DWARF, our component-based distributed wearable augmented reality framework, facilitates a rapid prototyping and online development process for building, debugging and altering a complex, distributed, highly interactive AR system. The SHEEP system was designed to test and demonstrate the potential of tangible user interfaces which dynamically visualize, manipulate and control complex operations of many inter-dependent processes. SHEEP allows the users more freedom of action and forms of interaction and collaboration, following the tool metaphor that bundles software with hardware in units that are easily understandable to the user. We describe how we developed SHEEP, showing the combined evolution of framework and application, as well as the progress from rapid prototype to final demonstration system. The dynamic aspects of DWARF facilitated testing and allowed us to rapidly evaluate new technologies. SHEEP has been shown successfully at various occasions. We describe our experiences with these demos.