Pacing diagrams are used in game design practice to represent player experience as a temporal structure. However, the term currently refers to heterogeneous and informal artifacts, limiting both interdisciplinary communication and computational use. We propose a minimal formal reference description of pacing diagrams for linear player experience sequences. The contribution defines the core structural elements of the diagram and frames pacing diagrams as both communicative design artifacts and structured representations. This formalization provides a shared point of reference for design practice while enabling consistent interpretation by tools and computational systems. As a first step, it establishes a foundation for more comparable, interoperable, and analyzable representations of player experience in game design.

Central to a game is the player experience, a relation characterized by the indirect nature of how the experience emerges through play. This second-order design problem of games introduces a significant challenge for the game design engineering process. In this paper, we propose a model of the game design engineering process emphasizing the player experience as the primary concern. By facilitating a comparison between a system as-it-is and a system as-it-should-be, this approach has the potential to shape future endeavors for planning, validating, and refining player experience in the context of design engineering.

This paper investigates how the car's systems can be integrated into entertainment experiences. Our experiment uses the vehicle's Vertical Dynamics (VD) actuators and interior light of a BMW i7. We aim to improve the immersion of players by integrating the automotive active suspension system into several game experiences. It is used to provide additional haptic feedback to players. In addition, we also use multi-color interior lighting to provide visual cues. We conducted a user study consisting of custom questions and the Player Experience Inventory (PXI) questionnaire. Users liked the experience and felt the haptic feedback correlated with their actions in-game. 16% of users experienced motion sickness when using the system. However, most of them to a mild extent. From the PXI, we could tell that the experience performed on par with the benchmark in regards to immersion. But the games need to be reworked by incorporating progressive feedback. We were able to show that a vehicle's VD actuators can be used to provide noticeable and in-synch haptic feedback. The average latency of around 120 ms of our setup proved to be a bottleneck, which needs to be addressed in future work.

Rehabilitation for patients with muscle dystrophy is based on performing a lifelong program of regular exercise in coordination with the therapist. However, traditional physiotherapy can be repetitive and boring, which can lead to frustration. In recent years, Serious Games for Exercise Rehabilitation (SGER) have shown promise in making these repetitive tasks more engaging. This paper targets how Virtual Reality (VR) and Augmented Reality (AR) can be used to create fun and motivating SGER. The development process was supported by a rehabilitation team in a neuromuscular research-clinical center responsible for muscle dystrophy treatment. Therefore, medical knowledge comes from clinicians, including a doctor, with the goal of designing immersive environments that encourage therapeutic movements by playing. Based on VR and AR, 15 mini-games were designed to replace traditional rehabilitation exercises, focusing on improving upper limb movement and adaptability to different types of muscle dystrophy. These games were tested with 15 patients in the presence of a doctor to acquire feedback. The results show that the patients were satisfied with the games specifically chosen for their type of disease. With future improvements, those games can be improved and integrated into the therapy plan. This work provides a foundation for the use of game-based systems in realworld rehabilitation.

This paper presents a time- and space-efficient adaptation of the Space Foundation System, a novel game engine system that abstracts space into semantic locations to bridge the gap between the implementation domain and the problem domain of spatial features in digital games. The presented algorithm discretizes the game space by employing a frontier-based flood fill on a voxel grid to partition the space into locations defined by anchors bounded by delimiters. Integrated into a game engine via extended Transform components, the system enables location queries and supports advanced spatial game features in real time. An evaluation across diverse game environments demonstrates that the algorithm meets practical development requirements regarding computation time and memory usage. The results suggest that the system offers a viable, design-oriented extension to traditional scene graphs.

In mixed reality (MR) telepresence applications, the differences between participants' physical environments can interfere with effective collaboration. For asymmetric tasks, users might need to access different resources (information, objects, tools) distributed throughout their room. Existing intersection methods do not support such interactions, because a large portion of the telepresence participants' rooms become inaccessible, along with the relevant task resources. We propose MRUnion, a Mixed Reality Telepresence pipeline for asymmetric task-aware 3D mutual scene generation. The key concept of our approach is to enable a user in an asymmetric telecollaboration scenario to access the entire room, while still being able to communicate with remote users in a shared space. For this purpose, we introduce a novel mutual room layout called Union. We evaluated 882 space combinations quantitatively involving two, three, and four combined remote spaces and compared it to a conventional Intersect room layout. The results show that our method outperforms existing intersection methods and enables a significant increase in space and accessibility to resources within the shared space. In an exploratory user study (N=24), we investigated the applicability of the synthetic mutual scene in both MR and VR setups, where users collaborated on an asymmetric remote assembly task. The study results showed that our method achieved comparable results to the intersect method but requires further investigation in terms of social presence, safety and support of collaboration. From this study, we derived design implications for synthetic mutual spaces.

The IEEE VR Steering Committee congratulates and offers gratitude to the enormous efforts of the IEEE VR Conference Organizing Committees—with special recognition for the multiple years of hard work of the General Chairs, Carolina Cruz-Neira, Greg Welch, and Xubo Yang. To realize the conference requires a sizeable, motivated team of volunteers to work with the IEEE conference management team. Maintaining the conference’s status as the premier international virtual reality conference is a testament to the commitment of all the organizers and participants that make up the IEEE Virtual Reality community.

The vision of a ‘metaverse’ may soon bring a ubiquitous(ly) Augmented Reality (UAR) delivering context-aware, geo-located, and continuous blends of real and virtual elements into reach. This paper draws on speculative design to explore, question, and problematize consequences of AR becoming pervasive. Elaborating on Desjardin et al.’s bespoke booklets, we co-speculate together with 12 globally dispersed participants. Each participant received a custom-made design workbook containing pictures of their immediate surroundings, which they elaborated on in situated brainstorming activities. We present an integration of their speculative ideas and lived experiences in 3 overarching themes from which 7 ‘dark’ scenarios caused by UAR were formed. The Scenarios are indicative of deceptive design patterns that can (and likely will be) devised to misuse UAR, and anti-patterns that could cause unintended consequences. These contributions enable the timely discussion of potential antidotes and to which extent they can mitigate imminent harms of UAR.

Creating digital representations of industrial facilities presents substantial opportunities for the industry. The development of digital twins opens up many prospective applications for industrial plants, such as facilitating virtual training environments, streamlining change management, and providing real-time understanding of remote scenes. However, creating and maintaining these digital twins involves complex tasks, including scanning industrial plants and annotating their subsystems. Many use cases necessitate specifying a Volume of Interest (VoI) for further processing while presenting severe environmental challenges for segmentation quality and worker safety. Besides that, widespread adoption also depends on creating methods that are easy to use, even for workers with limited 3D interaction knowledge. Currently, no commonly implemented method fulfills all these diverse requirements. This paper introduces an approach for specifying a VoI in industrial scenarios. Our approach defines a volume by intersecting the projection of hand-drawn surroundings on a small number of pictures of the target volume, utilizing Augmented Reality and Voxel Carving. It can successfully handle various sizes of target volumes and delivers an appropriately detailed result. Applying qualitative discussions and a quantitative evaluation, we ensure our application meets all requirements posed by the scenarios. This simple interaction metaphor, tailored to specific use cases, can serve as a versatile pattern for various digital twin scenarios. It offers a valuable alternative to 3D primitive-based segmentation methods.

In this project, the full spectrum provided by Milgram’s RealityVirtuality Continuum is utilized to enhance presence, usability, and interactions in an authentic Virtual Reality (VR) supermarket simulation used as a standardized evaluation platform for mHealth apps. We introduce a unique Unity-based modeling platform for supermarket environments and the option to design high-quality customized products. To achieve that, solutions are demonstrated by focusing on a recognizable replica of a discounter and utilizing Augmented Virtuality (AV) to include a physical smartphone in the virtual simulation. The user is able to manipulate the simulation from within the smartphone app through this versatile, highly usability-centered controller. To achieve reliable tracking of the smartphone screen, we propose a hybrid approach, which combines fiducial marker tracking with data acquired through a WiFi connection between the VR system and the smartphone. Furthermore, the AV concept is utilized to build scenarios for Mixed Reality (MR) use cases such as simulated AR to navigate to a chosen product in the market. After an initial pre-study with important insights to strengthen the platform, a broad user study involving the physical smartphone with a simulated AR scenario has been conducted. The goals with 30 participants were to evaluate spatial presence, involvement, experienced realism (IPQ), and usability of the system (SUS). Results showed ”Good” (SUS) usability and recorded data as well as the participants’ feedback brought important insights. We plan to release this unique VR supermarket platform to contribute to the science community and mHealth industry.

The IEEE VR Steering Committee congratulates and offers gratitude to the enormous efforts of the IEEE VR Conference Organizing Committees — with special recognition for the multiple years of hard work of the General Chairs, Xubo Yang, Kun Zhou, Tobias Langlotz, and Stephan Lukosch. To realize the conference requires a sizeable, motivated team of volunteers to work with the IEEE conference management team. Maintaining the conference's status as the premier international virtual reality conference is a testament to the commitment of all the organizers and participants that make up the IEEE Virtual Reality community.

We present our interdisciplinary research on gamification as a communication tool in urban planning. Approaching the topic from a user-centered perspective, we combine knowledge from the domains of architecture and human-computer interaction and define domain-specific requirements and areas of interest. To substantiate our findings, we present a prototypical case study of a public participation application, developed in consultation with the urban planning project driver and evaluated on-location with members of the public. Our results highlight the opportunities in pursuing a new sub-field of gamification within urban planning and include concepts of game interfaces and spaces in this research. In our case study, we present a methodological approach to assessing gamification in urban planning participation and provide findings on how gamification changes the users’ relationship to participation as well as evidence that the “avatar” game element can affect the motivational affordances meaning of task, perceived competence, and social relatedness.

Augmented Reality (AR) is increasingly considered for use in real industrial applications [8]. In our industrial research lab at Siemens Technology, we are continuously discussing suitable AR scenarios in business sectors involving power plants, wind turbine plants, and oil and gas factories. We have developed a company-internal AR system architecture, Hololayer, which provides AR facilities in a reusable manner to development engineers such that they can build their own AR applications for their specialized use cases. The integration of AR technology into industrial processes encounters many complex issues: we need to adhere to many safety, security and quality guarantees while also adapting quickly to the rapidly changing state of the art of AR devices (HMDs, tablets). To increase flexibility, it might be good to integrate Hololayer with one of the open frameworks that have recently been proposed [11], [29], [38]. Yet, care must be taken when converting an already existing, large company-owned framework like Hololayer to such platforms. Some of the proposed standardized APIs and communication protocols may be difficult to abide by, without requiring significant rewriting efforts or even violating company-internal regulations. In this paper, we report on our efforts to combine Hololayer with one such platform, Ubi-Interact [38]. This is a collaboration between Siemens technology and the FAR group at TU Munich. After exemplary descriptions of typical application scenarios, we present the underlying principles of Hololayer to support this range of applications. We then describe rudimentary concepts of Ubi-Interact, followed by an elaboration on the efforts to combine both systems for a selected application scenario and a discussion of the results achieved thus far.

The 2022 IEEE VGTC Virtual Reality Lifetime Achievement Award goes to Gudrun Klinker of the Technical University of Munich, in recognition of her lifetime contributions to the development of ubiquitous augmented reality systems.

Augmented reality applications use object tracking to estimate the pose of a camera and to superimpose virtual content onto the observed object. Today, a number of tracking systems are available, ready to be used in industrial applications. However, such systems are hard to handle for a service maintenance engineer, due to obscure configuration procedures. In this article, we investigate options towards replacing the manual configuration process with a machine learning approach based on automatically synthesized data. We present an automated process of creating object tracker facilities exclusively from synthetic data. The data is highly enhanced to train a convolutional neural network, while still being able to receive reliable and robust results during real world applications only from simple RGB cameras. Comparison against related work using the LINEMOD dataset showed that we are able to outperform similar approaches. For our intended industrial applications with high accuracy demands, its performance is still lower than common object tracking methods with manual configuration. Yet, it can greatly support those as an add-on during initialization, due to its higher reliability.

The perseverance when using established but outdated learning systems and approaches in a constantly changing society leads to reduced effectiveness when attempting to reach younger generations. Applying future-oriented solutions, which differ from established learning methods, could support the success of new learning approaches. In this paper, we want to introduce a novel learning concept by combining in-game advertising and learning.

Virtual Reality (VR) offers significant potential for public speaking training. Virtual Reality Speech Training (VR-ST) helps trainees develop presentation skills and practice their application in the real world. Additionally, participants with public speaking anxiety can improve their presentation skills in a safe virtual environment without fear of judgment. Another benefit is direct feedback based on gamification principles, which provides users with information about their performance during training and allows for the adjustment of behavior in real-time. However, it is not yet clear if direct feedback based on visualization through icons is accepted by participants, such that it may support learning transfer in VR training applications. As a result, we set out to investigate how direct feedback in a VR -ST affects the participants' technology acceptance based on the Technology Acceptance Model (TAM). We conducted a between-subjects experimental study in order to compare a VR-ST with direct feedback (n = 100) with a simulation-based VR-ST (n = 100). The resulting MANOVAs demonstrated a preference for the direct feedback version for all TAM determinations, showing that direct feedback offers benefits to trainees by improving technology acceptance, independent of location and without supervision by trainers. Further results show that VR-ST is generally more accepted by participants without public speaking anxiety. Our findings indicate that developers of VR public speaking applications should focus on the inclusion of meaningful direct feedback and consider individual differences between users in order to optimally implement training measures.

This paper investigated the impact of 3D serious games on wayfinding in public buildings. Optimal wayfinding is essential for security, economy and architectural design of public buildings. A comparative study is performed between two groups that used 2D drawings and virtual reality on the specific use-case of a museum. Following the model of Weisman, a particular questionnaire is developed. The results show "figuring out" the building was easier for the 2D group while wayfinding was easier for the VR group. The outcome demonstrates 3D games can be used as an effective tool in architecture for participatory and evidence-based design. Furthermore, the results show that virtual reality may be an effective tool for students to assess their designs.

In order to keep up with technological developments and to be able to work efficiently, the training methodology must be kept up to date. A typical approach when innovating corporate training is to adopt new technologies. We attempt to deliver a common set of characterisations and concepts that allow us to classify and define extended reality training.

The demand for high-quality video games is ever increasing and the ambition to tell truly interactive and dynamic stories is a significant factor contributing to this trend. This paper examines how generated narrative text is perceived by players in terms of meaningfulness, immersion, and flow and furthermore that the presented novel approach can be a valid method to implement computer-generated dialogues. For this approach, generative grammars are used to create written dialogue within a conversation for a learning application. With the support of sentiment analysis, the generated text is analysed with a focus on its semantics. Suitable text lines based on the current game state are provided by a dialogue system. Principles of gamification are used to create a learning application that renders such a generated dialogue scenario playable. To test this hypothesis, a user study examines the capabilities of the process by having players assess factors, which are imperative for a narrative game. The learning application shows strong potential in terms of text variation and dialogue that is easy to follow.

Ubi-Interact is a framework designed to facilitate interactive applications consisting of a variety of individual systems and devices that are distributed over a (local) network. It is designed to allow easy integration of devices and exchange of data based on extendable common data formats. It features edge computing capabilities, allowing to analyze and transform device data. The same computing modules can also be used to manage an arbitrary number of devices in an abstract fashion, allowing ad-hoc handling of (dis-)connecting devices and making devices with similar components/capabilities interchangeable while lowering the necessity to re-implement parts of the overall system behaviour.

Virtual Reality (VR) has been gaining importance due to its numerous advantages as an immersive technology for learning applications. Next to being used in fields like manufacturing, transportation, communication, retail and real estate, it has also increased in significance for human resources development. Virtual human training programs offer exposure to difficult situations without supervision in a safe and controlled environment, as they simulate nuanced interpersonal situations and therefore allow users to gain new skills and apply them to real-life situations. Furthermore, VR training engages employees in a training session by being presented with a realistic situation designed to challenge and engage them. Practising in a controlled virtual environment also allows for easy and objective measurements of user development and the identification of strengths and weaknesses. In this paper, we evaluate if there is acceptance for practising a presentation in a VR-Speech Training (VR-ST) session using six Degrees of Freedom (6DoF) and if the 44 participants of this study improved, from a subjective point of view, valuable soft skills needed to give a convincing presentation in real life. We observed a positive tendency in the acceptance and effectiveness of the VR-ST.

Most AR games focus primarily on static augmentation, leaving out the potential for rich and tangible interactions. Recently, with the creation of the Superhuman Sports genre, new AR games emerged with the goal to extend the sports experience by utilizing technology. With the understanding of various related projects towards physical object interaction in AR and with the rich selection of sensors in mobile devices, we want to present two innovative visions in the direction of AR Sports Games: 1.Floating Ball: A ball shaped object is augmented and while flying in a circular pattern, two players are competing against each other by interacting with it. 2.Augmented Drone: Resulting out of the experiences with the Floating Ball, a competitive game concept with a drone as a versatile augmented game ball will be presented. We want to utilize the results as a step towards a competitive multi-player AR Sports Game based on mobile devices. In both concepts, one device is functioning as a Tangible AR Interface with the task to let the player manipulate the flying path of the physical ball. For this purpose, the ball is incorporated into the augmented world to create a realistic, perceivable playing experience.

In this paper both receptive and productive Augmented Reality (AR)/Virtual Realty (VR)-Concepts to teach art are presented and discussed. Receptive means learning about art as an (immersed) spectator while productive means offering opportunities for own mixed reality artistic creations. The prototypes presented are pioneer research in terms of VR and AR, so called Mixed Reality (MR), informed art education. One of the core aims is to exemplify art educational applications by exploring usage of MR beyond traditional. The three MR based art educational concepts presented here invite users to experience art in heterogenous spatial contexts. MR is used here to highlight semantics of historically and art historically relevant spaces offering immersive as well as creative and participatory ways of learning. Three prototypes of MR informed art education are presented. First, a AR-based art history city guide is described with integration of teacher-based evaluation. Art is taught here in an interdisciplinary approach as situated learning experience visiting Munich's art historical sites and architecture. Teachers' attitudes towards AR/VR-based art history learning were conducted after trial of one of the applications. According to the educator's feedback, motivation, attention but also spatial imagination and perception are estimated to be fostered. Next, an AR-based installation of a missing painting by Franz Marc is presented conveying complex meanings of both architecture in art historical context as well as the art work itself. Finally, an immersive VR-art educational concept integrating two paintings of Caspar David Friedrich is described, offering sequential ways to inform about the painting's iconography. When it comes to implementing MR-concepts in day-to-day practice, media competence as well es technology acceptance or media cultural negotiations are vital aspects beyond the questions of it-technological equipment and infrastructure.

In this paper we present two different approaches with Mixed (Virtual and Augmented) Reality to give pupils an understanding of ancient Greek culture and history in a motivating way. We identify the possibilities of AR in a museum for ancient statues to represent virtual information and to guide the visitor. Moreover, we discuss typical issues of mobile and stationary Virtual Reality (VR) systems in the context of public usage within schools and museums. Additionally, a new VR-streaming approach called SaMaXVR is presented that combines the benefits of mobile and stationary consumer ready VR systems in terms of usability, maintenance, safety and graphics-quality by mitigating many of their individual disadvantages when used in daily business. As this streaming solution consists of cost-effective hardware, this approach can be seen as a valuable and affordable alternative for schools, museums and students. To demonstrate its potential, an application was developed that visualizes complex 3D scans of cultural heritage in their former original environment to give users the possibility to travel back in time (and place) to see how the artifacts might have looked like in the past.

The introduction of Head-Up-Displays (HUDs) have opened up avenues for a whole range of novel AR applications. However, until these applications become available for the mass market a number of problems need to be tackled. For example, the field of view (FoV) of current HUDs is extremely limited, and real world tracking and 3D reconstruction are still not precise enough to show driving information embedded into wide areas of complex traffic environment. It is not possible to show true AR-visualizations in the display areas provided by the current FoVs. In this paper, we investigate how an AR-like visualization approach in current HUDs (with a limited FoV) can support drivers in foreseeing the future road course. This visualisation uses the already established concept of an electronic horizon. By complying with automotive standards, our application can be easily adapted for series production. With this visualisation we performed a user study, investigating the effect on drivers' gaze behaviour. For this reason the test subjects were equipped with an eye tracking system. The results showed a decrease in both, the number of gazes as well as total glance time on the head unit and the instrument cluster. We also investigated the test subjects' braking behaviour around sharp bends of the road which showed an overall improvement when the visualisation was enabled. Furthermore it showed an increase of the mean glance duration in the area of the HUD. Note that the eye tracking system is not capable of distinguishing between glances at the visualisation in the HUD and the users' glance at objects behind the visualisation - overlapping with the HUD. This would require tracking the test persons' depth of focus. The study showed that developers need to be concerned about not displaying excessively in the HUD, so as not to distract drivers. It furthermore showed that AR-like visualizations have the potential to decrease the time the driver is not looking at the road creating a safer driving experience.

Properly calibrating an optical see-through head-mounted display (OST-HMD) and maintaining a consistent calibration over time can be a very challenging task. Automated methods need an accurate model of both the OST-HMD screen and the user's constantly changing eye-position to correctly project virtual information. While some automated methods exist, they often have restrictions, including fixed eye-cameras that cannot be adjusted for different users.To address this problem, we have developed a method that automatically determines the position of an adjustable eye-tracking camera and its unconstrained position relative to the display. Unlike methods that require a fixed pose between the HMD and eye camera, our framework allows for automatic calibration even after adjustments of the camera to a particular individual's eye and even after the HMD moves on the user's face. Using two sets of IR-LEDs rigidly attached to the camera and OST-HMD frame, we can calculate the correct projection for different eye positions in real time and changes in HMD position within several frames. To verify the accuracy of our method, we conducted two experiments with a commercial HMD by calibrating a number of different eye and camera positions. Ground truth was measured through markers on both the camera and HMD screens, and we achieve a viewing accuracy of 1.66 degrees for the eyes of 5 different experiment participants.

Many networked devices that can interface their physical environments are available off-the-shelf or can be built in 2016. A comprehensive management of those Smart Devices is required to unlock the existing potential. However, the amount and heterogeneity of the devices make their management difficult. A suitable abstraction is missing. This paper identifies requirements on managing Smart Devices from diverse research fields, assesses relevant existing work, proposes a new management middleware design, and evaluates it quantitatively and qualitatively. The presented novel middleware architecture could become an enabler for a software maker culture.

We present an off-the-shelf, low-latency Optical See-through Head-Mounted Displays (OST-HMD) for Augmented Reality (AR). Temporally consistent visualization is crucial for realizing immersive AR experiences. This is challenging since it requires both accurate head-tracking and low-latency rendering of AR content. Building a system which meets both constraints usually requires experts on computer vision/graphics and expensive display hardware. This work demonstrates that such high spatio-temporal fidelity is achievable with commodity hardware available today. We build a custom OST-HMD system that consists of a virtual reality HMD, i.e., the Oculus Rift DK2, and half-mirror optics, and adapt the rendering pipeline in order to integrate the OST-HMD calibration framework. An evaluation with a user-perspective camera shows that the system achieves mean temporal error of <;1 ms (95% reduction of the latency from naive, no-predictive rendering), and median spatial error <;0.3° in the viewing angle with maximum error at most 1.0°.

We propose a method to calibrate viewpoint-dependent, channel-wise image blur of near-eye displays, especially of Optical See-Through Head-Mounted Displays (OST-HMDs). Imperfections in HMD optics cause channel-wise image shift and blur that degrade the image quality of the display at a user's viewpoint. If we can estimate such characteristics perfectly, we could mitigate the effect by applying correction techniques from the computational photography in computer vision as analogous to cameras. Unfortunately, directly applying existing calibration techniques of cameras to OST-HMDs is not a straightforward task. Unlike ordinary imaging systems, image blur in OST-HMDs is viewpoint-dependent, i.e., the optical characteristic of a display dynamically changes depending on the current viewpoint of the user. This constraint makes the problem challenging since we must measure image blur of an HMD, ideally, over the entire 3D eyebox in which a user can see an image. To overcome this problem, we model the viewpoint-dependent blur as a Gaussian Light Field (GLF) that stores spatial information of the display screen as a (4D) light field with depth information and the blur as point-spread functions in the form of Gaussian kernels, respectively. We first describe both our GLF model and a calibration procedure to learn a GLF for a given OST-HMD. We then apply our calibration method to two HMDs that use different optics: a cubic prism or holographic gratings. The results show that our method achieves significantly better accuracy in Point-Spread Function (PSF) estimations with an accuracy about 2 to 7 dB in Peak SNR.

Shader lamps can augment physical objects with projected virtual replications using a camera-projector system, provided that the physical and virtual object are well registered to each other. Precise registration and tracking has been a cumbersome and intrusive process in the past. In this paper, we present a new method for tracking complex-shaped physical objects interactively. In contrast to previous approaches our system is mobile and makes solely use of the projection of the virtual replication to track the physical object and "stick" the projection to it. Our method consists of two stages, a fast pose initialization based on structured light patterns and a non-intrusive frame-by-frame tracking based on features detected in the projection. During the tracking phase, a radiometrically corrected virtual camera view based on the current pose prediction is rendered and compared to the captured image. Matched features are triangulated providing a sparse set of surface points that is robustly aligned to the virtual model. The alignment transformation serves as an input for the new pose prediction. Detailed experiments including the evaluation of the overlay accuracy show that our approach can accurately and robustly track complex objects at interactive rates.

In order to produce realistic simulations and enhance immersion in augmented reality systems, solutions must not only present a realistic visual rendering of virtual objects, but also allow natural hand interactions. Most approaches capable of understanding user interaction with virtual content can often be restrictive or computationally expensive. To cope with these problems, we demonstrate a method which employs user's thumb and forefinger to interact with the virtual content in a natural way, utilizing a single RGB-D camera. Based on this method, we develop and realise an augmented reality chess game, focused on providing an immersive experience to users, so that they are able to manipulate virtual chess pieces seamlessly over a board of markers and play against a chess engine.

Location tracking systems have become popular in recent years and have been in use in many areas such as logistics, transportation and medicine. In healthcare, various companies have been offering real time location system solutions to hospitals with great success. These systems are used for asset management, workflow improvement as well as patient and staff tracking. For test runs we decided on a low-cost location tracking system based on proximity detection and the technology Bluetooth Low Energy. The system was developed for the use in OR environments which comes with specific requirements. The goal of this work is to evaluate if people with Bluetooth Low Energy tags are reliably tracked and detected. These tests were conducted in the first stages of a development phase to see, how the tracking system behaved in a controlled office environment. Problems such as the inactivity of the tag and random changes into other nearby rooms because of multipath effects arise and are discussed.

In Augmented Reality (AR) with an Optical See-Through Head-Mounted Display (OST-HMD), the spatial calibration between a user's eye and the display screen is a crucial issue in realizing seamless AR experiences. A successful calibration hinges upon proper modeling of the display system which is conceptually broken down into an eye part and an HMD part. This paper breaks the HMD part down even further to investigate optical aberration issues. The display optics causes two different optical aberrations that degrade the calibration quality: the distortion of incoming light from the physical world, and that of light from the image source of the HMD. While methods exist for correcting either of the two distortions independently, there is, to our knowledge, no method which corrects for both simultaneously. This paper proposes a calibration method that corrects both of the two distortions simultaneously for an arbitrary eye position given an OST-HMD system. We expand a light-field (LF) correction approach [8] originally designed for the former distortion. Our method is camera-based and has an offline learning and an online correction step. We verify our method in exemplary calibrations of two different OST-HMDs: a professional and a consumer OST-HMD. The results show that our method significantly improves the calibration quality compared to a conventional method with the accuracy comparable to 20/50 visual acuity. The results also indicate that only by correcting both the distortions simultaneously can improve the quality.

AR systems have been developed for many years now, ranging from systems consisting of a single sensor and output device to systems with a multitude of sensors and/or output devices. With the increasing complexity of the setup, the complexity of handling the different sensors as well as the necessary calibrations and registrations increases accordingly. A much needed (yet missing) area of augmented reality applications is to support AR system engineers when they set up and maintain an AR system by providing visual guides and giving immediate feedback on the current quality of their calibration measurements. In this poster we present an approach to use Augmented Reality itself to support the user in calibrating an Augmented Reality system.

Usability evaluations are important to the development of augmented reality systems. However, conducting large-scale longitudinal studies remains challenging because of the lack of inexpensive but appropriate methods. In response, we propose a method for implicitly estimating usability ratings based on readily available sensor logs. To demonstrate our idea, we explored the use of features of accelerometer data in estimating usability ratings in an annotation task. Results show that our implicit method corresponds with explicit usability ratings at 79% and 84%. These results should be investigated further in other use cases, with other sensor logs.

In recent years, location tracking systems have become important in areas such as transportation, shopping, logistics and medicine. One of the most recent approaches are proximity-based location tracking systems, which use the received signal strength (RSSI) measured between a sender and an anchor module. The advantages of these systems are high scalability, minimal calibration effort and low costs. However, there are also disadvantages e.g. the fluctuating signal strength under certain circumstances in the environment. To detect, if an object has entered a predefined region, the RSSI must exceed a specified threshold. When the current signal fluctuates due to obstacles (other objects, people, etc.) the static threshold does not apply for the defined region any longer. In this case automatic adjustments of the threshold are made with the help of a reference module. Various factors are measured between reference, anchor as well as a sender module and subsequently used to correct the static threshold. The dynamic adjustment was tested through three experiments and showed satisfactory results concerning the hypotheses. For further research, additional factors might need to be included to make the proposed method more dynamic and responsive to the environment.

We propose a new semi-automatic calibration approach for projector-camera systems that - unlike existing auto-calibration approaches - additionally recovers the necessary global scale by projecting on an arbitrary object of known geometry from one view. Our method therefore combines surface registration with bundle adjustment optimization on points reconstructed from structured light projections. In simulations on virtual data and experiments with real data we demonstrate that our approach estimates the global scale robustly and is furthermore able to improve incorrectly guessed intrinsic and extrinsic calibration parameters.

Visuo-haptic augmented reality systems enable users to see and touch digital information that is embedded in the real world. PHANToM haptic devices are often employed to provide haptic feedback. Precise co-location of computer-generated graphics and the haptic stylus is necessary to provide a realistic user experience. Previous work has focused on calibration procedures that compensate the non-linear position error caused by inaccuracies in the joint angle sensors. In this article we present a more complete procedure that additionally compensates for errors in the gimbal sensors and improves position calibration. The proposed procedure further includes software-based temporal alignment of sensor data and a method for the estimation of a reference for position calibration, resulting in increased robustness against haptic device initialization and external tracker noise. We designed our procedure to require minimal user input to maximize usability. We conducted an extensive evaluation with two different PHANToMs, two different optical trackers, and a mechanical tracker. Compared to state-of-the-art calibration procedures, our approach significantly improves the co-location of the haptic stylus. This results in higher fidelity visual and haptic augmentations, which are crucial for fine-motor tasks in areas such as medical training simulators, assembly planning tools, or rapid prototyping applications.

Presents the introductory editorial for this issue of the publication.

The fundamental issues in Augmented Reality (AR) are on how to naturally mediate the reality with virtual content as seen by users. In AR applications with Optical See-Through Head-Mounted Displays (OST-HMD), the issues often raise the problem of rendering color on the OST-HMD consistently to input colors. However, due to various display constraints and eye properties, it is still a challenging task to indistinguishably reproduce the colors on OST-HMDs. An approach to solve this problem is to pre-process the input color so that a user perceives the output color on the display to be the same as the input. We propose a color calibration method for OST-HMDs. We start from modeling the physical optics in the rendering and perception process between the HMD and the eye. We treat the color distortion as a semi-parametric model which separates the non-linear color distortion and the linear color shift. We demonstrate that calibrated images regain their original appearance on two OST-HMD setups with both synthetic and real datasets. Furthermore, we analyze the limitations of the proposed method and remaining problems of the color reproduction in OST-HMDs. We then discuss how to realize more practical color reproduction methods for future HMD-eye system.

A critical requirement for AR applications with Optical See-Through Head-Mounted Displays (OST-HMD) is to project 3D information correctly into the current viewpoint of the user - more particularly, according to the user's eye position. Recently-proposed interaction-free calibration methods [16], [17] automatically estimate this projection by tracking the user's eye position, thereby freeing users from tedious manual calibrations. However, the method is still prone to contain systematic calibration errors. Such errors stem from eye-/HMD-related factors and are not represented in the conventional eye-HMD model used for HMD calibration. This paper investigates one of these factors - the fact that optical elements of OST-HMDs distort incoming world-light rays before they reach the eye, just as corrective glasses do. Any OST-HMD requires an optical element to display a virtual screen. Each such optical element has different distortions. Since users see a distorted world through the element, ignoring this distortion degenerates the projection quality. We propose a light-field correction method, based on a machine learning technique, which compensates the world-scene distortion caused by OST-HMD optics. We demonstrate that our method reduces the systematic error and significantly increases the calibration accuracy of the interaction-free calibration.

With the growing availability of optical see-through (OST) head-mounted displays (HMDs) there is a present need for robust, uncomplicated, and automatic calibration methods suited for non-expert users. This work presents the results of a user study which both objectively and subjectively examines registration accuracy produced by three OST HMD calibration methods: (1) SPAAM, (2) Degraded SPAAM, and (3) Recycled INDICA, a recently developed semi-automatic calibration method. Accuracy metrics used for evaluation include subject provided quality values and error between perceived and absolute registration coordinates. Our results show all three calibration methods produce very accurate registration in the horizontal direction but caused subjects to perceive the distance of virtual objects to be closer than intended. Surprisingly, the semi-automatic calibration method produced more accurate registration vertically and in perceived object distance overall. User assessed quality values were also the highest for Recycled INDICA, particularly when objects were shown at distance. The results of this study confirm that Recycled INDICA is capable of producing equal or superior on-screen registration compared to common OST HMD calibration methods. We also identify a potential hazard in using reprojection error as a quantitative analysis technique to predict registration accuracy. We conclude with discussing the further need for examining INDICA calibration in binocular HMD systems, and the present possibility for creation of a closed-loop continuous calibration method for OST Augmented Reality.

In recent years optical see-through head-mounted displays (OST-HMDs) have moved from conceptual research to a market of mass-produced devices with new models and applications being released continuously. It remains challenging to deploy augmented reality (AR) applications that require consistent spatial visualization. Examples include maintenance, training and medical tasks, as the view of the attached scene camera is shifted from the user's view. A calibration step can compute the relationship between the HMD-screen and the user's eye to align the digital content. However, this alignment is only viable as long as the display does not move, an assumption that rarely holds for an extended period of time. As a consequence, continuous recalibration is necessary. Manual calibration methods are tedious and rarely support practical applications. Existing automated methods do not account for user-specific parameters and are error prone. We propose the combination of a pre-calibrated display with a per-frame estimation of the user's cornea position to estimate the individual eye center and continuously recalibrate the system. With this, we also obtain the gaze direction, which allows for instantaneous uncalibrated eye gaze tracking, without the need for additional hardware and complex illumination. Contrary to existing methods, we use simple image processing and do not rely on iris tracking, which is typically noisy and can be ambiguous. Evaluation with simulated and real data shows that our approach achieves a more accurate and stable eye pose estimation, which results in an improved and practical calibration with a largely improved distribution of projection error.

In the Shader Lamps concept, a projector-camera system augments physical objects with projected virtual textures, provided that a precise intrinsic and extrinsic calibration of the system is available. Calibrating such systems has been an elaborate and lengthy task in the past and required a special calibration apparatus. Self-calibration methods in turn are able to estimate calibration parameters automatically with no effort. However they inherently lack global scale and are fairly sensitive to input data. We propose a new semi-automatic calibration approach for projector-camera systems that - unlike existing auto-calibration approaches - additionally recovers the necessary global scale by projecting on an arbitrary object of known geometry. To this end our method combines surface registration with bundle adjustment optimization on points reconstructed from structured light projections to refine a solution that is computed from the decomposition of the fundamental matrix. In simulations on virtual data and experiments with real data we demonstrate that our approach estimates the global scale robustly and is furthermore able to improve incorrectly guessed intrinsic and extrinsic calibration parameters thus outperforming comparable metric rectification algorithms.

Visuo-haptic augmented reality systems enable users to see and touch digital information that is embedded in the real world. Precise co-location of computer graphics and the haptic stylus is necessary to provide a realistic user experience. PHANToM haptic devices are often used in such systems to provide haptic feedback. They consist of two interlinked joints, whose angles define the position of the haptic stylus and three sensors at the gimbal to sense its orientation. Previous work has focused on calibration procedures that align the haptic workspace within a global reference coordinate system and developing algorithms that compensate the non-linear position error, caused by inaccuracies in the joint angle sensors. In this paper, we present an improved workspace calibration that additionally compensates for errors in the gimbal sensors. This enables us to also align the orientation of the haptic stylus with high precision. To reduce the required time for calibration and to increase the sampling coverage, we utilize time-delay estimation to temporally align external sensor readings. This enables users to continuously move the haptic stylus during the calibration process, as opposed to commonly used point and hold processes. We conducted an evaluation of the calibration procedure for visuo-haptic augmented reality setups with two different PHANToMs and two different optical trackers. Our results show a significant improvement of orientation alignment for both setups over the previous state of the art calibration procedure. Improved position and orientation accuracy results in higher fidelity visual and haptic augmentations, which is crucial for fine-motor tasks in areas including medical training simulators, assembly planning tools, or rapid prototyping applications. A user friendly calibration procedure is essential for real-world applications of VHAR.

Visuo-haptic augmented reality systems enable users to see and touch digital information that is embedded in the real world. Precise colocation of computer graphics and the haptic stylus is necessary to provide a realistic user experience. PHANToM haptic devices are often used in such systems to provide haptic feedback. They consist of two interlinked joints, whose angles define the position of the haptic stylus and three sensors at the gimbal to sense its orientation. Previous work has focused on a calibration procedures that align the haptic workspace within a global reference coordinate system and an algorithms that compensate the non-linear position error, which is caused by inaccuracies in the joint angle sensors. In our science and technology paper “Comprehensive Workspace Calibration for Visuo-Haptic Augmented Reality” [1], we present an improved workspace calibration that additionally compensates for errors in the gimbal sensors. This enables us to also align the orientation of the haptic stylus with high precision. To reduce the required time for calibration and to increase the sampling coverage, we utilize time-delay estimation to temporally align external sensor readings. This enables users to continuously move the haptic stylus during the calibration process, as opposed to commonly used point and hold processes. This demonstration showcases the complete workspace calibration procedure as described in our paper including a mixed reality demo scenario, that allows users to experience the calibrated workspace. Additionally, we demonstrate an early stage of our proposed future work in improved user guidance during the calibration procedure using visual guides.

An issue in AR applications with Optical See-Through Head-Mounted Display (OST-HMD) is to correctly project 3D information to the current viewpoint of the user. Manual calibration methods give the projection as a black box which explains observed 2D-3D relationships well (Fig. 1). Recently, we have proposed an INteraction-free DIsplay CAlibration method (INDICA) for OST-HMD, utilizing camera-based eye tracking [7]. It reformulates the projection in two ways: a black box with an actual eye model (Recycle Setup), and a combination of an explicit display model and an eye model (Full Setup). Although we have shown the former performs more stably than a repeated SPAAM calibration, we could not yet prove whether the same holds for the Full Setup. More importantly, it is still unclear how the error in the calibration parameters affects the final results. Thus, the users can not know how accurately they need to estimate each parameter in practice. We provide: (1) the fact that the Full Setup performs as accurately as the Recycle Setup under a marker-based display calibration, (2) an error sensitivity analysis for both SPAAM and INDICA over the on-/offline parameters, and (3) an investigation of the theoretical sensitivity on an OST-HMD justified by the real measurements.

A correct spatial registration of Optical See-Through Head-Mounted Displays (OST-HMD) w.r.t. a user's eye(s) is an essential problem for any AR application using the such HMDs (Fig. 1). Maintaining the correct registration demands frequent (re)calibrations for the end-users whenever they move the HMD on their head. Thus, a calibration technique should be simple and accurate for the universal, long-run use of the displays. This demonstration showcases INDICA, an automatic OST-HMD calibration approach presented in our previous work[1] and ISMAR 2014 paper [2]. The method calibrates the display to the user's current eyeball position by combining online eye-position tracking with offline parameters. Visitors of our demonstration can try our both manual calibration and our interaction-free calibration on a customized OST-HMD.

Shader lamps can augment physical objects with projected virtual replications using a camera-projector system, provided that the physical and virtual object are well registered. Precise registration and tracking has been a cumbersome and intrusive process in the past. In this paper, we present a new method for tracking arbitrarily shaped physical objects interactively. In contrast to previous approaches our system is mobile and makes solely use of the projection of the virtual replication to track the physical object and “stick” the projection to it. Our method consists of two stages, a fast pose initialization based on structured light patterns and a non-intrusive frame-by-frame tracking based on features detected in the projection. In the initialization phase a dense point cloud of the physical object is reconstructed and precisely matched to the virtual model to perfectly overlay the projection. During the tracking phase, a radiometrically corrected virtual camera view based on the current pose prediction is rendered and compared to the captured image. Matched features are triangulated providing a sparse set of surface points that is robustly aligned to the virtual model. The alignment transformation serves as an input for the new pose prediction. Quantitative experiments show that our approach can robustly track complex objects at interactive rates.

The early design phase for a new building is a crucial stage in the design process of architects. It has to be ensured that the building fits into the future environment. The Collaborative Design Platform targets this issue by integrating modern digital means with well known traditional concepts. Well-used styrofoam blocks are still cut by hand but are now tracked, placed and visualized in 3D by use of a tabletop platform and a TV screen showing an arbitrary view of the scenery. With this demonstration, we get one step further and provide an interactive visualization at the proposed building site, further enhancing collaboration between different audiences. Mobile phones and tablet devices are used to visualize marker-less registered virtual building structures and immediately show changes made to the models in the Collaborative Design laboratory. This way, architects can get a direct impression about how a building will integrate within the environment and residents can get an early impression about future plans.

Gaze tracking facilities have yet mainly been used in general for marketing or the disabled and, more specifically, in Augmented Reality, for interaction with control triggers, such as buttons. We go one step further and use the line of sight of the user to attach information. While any information may not conceal the view of the user, we displace the information by an angular degree and provide means for the user to capture the information by looking at it. With such an apporach we see a potential for faster resuming times of the original task for which a required information needs to be accessed. The demonstration shows a comparably complex primary task assisted by our gaze-mounted information and illustrates the inherent differences for information access w.r.t. conventional methods, such as listing action items at a fix position in space or on a screen.

Integrating different mental models and the corresponding interaction techniques for navigation and object manipulation tasks is a topic of concern for providing user interfaces for the wide variety of potential users. With a user controlled, egocentric hand-held device we aim at integrating the so far identified three major interaction techniques based on the metaphors of a steering wheel, a toy airplane and a window frame. Here, specifically the toy airplane and the window frame contradict each other, leaving the issue at which postural angle of the hand-held device the mental model of the users changes.

Like in every process model, agile processes (e.g. Scrum, eXtreme Programming) depend on accurate estimations to enable meaningful prioritization, iteration- and release planning. The emphasis of this paper is on "Planning Poker", a widely used estimation technique in agile context. The goal is to identify inaccurate effort estimates to enable more precise project scheduling and release planning. In a first step, basic terms (e.g. development effort, functional effort) of traditional effort estimation were declared in the agile context. Afterwards-within the main part of the paper-a set of metrics is proposed, which can be used to evaluate accuracy of estimates. These metrics are based on the estimated efforts and interim results of the estimation process. Since the usage of these metrics needs to be seamlessly integrated into the Planning Poker process, we have conceptualized a computer-aided tool to collect and evaluate necessary data. To get a first proof of concept and preliminary feedback, we developed a corresponding prototype and applied it to a students' project. The prototype is based on a stationary multi-touch device and offers an intuitive user interface to perform the Planning Poker process. While estimation takes place, it collects required data and identifies inaccurate estimates with the help of the proposed metrics. The second part of the paper summarizes assets and drawbacks concerning usability and the user interface of the prototype.

Abstract: The spatial understanding of chemical molecules is crucial for learning chemistry at school. With a good 3D understanding of molecules, chemical processes become obvious compared to a 2D representation in textbooks or just the molecular formula. With the increasing spread of computers, smartphones and tablets, the field of computer aided learning becomes more and more important. Common molecular viewers such as Jmol (Jmol, 2012) present chemical simulations as 3D renderings on a regular computer screen in combination with desktop-based user interfaces using a mouse and a keyboard to manipulate 3D molecules. Such interfaces may be cumbersome to use since users have to associate 2D mouse motion and key presses with 3D object motions. In this paper we investigate the hypothesis that the understanding of spatial structures of molecules is enhanced by Augmented-Reality-based 3D user interfaces with which students can directly manipulate the virtual 3D molecules by freely moving and rotating a 3D object in air with their hands. Our results show that a direct manipulation 3D user interface improves the 3D understanding in comparison to the traditional desktop-based user interface with mouse and keyboard. 1

We introduce the first publicly available test dataset for outdoor handheld camera localization comprising over 45,000 real camera images of an urban environment captured under natural camera motions and different illumination settings. For all these images the dataset not only contains readings of the sensors attached to the camera, but also ground truth information on the geometry and texture of the environment and the full 6DoF ground truth camera pose. This poster describes the extensive process of creating this comprehensive dataset that we have made available to the public. We hope this not only enables researchers to objectively evaluate their camera localization and tracking algorithms and frameworks on realistic data but also stimulates further research.

Current Augmented Reality navigation applications for pedestrians usually do not visualize tracking errors. However, tracking uncertainties can accumulate so that the user is presented with a distorted impression of navigation accuracy. To increase the awareness of users about potential imperfections of the tracking at a given time, we alter the visualization of the navigation system. We developed four visualization and error visualization concepts and used a controlled Mixed Reality environment to conduct a pilot study. We found that, while error visualization has the potential to improve AR navigation systems, it is difficult to find suitable visualizations, which are correctly understood among the users.

Already 3D visualization environments provide a large design space not being investigated to the same extent as traditional WIMP-spaces. When using this design space in combination with AR, the design space even further grows. Information can not only be presented in a 3D space, AR also puts virtual information in relation to real objects, locations or events. The different properties of presentation in AR need to be investigated to develop a comprehensive set of dimensions of presentation principles.

Marker-based optical outside-in tracking is a mature and robust technology used by many AR, VR and motion capture applications. However, in small environments the tracking cameras are often difficult to install. An example scenario are ergonomic studies in car manufacturing, where the motion of a worker needs to be tracked in small spaces such as the trunk of a car. In this paper, we describe how to extend the tracking volume in small, cluttered environments using small and flexible wireless cameras in form of unmodified mobile phones that can quickly be installed. Since those small cameras are not synchronized with the main tracking cameras, we describe several modifications to the tracking algorithms, such as inter-frame interpolation, the replacement of the least-squares adjustment by a Kalman filter and the integration of rolling-shutter compensation. To support the quick setup of additional cameras while the tracking system is running, the system is extended by an on-line calibration technique that determines the extrinsic camera parameters without requiring a dedicated calibration step.

The calibration of optical see-through head-mounted displays is an important fundament for correct object alignment in augmented reality. Any calibration process for OSTHMDs requires users to align 2D points in screen space with 3D points in the real world and to confirm each alignment. In this poster, we present the results of our empiric evaluation where we compared four confirmation methods: Keyboard, Hand-held, Voice, and Waiting. The Waiting method, designed to reduce head motion during confirmation, showed a significantly higher accuracy than all other methods. Averaging over a time frame for sampling user input before the time of confirmation improved the accuracy of all methods in addition. We conducted a further expert study proving that the results achieved with a video see-through head-mounted display showed valid for optical see-through head-mounted display calibration, too.

The two papers in this special section are extended versions of papers originally presented at the International Symposium on Mixed and Augmented Reality (ISMAR) 2007. These two papers won awards at the symposium.

We describe the design and implementation of a physical and virtual model of an imaginary urban scene-the “City of Sights”- that can serve as a backdrop or “stage” for a variety of Augmented Reality (AR) research. We argue that the AR research community would benefit from such a standard model dataset which can be used for evaluation of such AR topics as tracking systems, modeling, spatial AR, rendering tests, collaborative AR and user interface design. By openly sharing the digital blueprints and assembly instructions for our models, we allow the proposed set to be physically replicable by anyone and permit customization and experimental changes to the stage design which enable comprehensive exploration of algorithms and methods. Furthermore we provide an accompanying rich dataset consisting of video sequences under varying conditions with ground truth camera pose. We employed three different ground truth acquisition methods to support a broad range of use cases. The goal of our design is to enable and improve the replicability and evaluation of future augmented reality research.

The accuracy of a real time tracking system for industrial AR (IAR) applications often needs to comply with production tolerances. Such a system typically incorporates different off-/online devices so that the overall precision and accuracy cannot be trivially stated. Additionally, tracking needs to be flexible to not interfere with existing working processes and it needs to be operated and maintained free of error by on-site personnel who typically have a quality management (QM) background. For the final validation of such a complex tracking setup, empiric testing alone is either too expensive or lacks generality. This paper demonstrates a new approach to define and verify, deploy and validate, as well as to operate and maintain an IAR tracking infrastructure. We develop our concepts on the basis of an IAR application in the field of QM in the aircraft production process. It integrates a qualitative visual comparison with accurate quantitative measurements of 3D coordinates using a metrological probe. The focus is on the verification, validation, and error free operation. Monte Carlo simulation predicts the error for arbitrary system states. Using a limited set of empiric measurements in the target environment allows us to validate the simulation and thereby validate the application. This combination assures compliance of the IAR application with the required production tolerances. We show that our simulation model yields realistic results, using an in-depth analysis of an optical IR tracking system and a high-precision coordinate measurement machine capable of densely sampling the entire tracking volume. Additionally, it allows for a straightforward derivation of run-time consistency checks for the automatic identification of possible system failures. Also, estimation of the system performance during the planning and definition phases becomes possible, using the elementary accuracy specifications of the involved sensor systems.

Systems using two-handed spatial manipulation techniques also require strategies to enable system control tasks. These strategies make it possible to interact with the system comfortably while controlling two hand-held objects simultaneously. The Augmented Chemical Reactions project makes intense use of such two-handed interaction tasks. Users control virtual molecules and subsets that are registered to physical markers and try to combine those by selecting and then confirming a specific bond. When a desired bond has been selected, the user needs a way to confirm that bond without letting an atom go out of position. We developed and investigated two separate methods of confirming a selected bond when both hands are already doing a two-handed symmetric interaction task. The first method is a waiting method and the second method is a back&forth motion gesture. We evaluated the two methods in a user study, showing that the first technique, holding still, outperforms the other technique.

In many augmented reality applications, in particular in the medical and industrial domains, knowledge about tracking errors is important. Most current approaches characterize tracking errors by 6×6 covariance matrices that describe the uncertainty of a 6DOF pose, where the center of rotational error lies in the origin of a target coordinate system. This origin is assumed to coincide with the geometric centroid of a tracking target. In this paper, we show that, in case of a multi-camera fiducial tracking system, the geometric centroid of a body does not necessarily coincide with the point of minimum error. The latter is not fixed to a particular location, but moves, depending on the individual observations. We describe how to compute this point of minimum error given a covariance matrix and verify the validity of the approach using Monte Carlo simulations on a number of scenarios. Looking at the movement of the point of minimum error, we find that it can be located surprisingly far away from its expected position. This is further validated by an experiment using a real camera system.

Indoor tracking scenarios still face challenges in providing continuous tracking support in wide-area workplaces. This is especially the case in Augmented Reality since such augmentations generally require exact full 6DOF pose measurements in order to continuously display 3D graphics from user-related view points. Many single sensor systems have been explored but only few of them have the capability to track reliably in wide-area environments. We introduce a mobile multi-sensor platform to overcome the shortcomings of single sensor systems. The platform is equipped with a detachable optical camera and a rigidly mounted odometric measurement system providing relative positions and orientations with respect to the ground plane. The camera is used for marker-based as well as for marker-less (feature-based) inside-out tracking as part of a hybrid approach. We explain the principle tracking technologies in our competitive/cooperative fusion approach and show possible enhancements to further developments. This inside-out approach scales well with increasing tracking range, as opposed to stationary outside-in tracking.

In recent years, a large number of multitouch sensor concepts have been presented. Particularly optical sensors are highly popular due to their versatility. However, especially camera-based systems often require a significant amount of space behind the screen and are not well suited to flatscreen-based setups. While integrated sensors for flatscreens have already been presented, they are mostly complex, expensive or both.

Abstract Order picking is one of the most important process steps in logistics. Because of their flexibility, human beings cannot be replaced by machines. But if workers, in order, picking systems are equipped with a head‐mounted display, Augmented Reality can improve the information visualization. In this paper, the development of such a system—called Pick‐by‐Vision—is presented. The system is evaluated in a user study performed in a real storage environment. Important logistics figures as well as the subjective strain were measured. The results show that a Pick‐by‐Vision system can considerably improve industrial order picking processes.

Spatial tracking is one of the most challenging parts of Augmented Reality. Many AR applications rely on the fusion of several tracking systems in order to optimize the overall performance. While the topic of sensor fusion has already seen considerable interest, most results only deal with the integration of particular setups. A crucial part of sensor fusion is the temporal alignment of the sensor signals, as sensors in general are not synchronized. We present a general method to calibrate the temporal offset between different sensors by applying the normalized cross correlation method.

We describe a mobile augmented reality application that is based on 3D snapshotting using multiple photographs. Optical square markers provide the anchor for reconstructed virtual objects in the scene. A novel approach based on pixel flow highly improves tracking performance. This dual tracking approach also allows for a new single-button user interface metaphor for moving virtual objects in the scene. The development of the AR viewer was accompanied by user studies confirming the chosen approach.

In this paper we report on our ongoing studies around the application of augmented reality methods to support the order picking process of logistics applications. Order picking is the gathering of goods out of a prepared range of items following some customer orders. We named the visual support of this order picking process using head-mounted displays ldquopick-by-visionrdquo. This work presents the case study of bringing our previously developed pick-by-vision system from the lab to an experimental factory hall to evaluate it under more realistic conditions. This includes the execution of two user studies. In the first one we compared our pick-by-vision system with and without tracking to picking using a paper list to check picking performance and quality in general. In a second test we had subjects using the pick-by-vision system continuously for two hours to gain in-depth insight into the longer use of our system, checking user strain besides the general performance. Furthermore, we report on the general obstacles of trying to use HMD-based AR in an industrial setup and discuss our observations of user behaviour.

The task of vision based people tracking is a major research problem in the context of surveillance applications or human behavior estimation, but it has had only minimal impact on (Ubiquitous) Augmented Reality applications thus far. Deploying stationary infrastructural cameras within indoor environments for the purpose of Augmented Reality could provide a users' devices with additional functionality that a small device and mobile sensors cannot provide to its user. Therefore people tracking could be expected to become an ubiquitously available infrastructural element in buildings since surveillance cameras are widely used. The use for scenarios indoors or close to buildings is obvious. We present and discuss several different ways where people tracking in real-time could influence the fields of Augmented Reality and further vision based applications.

Many types of tangible interaction systems, such as interactive surfaces and gesture-based interfaces, are based on various kinds of optical tracking, using infrared illuminators and cameras. One drawback of these setups is that they suffer from problems common to optical trackers, such as sensitivity to stray environment light from artificial and natural sources. In this paper, we present a method to significantly enhance tracking robustness for those systems which employ active illumination. Through addition of a small electronic circuit which modulates the LEDs used to illuminate the scene, contrast can be significantly increased.

Multi-touch interfaces have been a focus of research in recent years, resulting in development of various innovative UI concepts. Support for existing WIMP interfaces, however, should not be overlooked. Although several approaches exist, there is still room for improvement, particularly regarding implementation of the "hover" state, commonly used in mouse-based interfaces.

Ubiquitous tracking (Ubitrack) setups, consisting of many previously unknown sensors, offer many possibilities to perform sensor fusion in order to increase robustness and accuracy. In particular, the dynamic combination of mobile and stationary trackers enables the creation of new wide-area tracking concepts. In this work, we present a setup in which a gyroscope is dynamically fused with three different mobile and stationary sensors, based on the concepts of spatial relationship graphs (SRGs) and patterns. For this, we contribute new patterns that, based on well-known algorithms, enable the transformation of rotation velocity and the fusion with different absolute trackers. The usefulness of the approach is shown in a system that automatically reconfigures the SRG based on course tracking data, and, depending on the structure of this SRG, automatically selects a suitable fusion algorithm.

We report on recent progress in the iterative process of exploring, evaluating and refining Augmented Reality-based methods to support the order picking process. We present our findings from three user studies and from demonstrations at several exhibitions. The resulting setup is a combined visualization to precisely and efficiently guide the user, even if the augmentation is not always in the field of view of the HMD.

Because the resolution of see-through displays is lower than the resolution of the human eye, perception of AR schemes is complicated in large distances. To discover, how design issues of perception correlate with presentation in large distances, we developed three different variants of arrow-based route guidance systems. With a large-scale head-up display having a large focal depth, we tested the variants under different conditions on perception and interpretability.

It is widely recognized that inertial sensors, in particular gyroscopes, can improve the latency and accuracy of orientation tracking by fusing the inertial measurements with data from other sensors. In our previous work, we introduced the concepts of spatial relationship graphs and spatial relationship patterns to formally model multi-sensor tracking setups and derive valid applications of well-known algorithms in order to infer new spatial relationships for tracking and calibration. In this work, we extend our approach by providing additional spatial relationship patterns that transform incremental rotations and add gyroscope alignment and fusion. The usefulness of the resulting tracking configurations is evaluated in two different scenarios with both inside-out and outside-in tracking.

The paper explores the use of laser projectors as an alternative to head-mounted displays for Augmented Reality. We describe the development of an Augmented Reality Laser Projector and report on experiences setting up AR systems that use laser projectors, reasoning about several design criteria.

Ubiquitous tracking setups, covering large tracking areas with many heterogeneous sensors of varying accuracy, require dedicated middleware to facilitate development of stationary and mobile applications by providing a simple interface and encapsulating the details of sensing, calibration and sensor fusion. In this paper we present a centrally coordinated peer-to-peer architecture for ubiquitous tracking, where a server computes optimal data flow configurations for sensor and application clients, which are directly exchanging tracking data with low latency using a light-weight data flow framework. The server's decisions are inferred from an actively maintained central spatial relationship graph (SRG) using spatial relationship patterns. The system is compared to a previous Ubitrack implementation using the highly distributed DWARF middleware. It exhibits significantly better performance in a reference scenario.

Most car accidents occur due to longitudinal collisions or lane departure. We assume that the number of such accidents can be reduced, if the driver knows more precisely, where the car is heading and at which distance it can stop. To provide drivers with this kind of anticipation, we have developed two augmented reality based visualization schemes for longitudinal and lateral driver assistance in the head-up display (HUD) of cars. One presentation scheme indicates the braking distance by a virtual bar on the road. The second scheme adds the visualization of a drive-path between the car and the bar, zoning the entire region that the car will pass before coming to a complete halt. We have tested both schemes in a driving simulator in comparison to a baseline without visual assistance. Our results show, among other findings, that the bar is preferred, that it supports driving performance and that it does not increase mental workload.

Spatial sensor systems in cars are gaining more and more importance. Such sensor systems are the foundation of future safety systems, such as automatic emergency brakes, as well as for interactive driver assistance systems. We have developed a system that can visualize such spatial sensor data. Two environments are supported: a laboratory setup for off-line experience and a car setup that enables live experience of spatially aligned laser scanner and video data in real traffic. We have used two visualization devices, a video see-through LCD flat panel (TFT) and an optical see-through head-mounted display (HMD) in both setups. For the laboratory setup, a back-projection table has been integrated as well. To present data in correct spatial alignment, we have installed tracking systems in both environments. Visualization schemes for spatial sensor data and for geometric models that outline recognized objects have been developed. We report on our system and discuss experiences from the development and realization phases. The system is not intended to be used as a component of real driver assistance systems. Rather, it can bridge the gap between human machine interface (HMI) designers and sensing engineers during the development phase. Furthermore, it can be both a debugging tool for the realization of environmental perception systems and an experimental platform for the design of presentation schemes for upcoming driver assistance systems.

After the successful transfer of hierarchical menu-structures from the computer domain to an automotive environment, it is time to discuss the potential of 3D metaphors to meet the strong requirements for in-vehicle information systems (IVIS). The idea is to increase learnability, efficiency and joy of use of IVIS by providing a 3D interaction concept that is based on cognitive capabilities of humans. We present a 3D interaction metaphor, iFlip, which consists of displaying information on the reverse side of thin interaction objects and a preview to current submenu states.A comparison with a traditional list-based 2D menu for IVIS has shown that iFlip fulfills automotive requirements and can even enhance usability and likeability of IVIS.

Marker-based optical tracking systems are widely used in augmented reality, medical navigation and industrial applications. We propose a model for the prediction of the target registration error (TRE) in these kinds of tracking systems by estimating the fiducial location error (FLE) from two-dimensional errors on the image plane and propagating that error to a given point of interest. We have designed a set of experiments in order to estimate the actual parameters of the model for any given tracking system. We present the results of a study which we used to demonstrate the effect of different sources of error. The method is applied to real applications to show the usefulness for any kind of augmented reality system. We also present a set of tools that can be used to visualize the accuracy at design time.

Augmented reality (AR) can be used in the automotive industry to compare real parts of a car with their associated construction data. The real parts have to be checked whether they correspond to the latest version of the design and whether they have been manufactured with appropriate precision. With AR, CAD construction data can be superimposed on the real parts striving for maximum correspondence. The real and the virtual part should both be visible at the same time and in the same place. Therefore, for this kind of overlay, a special method of augmentation is needed. We present and discuss some visualization schemes.

With tracking setups becoming increasingly complex, it gets more difficult to find suitable algorithms for tracking, calibration and sensor fusion. A large number of solutions exists in the literature for various combinations of sensors, however, no development methodology is available for systematic analysis of tracking setups. When modeling a system as a spatial relationship graph (SRG), which describes coordinate systems and known transformations, all algorithms used for tracking and calibration correspond to certain patterns in the graph. This paper introduces a formal model for representing such spatial relationship patterns and presents a small catalog of patterns frequently used in augmented reality systems. We also describe an algorithm to identify patterns in SRGs at runtime for automatic construction of data flows networks for tracking and calibration.

In cars, augmented reality is becoming an interesting means to enhance active safety in the driving task. Guiding a driver's attention to an imminent danger somewhere around the car is a potential application. In a research project with the automotive industry, we are exploring different approaches towards alerting drivers to such dangers. First results were presented last year. We have extended two of these approaches. One uses AR to visualize the source of danger in the driver's frame of reference while the other one presents information in a bird's eye schematic map. Our extensions were the incorporation of a real head-up display, improved visual perception and acoustic support. Both schemes were evaluated both with and without 3D encoded sound. This paper reports on a user test in which 24 participants provided objective and subjective measurements. The results indicate that the AR-based three-dimensional presentation scheme with and without sound support systematically outperforms the bird's eye schematic map.

User interfaces for ubiquitous augmented reality incorporate a wide variety of concepts such as multi-modal, multi-user, multi-device aspects and new input/output devices. In this paper we present a twofold approach that consists of an execution engine for ubiquitous augmented reality user interfaces and a runtime development environment that enables rapid prototyping and live system adaption for such advanced user interfaces.

We introduce a sensor fusion approach for automated initialization of marker-less tracking systems. It is not limited in tracking range and working environment, given a 3D model of the objects or the real scene. This is achieved based on a statistical analysis and probabilistic estimation of the uncertainties of the tracking sensors. The explicit representation of the error distribution allows the fusion of different sensor data. This methodology was applied to an augmented reality system, using a mobile camera and several stationary tracking sensors, and can be easily extended to the case of any additional sensor. In order to solve the initialization problem, we adapt, modify and integrate advanced techniques such as plenoptic viewing, intensity-based registration, and ICP. Thereby, the registration error is minimized in 3D object space rather than in 2D image. Experimental results show how complex objects can be registered efficiently and accurately to a single image.

Augmented reality (AR) provides a natural interface to the "calm" pervasive technology anticipated in large-scale ubiquitous computing environments. However, the range of classic AR applications has been limited by the scope, range and cost of sensors used for tracking. Hybrid tracking approaches can go some way to extending this range. We propose an approach, called ubiquitous tracking, in which data from widespread and diverse heterogeneous tracking sensors is automatically and dynamically fused, and then transparently provided to applications. A formal model represents spatial relationships between objects as a graph attributed with quality-of-service parameters. This paper presents a software implementation, in which a dynamic data flow network of distributed software components is thereby constructed in response to queries and optimisation criteria specified by applications. This implementation is demonstrated using a small laboratory example, and larger setups modelled in a simulation environment.

We present a workbench to build, evaluate and iteratively develop user interfaces using augmented reality, eye tracking, and visual programming. To test our system, we have developed an attentive user interface (AUI) for automotive environments, which coordinates its activities based on the context and visual attention of the user. Development of AUI requires interdisciplinary teams like psychologists, human-factor engineers, designers and computer scientists to work together. The main problem of interdisciplinary communication is a lack of common language and different notion of the system. We have developed a workbench, which facilitates the communication between the team members and enhances the comprehension of the system by visualizing users' attention and system reactions.

Augmented reality research currently aims at extending the working range of applications by combining multiple trackers in adjacent areas. The transition between two such devices leads to discontinuities in the trajectory of a tracked object. The result are "jumps" in visual augmentations shown to the user. We present a three step unsupervised learning algorithm that determines the working areas of involved trackers and the area they overlap, and permanently observes the tracked object's position with regard to these areas in order to enable a smooth interpolation within the overlapping area from one tracker's readings to another's. We have tested the algorithm's performance in an experimental setup. The results show that the method is feasible and only adds a negligible overhead to AR systems.

In the past, architectures of augmented reality systems have been widely different and tailored to specific tasks. In this paper, we use the example of the SHEEP game to show how the structural flexibility of DWARF, our component-based distributed wearable augmented reality framework, facilitates a rapid prototyping and online development process for building, debugging and altering a complex, distributed, highly interactive AR system. The SHEEP system was designed to test and demonstrate the potential of tangible user interfaces which dynamically visualize, manipulate and control complex operations of many inter-dependent processes. SHEEP allows the users more freedom of action and forms of interaction and collaboration, following the tool metaphor that bundles software with hardware in units that are easily understandable to the user. We describe how we developed SHEEP, showing the combined evolution of framework and application, as well as the progress from rapid prototype to final demonstration system. The dynamic aspects of DWARF facilitated testing and allowed us to rapidly evaluate new technologies. SHEEP has been shown successfully at various occasions. We describe our experiences with these demos.

This demo shows a robust model-based tracker using stereovision. The combined use of a 3D model with stereoscopic analysis allows accurate pose estimation in the presence of partial occlusions by non rigid objects like the hands of the user. Furthermore, using a second camera improves the stability of tracking and also simplifies the algorithm.

Research prototypes in AR usually do not emphasize software architecture. Nevertheless, their architecture is not arbitrary, but results from specific needs. Architectural approaches embodying research contributions are of particular value for reuse at component and architectural levels. We have conducted a study of existing AR software architectures for the ARVIKA project by W. Friedrich et al (2001). This has resulted in a catalog of important desired quality attributes for AR systems, a reference architecture for comparison of AR architectures, and a catalog of architectural approaches used in current AR systems. We believe this lays the foundation for further research in AR software architectures.

Mobile augmented reality applications promise substantial savings in time and costs for product designers, in particular, for large products requiring scale models and expensive clay mockups (e.g., cars). Such applications are novel and introduce interesting challenges when attempting to describe them to potential users and stakeholders. For example, it is difficult, a priori, to assess the nonfunctional requirements of such applications and anticipate the usability issues that the product designers are likely to raise. In this paper, we describe our efforts to develop a proof-of-concept AR system for car designers. Over the course of a year, we developed two prototypes, one within a university context, the other at a car manufacturer. The lessons learned from both efforts illustrate the technical and human challenges encountered when closely collaborating with the end user in the design of a novel application.

Augmented Reality (AR), mobile computing, and ubiquitous computing have each individually provided technology enablers for new types of applications. Researchers have demonstrated over the years spectacular toy systems that fire researchers' imaginations. However, cross-fertilization between these areas has only just begun. Combining these technology enablers into robust information systems that can be used in the field present many software engineering challenges that have not been taken into account by the toy systems mentioned above. In this paper, we report our efforts at TU Munich to analyze information generation, retrieval, transmission, and visualization in the context of maintenance procedures in nuclear powerplants. The use of AR to present such information online has significant implications on the overall system architecture. This paper focuses on pointing out open questions, discussing options for addressing them, and evaluating them in prototypical implementations.

Augmented reality deals with the problem of dynamically augmenting or enhancing (images or live video of) the real world with computer generated data (e.g., graphics of virtual objects). This poses two major problems: (a) determining the precise alignment of real and virtual coordinate frames for overlay, and (b) capturing the 3D environment including camera and object motions. The latter is important for interactive augmented reality applications where users can interact with both real and virtual objects. Here we address the problem of accurately tracking the 3D motion of a monocular camera in a known 3D environment and dynamically estimating the 3D camera location. We utilize fully automated landmark-based camera calibration to initialize the motion estimation and employ extended Kalman filter techniques to track landmarks and to estimate the camera location. The implementation of our approach has been proven to be efficient and robust and our system successfully tracks in real-tim...

This paper presents an environment for telecollaborative data exploration. It provides the following capabilities essential to data exploration: (1) users can probe the data, defining regions of interest with arbitrary shapes. (2) The selected data can be transformed and displayed in many different ways. (3) Linked cursors can be established between several windows showing data sets with arbitrary relationships. (4) Data can be displayed on any screen across a computer network, allowing for telecollaboration arrangements with linked cursors around the world. (5) Our system is user-extensible, allowing programmers to change any component of it while keeping the remaining functionality. We demonstrate how the system can be used in several applications, such as biomedical imaging, robotics, and wood classification.>

In this paper, we present an approach to color image understanding that can be used to segment and analyze surfaces with color variations due to highlights and shading. We begin with a theory - the Dichromatic Reflection Model - that relates the reflected light from dielectric materials, such as plastic, to fundamental physical reflection processes, and describes the color of the reflected light as a linear combination of the color of the light due to surface reflection (highlights) and body reflection (object color). This dichromatic theory is used in an algorithm that separates a color image into two parts: an image of just the highlights, and the original image with the highlights removed. In the past, we have applied this method to handsegmented images. This paper shows how to perform automatic segmentation by applying this theory in stages to identify the object and highlight colors. The result is a combination of segmentation and reflection analysis that is better than traditional heuristic segmentation methods and provides important physical information about the surface geometry and material properties at the same time. This line of research can lead to physics-based image understanding methods that are both more reliable and more useful than traditional methods.