In recent years, the development of Augmented Reality (AR) frameworks made AR application development widely accessible to developers without AR expert background. With this development, new application fields for AR are on the rise. This comes with an increased need for visualization techniques that are suitable for a wide range of application areas. It becomes more important for a wider audience to gain a better understanding of existing AR visualization techniques. In this article we provide a taxonomy of existing works on visualization techniques in AR. The taxonomy aims to give researchers and developers without an in-depth background in Augmented Reality the information to successively apply visualization techniques in Augmented Reality environments. We also describe required components and methods and analyze common patterns.

Augmented Reality (AR) applications require knowledge about the real world environment in which they are used. This knowledge is often gathered while developing the AR application and stored for future uses of the application. Consequently, changes to the real world lead to a mismatch between the previously recorded data and the real world. New capturing techniques based on dense Simultaneous Localization and Mapping (SLAM) not only allow users to capture real world scenes at run-time, but also enables them to capture changes of the world. However, instead of using previously recorded and prepared scenes, users must interact with an unprepared environment. In this paper, we present a set of new interaction techniques that support users in handling captured real world environments. The techniques present virtual viewpoints of the scene based on a scene analysis and provide natural transitions between the AR view and virtual viewpoints. We demonstrate our approach with a SLAM based prototype that allows us to capture a real world scene and describe example applications of our system.

Annotations of objects in 3D environments are commonly controlled using view management techniques. State-of-the-art view management strategies for external labels operate in 2D image space. This creates problems, because the 2D view of a 3D scene changes over time, and temporal behavior of elements in a 3D scene is not obvious in 2D image space. We propose managing the placement of external labels in 3D object space instead. We use 3D geometric constraints to achieve label placement that fulfills the desired objectives (e.g., avoiding overlapping labels), but also behaves consistently over time as the viewpoint changes. We propose two geometric constraints: a 3D pole constraint, where labels move along a 3D pole sticking out from the annotated object, and a plane constraint, where labels move in a dominant plane in the world. This formulation is compatible with standard optimization approaches for labeling, but overcomes the lack of temporal coherence.

As low-level hardware will soon allow us to visualize virtual content anywhere in the real world, managing it in a more structured manner still needs to be addressed. Augmented reality (AR) browser technology is the gateway to such structured software platform and an anywhere AR user experience. AR browsers are the substitute of Web browsers in the real world, permitting overlay of interactive multimedia content on the physical world or objects they refer to. As the current generation allows us to barely see floating virtual items in the physical world, a tighter coupling with our reality has not yet been explored. This paper presents our recent effort to create rich, seamless, and adaptive AR browsers. We discuss major challenges in the area and present an agenda on future research directions for an everyday augmented world.

The development of new Simultaneous Localization and Mapping (SLAM) techniques is quickly advancing in research communities and rapidly transitioning into commercial products. Creating accurate and high-quality SLAM maps relies on a robust initialization process. However, the robustness and usability of SLAM initialization for end-users has often been disregarded. This paper presents and evaluates a novel tracking system for 6DOF pose tracking between a single keyframe and the current camera frame, without any prior scene knowledge. Our system is particularly suitable for SLAM initialization, since it allows 6DOF pose tracking in the intermediate frames before a wide-enough baseline between two keyframes has formed. We investigate how our tracking system can be used to interactively guide users in performing an optimal motion for SLAM initialization. However, our findings from a pilot study indicate that the need for such motion can be completely hidden from the user and outsourced to our tracking system. Results from a second user study show that letting our tracking system create a SLAM map as soon as possible is a viable and usable solution. Our work provides important insight for SLAM systems, showing how our novel tracking system can be integrated with a user interface to support fast, robust and user-friendly SLAM initialization.

In-situ 3D Modeling becomes increasingly prominent in current Augmented Reality research, particularly for mobile scenarios. However, real-time performance and qualitative modeling remain highly challenging. In this work, we propose a new interactive 3D modeling approach for indoor environments, combining an assistive user interface and constrained reconstruction with a device consisting of a single-point laser range finder and a camera. Using our system, a user pans around capturing a panorama of the environment, while simultaneously measuring the distance to a single point per frame. An automatic detection process estimates planes from these sparse 3D measurements. The user can highlight specific geometric features in the environment, such as 2- or 3-way corners, with simple gestures, adding more 3D points to the estimation. The segmented planes are refined using a constrained optimization, enforcing orthogonality and parallel constraints as well as minimizing the number of planes used in the reconstruction. Finally a volumetric space-carving approach determines the geometry of the environment. Our reconstruction approach can output highly accurate models built only from simple, clean geometry. To examine the quantitative performance of our approach, we run evaluations on both synthetic and real data.

We are delighted to present the 12th edition of the demonstration program of the IEEE International Symposium on Mixed and Augmented Reality. The ISMAR demonstration program provide hands-on experience to the community of the most recent technical and applied advancement in Augmented Reality.

In this paper, we introduce a novel view management technique for placing labels in Augmented Reality systems. A common issue in many Augmented Reality applications is the absence of knowledge of the real environment, limiting the efficient representation and optimal layout of the digital information augmented onto the real world. To overcome this problem, we introduce an image-based approach, which combines a visual saliency algorithm with edge analysis to identify potentially important image regions and geometric constraints for placing labels. Our proposed solution also includes adaptive rendering techniques that allow a designer to control the appearance of depth cues. We describe the results obtained from a user study considering different scenarios, which we performed for validating our approach. Our technique will provide special benefits to Augmented Reality browsers that usually lack scene knowledge, but also to many other applications in the domain of Augmented Reality such as cultural heritage and maintenance applications.

Real-time three-dimensional acquisition of real-world scenes has many important applications in computer graphics, computer vision and human-computer interaction. Inexpensive depth sensors such as the Microsoft Kinect allow to leverage the development of such applications. However, this technology is still relatively recent, and no detailed studies on its scalability to dense and view-independent acquisition have been reported. This paper addresses the question of what can be done with a larger number of Kinects used simultaneously. We describe an interference-reducing physical setup, a calibration procedure and an extension to the KinectFusion algorithm, which allows to produce high quality volumetric reconstructions from multiple Kinects whilst overcoming systematic errors in the depth measurements. We also report on enhancing image based visual hull rendering by depth measurements, and compare the results to KinectFusion. Our system provides practical insight into achievable spatial and radial range and into bandwidth requirements for depth data acquisition. Finally, we present a number of practical applications of our system.

In this paper, we explore techniques that aim to improve site understanding for outdoor Augmented Reality (AR) applications. While the first person perspective in AR is a direct way of filtering and zooming on a portion of the data set, it severely narrows overview of the situation, particularly over large areas. We present two interactive techniques to overcome this problem: multi-view AR and variable perspective view. We describe in details the conceptual, visualization and interaction aspects of these techniques and their evaluation through a comparative user study. The results we have obtained strengthen the validity of our approach and the applicability of our methods to a large range of application domains.

In this paper we present importance-driven compositing window management, which considers windows not only as basic rectangular shapes but also integrates the importance of the windows' content using a bottom-up visual attention model. Based on this information, importance-driven compositing optimizes the spatial window layout for maximum visibility and interactivity of occluded content in combination with see-through windows. We employ this technique for emerging window manager functions to minimize information overlap caused by popping up windows or floating toolbars and to improve the access to occluded window content. An initial user study indicates that our technique provides a more effective and satisfactory access to occluded information than the well-adopted Alt+Tab window switching technique and see-through windows without optimized spatial layout.

This paper offers a contribution to an emerging culturally orientated discourse regarding mixed reality interaction. It seeks to analyse syncretic, hybridized agency, particularly in mixed reality data transfer systems. Recent developments in bridging autonomous relationships with digital representation through mixed reality interfacing, have brought about the need for further analysis of these new 'post-biological', hybridized states of being that traverse traditional paradigms of time and space. Roy Ascott's concept of syncretism may facilitate further understanding of multi-layered world views, both material and metaphysical, that are emerging from our engagement with such pervasive computational technologies and post-biological systems. Syncretism has traditionally been regarded as an attempt to harmonise and analogise [1] Citing recent examples of practical research outcomes, this paper will analyse what Gilles Deleuze and Fèlix Guattari have called 'deterritorialisation' of the human body through its dispersion throughout multiple reality manifestations and how mixed reality data transfer might constitute a 'reterritorialising' effect on syncretic post-biological digital identity construction [2].

Creating entertaining and educational books not only requires providing visually stimulating content but also means for students to interact, create, and express themselves. In this paper we present a new type of mixed-reality book experience, which augments an educational coloring book with user-generated three dimensional content. We explore a “pop-up book” metaphor and describe a process by which children's drawing and coloring is used as input to generate and change the appearance of the book content. Our system is based on natural feature tracking and image processing techniques that can be easily exploited for other AR publishing applications.

Nowadays, mobile devices provide new possibilities for gesture interaction due to the large range of embedded sensors they have and their physical form factor. In addition, auditory interfaces can now be more easily supported through advanced mobile computing capabilities. Although different types of gesture techniques have been proposed for handheld devices, there is still little knowledge about the acceptability and use of some of these techniques, especially in the context of an auditory interface. In this paper, we propose a novel approach to the problem by studying the design space of gestures proposed by end-users for a mobile auditory interface. We discuss the results of this explorative study, in terms of the scope of the gestures proposed, the tangible aspects, and the users' preferences. This study delivers some initial gestures recommendations for eyes-free auditory interfaces.

Sketching leverages human skills for various purposes. In-Place Augmented Reality Sketching experiences build on the intuitiveness and flexibility of hand sketching for tasks like content creation. In this paper we explore the design space of In-Place Augmented Reality Sketching, with particular attention to content authoring in games. We propose a contextual model that offers a framework for the exploration of this design space by the research community. We describe a sketch-based AR racing game we developed to demonstrate the proposed model. The game is developed on top of our shape recognition and 3D registration library for mobile AR.

We introduce a novel touch-based interaction technique for tangible user interfaces (TUIs) in Augmented Reality (AR) applications. The technique allows for direct access and manipulation of virtual content on a registered tracking target, is robust and lightweight, and can be applied in numerous tracking and interaction scenarios.

In this paper we present the results of our long term development of a mixed reality book. Most previous work in the area has focused on the technology of augmented reality books, such as providing registration using fiducial markers. In this work, however, we focused on exploring the design and development process of this type of application in a broader sense. We studied the semantics of a mixed reality book, the design space and the user experience with this type of interface.

As computers become more invisible, Augmented Reality (overlaying virtual images on the real world) is an increasingly important application area for computer graphics and user interface design. This tutorial will provide a detailed introduction to AR interface design and research, with reviews of topics as such as tracking and registration, interaction techniques, design principles, and usability evaluation, as well as areas for future AR research. Portions of the tutorial will be devoted to the ARToolKit and osgART open-source software tools for building AR applications. Participants will leave knowing how to build AR applications that provide a high quality user experience.

Recently researchers have shown that it is possible to use GPU hardware for image processing and computer vision algorithms. We have been exploring how to use GPU hardware to improve marker- based tracking for AR Applications. In this paper we describe our findings and explored issues in the context of a standard fiducial tracking pipeline. We demonstrate the implementation of a template matching process on the GPU and the performance improvement gained in comparison to a traditional CPU implementation.

In this paper we present the process of designing and developing a fully functional and interactive AR handheld device. Based on a human-centered approach we describe the results of our exploration; from the contextual usage to the different rapid prototyped models and the evaluation activities. We also discuss some recommendations for how to use similar methods for developing future new AR physical interfaces.

The Magic Lens concept is a focus and context technique which facilitates the visualization of complex and dense data. In this paper, we propose a new type of 3D tangible Magic Lens in the form of a flexible sheet. We describe new interaction techniques associated with this tool, and demonstrate how it can be applied in different AR applications.

Transitional Interfaces have emerged as a new way to interact and collaborate between different interactive spaces such as reality, virtual reality and augmented reality environments. In this paper we explore this concept further. We introduce a descriptive model of the concept, its collaborative aspect and how it can be generalized to describe natural and continuous transitions between contexts (e.g. across space, scale, viewpoints, and representation).

Recently augmented reality (AR) technology has been used to develop the next generation collaborative interfaces. First results have shown the value of using AR for co-located tasks based on egocentric viewpoints. In contrast, virtual reality (VR) seems to offer interesting advantages for immersive collaborative experiences with egocentric viewpoints. We focus on a new area: a mixed collaboration between AR and VR environments. We present a new conceptual model of transitional interfaces that allow users to move between AR and VR viewpoints. We then describe the results of a quantitative evaluation with an AR exocentric viewpoint and a VR egocentric viewpoint for a navigational task. We also conducted a second experiment on the impact of the relationship between the interaction and visualization space in mixed collaboration. Results of these studies can provide a better understanding of how to design interfaces for multispace and transitional collaboration.

We describe the AR Mask, a novel handheld augmented reality (AR) input and display device that not only provides support for a full range of traditional interaction techniques, but also facilitates new metaphors. Unlike other AR display and input technologies, our device consolidates input and output within a single piece of hardware. We demonstrate how this design provides a closed control loop between the user's input modalities and natural sensory receptors.

Mediated reality describes the concept of filtering or vision of reality, typically using a head-worn video mixing display. In this paper, we propose a generalized concept and new tools for interactively mediated reality. We present also our first prototype system for painting, grabbing and gluing together real and virtual elements.