Advancements in 3D rendering like Gaussian Splatting (GS) allow novel view synthesis and real-time rendering in virtual reality (VR). However, GS-created 3D environments are often difficult to edit. For scene enhancement or to incorporate 3D assets, segmenting Gaussians by class is essential. Existing segmentation approaches are typically limited to certain types of scenes, e.g., "circular" scenes, to determine clear object boundaries. However, this method is ineffective when removing large objects in non-"circling" scenes such as large outdoor scenes.We propose Semantics-Controlled GS (SCGS), a segmentation-driven GS approach, enabling the separation of large scene parts in uncontrolled, natural environments. SCGS allows scene editing and the extraction of scene parts for VR. Additionally, we introduce a challenging outdoor dataset, overcoming the "circling" setup. We outperform the state-of-the-art in visual quality on our dataset and in segmentation quality on the 3D-OVS dataset. We conducted an exploratory user study, comparing a 360-video, plain GS, and SCGS in VR with a fixed viewpoint. In our subsequent main study, users were allowed to move freely, evaluating plain GS and SCGS. Our main study results show that participants clearly prefer SCGS over plain GS. We overall present an innovative approach that surpasses the state-of-the-art both technically and in user experience.

We are pleased to present the proceedings of the 31st IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2024), held March 16–21, 2024, in Orlando, Florida, US, in on-site format. These proceedings contain 101 of the 180 IEEE VR 2024 papers, with the other 79 papers being published in the IEEE VR 2024 special issue of the Transactions on Visualization and Computer Graphics. IEEE VR 2024 had 627 submissions for an acceptance rate of 101/627=16.1% as conference papers, or 180/627=28.7% when including the journal papers.

In this special issue ofIEEE Transactions on Visualization and Computer Graphics(TVCG), we are pleased to present the top papers from the 31th IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2024), held March 16-21, 2024, in Orlando, Florida, US.

Abstract In augmented reality applications it is essential to know the position and orientation of the user to correctly register virtual 3D content in the user’s field of view. For this purpose, visual tracking through simultaneous localization and mapping (SLAM) is often used. However, when applied to the commonly occurring situation where the users are mostly stationary, many methods presented in previous research have two key limitations. First, SLAM techniques alone do not address the problem of global localization with respect to prior models of the environment. Global localization is essential in many applications where multiple users are expected to track within a shared space, such as spectators at a sporting event. Secondly, these methods often assume significant translational movement to accurately reconstruct and track from a local model of the environment, causing challenges for many stationary applications. In this paper, we extend recent research on Spherical Localization and Tracking to support relocalization after tracking failure, as well as global localization in large shared environments, and optimize the method for operation on mobile hardware. We also evaluate various state-of-the-art localization approaches, the robustness of our visual tracking method, and demonstrate the effectiveness of our system in real-life scenarios.

Head tracking in head-mounted displays (HMDs) enables users to explore a 360-degree virtual scene with free head movements. However, for seated use of HMDs such as users sitting on a chair or a couch, physically turning around 360-degree is not possible. Redirection techniques decouple tracked physical motion and virtual motion, allowing users to explore virtual environments with more flexibility. In seated situations with only head movements available, the difference of stimulus might cause the detection thresholds of rotation gains to differ from that of redirected walking. Therefore we present an experiment with a two-alternative forced-choice (2AFC) design to compare the thresholds for seated and standing situations. Results indicate that users are unable to discriminate rotation gains between 0.89 and 1.28, a smaller range compared to the standing condition. We further treated head amplification as an interaction technique and found that a gain of 2.5, though not a hard threshold, was near the largest gain that users consider applicable. Overall, our work aims to better understand human perception of rotation gains in seated VR and the results provide guidance for future design choices of its applications.

Redirected walking (RDW) allows users to explore virtual environments in limited physical spaces by imperceptibly steering them away from obstacles and space boundaries. However, even with those techniques, the risk of collision cannot always be avoided. For such situations, resetting techniques have been proposed to provide an immediate adjustment of the physical walking direction of a user. Existing resetting techniques are either applied in-place, where the user changes orientation but stays in the same position or out-of-place methods where the user is guided to move from the current position to a safe location all while freezing the movement in the virtual world. While out-of-place methods have the potential to provide more freedom to user movements after resetting, current out-of-place methods do not provide enough guidance for the users to move to optimal locations. In this work, we propose a novel out-of-place resetting strategy that guides users to optimal physical locations with the most potential for free movement and a smaller amount of resetting required for their further movements. For this purpose, we calculate a heat map of the walking area according to the average walking distance using a simulation of the currently used RDW algorithm. Based on this heat map, we identify the most suitable position for a one-step reset within a predefined searching range and use this one as the reset point. Our results show that our method increases the average moving distance within one cycle of resetting. Furthermore, our resetting method can be applied to any physical area with obstacles. That means that RDW methods that were not suitable for such environments (e.g., Steer to Center) combined with our resetting can also be extended to such complex walking areas. In addition, we present a user interface to provide a similar visual experience between these methods, using a two-arrows indicator to help users adjust their position and direction.

We investigate how real-time, 360° view synthesis can be achieved on current virtual reality hardware from a single panoramic image input. We introduce a light-weight method to automatically convert a single panoramic input into a multi-cylinder image representation that supports real-time, free-viewpoint view synthesis rendering for virtual reality. We apply an existing convolutional neural network trained on pinhole images to a cylindrical panorama with wrap padding to ensure agreement between the left and right edges. The network outputs a stack of semi-transparent panoramas at varying depths which can be easily rendered and composited with over blending. Quantitative experiments and a user study show that the method produces convincing parallax and fewer artifacts than a textured mesh representation.

Augmented Reality has traditionally been used to display digital overlays in real environments. Many AR applications such as remote collaboration, picking tasks, or navigation require highlighting physical objects for selection or guidance. These highlights use graphical cues such as outlines and arrows. Whilst effective, they greatly contribute to visual clutter, possibly occlude scene elements, and can be problematic for long-term use. Substituting those overlays, we explore saliency modulation to accentuate objects in the real environment to guide the user’s gaze. Instead of manipulating video streams, like done in perception and cognition research, we investigate saliency modulation of the real world using optical-see-through head-mounted displays. This is a new challenge, since we do not have full control over the view of the real environment. In this work we provide our specific solution to this challenge, including built prototypes and their evaluation.

Rapidly developing Redirected Walking (ROW) technologies have enabled VR applications to immerse users in large virtual environments (VE) while actually walking in relatively small physical environments (PE). When an unavoidable collision emerges in a PE, the ROW controller suspends the user's immersive experience and resets the user to a new direction in PE. Existing ROW methods mainly aim to reduce the number of resets. However, from the perspective of the user experience, when users are about to reach a point of interest (POI) in a VE, reset interruptions are more likely to have an impact on user experience. In this paper, we propose a new ROW method, aiming to keep resets occurring at a longer distance from the virtual target, as well as to reduce the number of resets. Simulation experiments and real user studies demonstrate that our method outperforms state-of-the-art ROW methods in the number of resets and dramatically increases the distance between the reset locations and the virtual targets.

Coherent rendering is important for generating plausible Mixed Reality presentations of virtual objects within a user’s real-world environment. Besides photo-realistic rendering and correct lighting, visual coherence requires simulating the imaging system that is used to capture the real environment. While existing approaches either focus on a specific camera or a specific component of the imaging system, we introduce Neural Cameras, the first approach that jointly simulates all major components of an arbitrary modern camera using neural networks. Our system allows for adding new cameras to the framework by learning the visual properties from a database of images that has been captured using the physical camera. We present qualitative and quantitative results and discuss future direction for research that emerge from using Neural Cameras.

The maturity of augmented reality (AR) technology and research now paves the way for dissemination of AR outside of the laboratory. However, it is still under-explored which factors are influencing the user experience of an AR application. In this poster, we describe some of the technical factors that could influence the user experience. We focus on a use-case in the field of on-site sports spectating with mobile AR. We present a study design which analyzes the influence of latency, registration accuracy, and jitter as factors on AR user experience.

In-stadium sports spectating delivers a unique social experience in a variety of sports. However, in contrast to broadcast delivery, it lacks the provision of real-time information augmentation, like game statistics overlaid on screen. In an earlier iteration, we developed ARSpectator, a prototypical, mobile system which can be brought to the stadium to experience both, the live sport action and situated infographics spatially augmented into the scene. In some situations it is difficult or often impossible to go to the stadium though, for instance because of limited stadium access during pandemics or when wanting to conduct controlled user studies. We address this by turning our ARSpectator system into an indirect augmented reality experience deployed to an immersive, virtual reality head-mounted display: The live stadium experience is delivered by way of a surrounding 360 video recording while maintaining and extending the provision of interactive, situated infographics. With our XRSpectator demo prototype presented here, users can have an ARSpectator experience of a rugby game in our local stadium.

We present computational phase-modulated eyeglasses, a see-through optical system that modulates the view of the user using phase-only spatial light modulators (PSLM). A PSLM is a programmable reflective device that can selectively retardate, or delay, the incoming light rays. As a result, a PSLM works as a computational dynamic lens device. We demonstrate our computational phase-modulated eyeglasses with either a single PSLM or dual PSLMs and show that the concept can realize various optical operations including focus correction, bi-focus, image shift, and field of view manipulation, namely optical zoom. Compared to other programmable optics, computational phase-modulated eyeglasses have the advantage in terms of its versatility. In addition, we also presents some prototypical focus-loop applications where the lens is dynamically optimized based on distances of objects observed by a scene camera. We further discuss the implementation, applications but also discuss limitations of the current prototypes and remaining issues that need to be addressed in future research.

In recent years, the development of Augmented Reality (AR) frameworks made AR application development widely accessible to developers without AR expert background. With this development, new application fields for AR are on the rise. This comes with an increased need for visualization techniques that are suitable for a wide range of application areas. It becomes more important for a wider audience to gain a better understanding of existing AR visualization techniques. In this article we provide a taxonomy of existing works on visualization techniques in AR. The taxonomy aims to give researchers and developers without an in-depth background in Augmented Reality the information to successively apply visualization techniques in Augmented Reality environments. We also describe required components and methods and analyze common patterns.

In this special issue of IEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present the TVCG papers from the 19th IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2020), which had been originally planned to hold in Recife/Porto de Galinhas, Brazil. In order to preserve the safety and well-being of all participants under the global pandemic of COVID-19, ISMAR 2020 will be held as a virtual conference between November 9 and 13, 2020. ISMAR continues the over twenty year long tradition of IWAR, ISMR, and ISAR, and is undoubtedly the premier conference for Mixed and Augmented Reality in the world.

Hand-held capture of stereo panoramas involves spinning the camera in a roughly circular path to acquire a dense set of views of the scene. However, most existing structure-from-motion pipelines fail when trying to reconstruct such trajectories, due to the small baseline between frames. In this work, we evaluate the use of spherical structure-from-motion for reconstructing handheld stereo panorama captures. The spherical motion constraint introduces a strong regularization on the structure-from-motion process which mitigates the small-baseline problem, making it well-suited to the use case of stereo panorama capture with a handheld camera. We demonstrate the effectiveness of spherical structure-from-motion for casual capture of high-resolution stereo panoramas and validate our results with a user study.

When implementing an Augmented Reality (AR) interface, it is essential to track camera motion in order to precisely register the virtual overlay in the view of the user. However, unlike most indoor AR scenarios, in many outdoor scenarios the user maintains a static position performing mostly rotational movements. Simultaneous Localization and Mapping (SLAM) methods typically used to solve the tracking problem require significant translational camera motion to perform reliably. The magnitude of the required translation is proportional to the size of the scene, exacerbating this problem in large environments such as open places or stadiums. In this paper, we present an alternative SLAM method, which combines spherical Structure-from-Motion and a robust 3D tracking method. We compare our method to ORB SLAM2 in synthetic and real tests, and show that our method can track more reliably in large spaces, with simpler calculation due to the spherical motion constraint. We discuss this issue in the context of implementing an AR interface for live sport events in stadiums or other open environments, but possible application scenarios for our technique go beyond and can be applied to handheld AR in many outdoor environments.

Thanks to the ubiquity of devices capable of recording and playing back video, the amount of video files is growing at a rapid rate. Most of us have now video recordings of major events in our lives. However, until today, these videos are captured mainly in 2D and are mostly used for screen-based video replay. Currently there is no way for watching them in more immersive environments such as on a VR headset. They are simply not optimized for playback in stereoscopic displays or even tracked Virtual Reality devices.

In this special issue ofIEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present theTVCGpapers from the 19th IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2020), which had been originally planned to hold in Recife/Porto de Galinhas, Brazil. In order to preserve the safety and well-being of all participants under the global pandemic of COVID-19, ISMAR 2020 will be held as a virtual conference between November 9 and 13, 2020. ISMAR continues the over twenty year long tradition of IWAR, ISMR, and ISAR, and is undoubtedly the premier conference for Mixed and Augmented Reality in the world.

Hand-held capture of stereo panoramas involves spinning the camera in a roughly circular path to acquire a dense set of views of the scene. However, most existing structure-from-motion pipelines fail when trying to reconstruct such trajectories, due to the small baseline between frames. In this work, we propose to use spherical structure-from-motion for reconstructing handheld stereo panorama captures. Our initial results show that spherical motion constraints are critical for reconstructing small-baseline, circular trajectories.

In this special issue ofIEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present theTVCGpapers from the 18th IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2019), held October 14–18 in Beijing, China. ISMAR continues the over 20-year long tradition of IWAR, ISMR, and ISAR, and is undoubtedly the premier conference for mixed and augmented reality in the world.

Prescription glasses are used by many people as a simple, and even fashionable way, to correct refractive problems of the eye. However, there are other visual impairments that cannot be treated with an optical lens in conventional glasses. In this work we present ChromaGlasses, Computational Glasses using optical head-mounted displays for compensating colour vision deficiency. Unlike prior work that required users to look at a screen in their visual periphery rather than at the environment directly, ChromaGlasses allow users to directly see the environment using a novel head-mounted displays design that analyzes the environment in real-time and changes the appearance of the environment with pixel precision to compensate the impairment of the user. In this work, we present first prototypes for ChromaGlasses and report on the results from several studies showing that ChromaGlasses are an effective method for managing colour blindness.

Augmented Reality is a technique that enables users to interact with their physical environment through the overlay of digital information. While being researched for decades, more recently, Augmented Reality moved out of the research labs and into the field. While most of the applications are used sporadically and for one particular task only, current and future scenarios will provide a continuous and multi-purpose user experience. Therefore, in this paper, we present the concept of Pervasive Augmented Reality, aiming to provide such an experience by sensing the user's current context and adapting the AR system based on the changing requirements and constraints. We present a taxonomy for Pervasive Augmented Reality and context-aware Augmented Reality, which classifies context sources and context targets relevant for implementing such a context-aware, continuous Augmented Reality experience. We further summarize existing approaches that contribute towards Pervasive Augmented Reality. Based our taxonomy and survey, we identify challenges for future research directions in Pervasive Augmented Reality.

Augmented reality (AR) allows for an on-site presentation of information that is registered to the physical environment. Applications from civil engineering, which require users to process complex information, are among those which can benefit particularly highly from such a presentation. In this paper, we will describe how to use AR to support monitoring and documentation of construction site progress. For these tasks, the responsible staff usually requires fast and comprehensible access to progress information to enable comparison to the as-built status as well as to as-planned data. Instead of tediously searching and mapping related information to the actual construction site environment, our AR system allows for the access of information right where it is needed. This is achieved by superimposing progress as well as as-planned information onto the user's view of the physical environment. For this purpose, we present an approach that uses aerial 3-D reconstruction to automatically capture progress information and a mobile AR client for on-site visualization. Within this paper, we will describe in greater detail how to capture 3-D, how to register the AR system within the physical outdoor environment, how to visualize progress information in a comprehensible way in an AR overlay, and how to interact with this kind of information. By implementing such an AR system, we are able to provide an overview about the possibilities and future applications of AR in the construction industry.

Micro aerial vehicles equipped with high-resolution cameras can be used to create aerial reconstructions of an area of interest. In that context automatic flight path planning and autonomous flying is often applied but so far cannot fully replace the human in the loop, supervising the flight on-site to assure that there are no collisions with obstacles. Unfortunately, this workflow yields several issues, such as the need to mentally transfer the aerial vehicle’s position between 2D map positions and the physical environment, and the complicated depth perception of objects flying in the distance. Augmented Reality can address these issues by bringing the flight planning process on-site and visualizing the spatial relationship between the planned or current positions of the vehicle and the physical environment. In this paper, we present Augmented Reality supported navigation and flight planning of micro aerial vehicles by augmenting the user’s view with relevant information for flight planning and live feedback for flight supervision. Furthermore, we introduce additional depth hints supporting the user in understanding the spatial relationship of virtual waypoints in the physical world and investigate the effect of these visualization techniques on the spatial understanding.

In Augmented Reality (AR), ghosted views allow a viewer to explore hidden structure within the real-world environment. A body of previous work has explored which features are suitable to support the structural interplay between occluding and occluded elements. However, the dynamics of AR environments pose serious challenges to the presentation of ghosted views. While a model of the real world may help determine distinctive structural features, changes in appearance or illumination detriment the composition of occluding and occluded structure. In this paper, we present an approach that considers the information value of the scene before and after generating the ghosted view. Hereby, a contrast adjustment of preserved occluding features is calculated, which adaptively varies their visual saliency within the ghosted view visualization. This allows us to not only preserve important features, but to also support their prominence after revealing occluded structure, thus achieving a positive effect on the perception of ghosted views.

In this paper we present an approach for visualizing time-oriented data of dynamic scenes in an on-site AR view. Visualizations of time-oriented data have special challenges compared to the visualization of arbitrary virtual objects. Usually, the 4D data occludes a large part of the real scene. Additionally, the data sets from different points in time may occlude each other. Thus, it is important to design adequate visualization techniques that provide a comprehensible visualization. In this paper we introduce a visualization concept that uses overview and detail techniques to present 4D data in different detail levels. These levels provide at first an overview of the 4D scene, at second information about the 4D change of a single object and at third detailed information about object appearance and geometry for specific points in time. Combining the three levels of detail with interactive transitions such as magic lenses or distorted viewing techniques enables the user to understand the relationship between them. Finally we show how to apply this concept for construction site documentation and monitoring.

A convincing combination of virtual and real data in an Augmented Reality (AR) application requires detailed 3D information about the real world scene. In many situations extensive model data is not available, while sparse representations such as outlines on a map exist. In this paper, we present a novel approach using such sparse 3D model data to seed automatic image segmentation and infer a dense depth map of an environment. Sparse 3D models of known landmarks, such as points and lines from GIS databases, are projected into a registered image and initialize 2D image segmentation at the projected locations in the image. For the segmentation we propose different techniques, which combine shape information, semantics given by the database, and the visual appearance in the referenced image. The resulting depth information of objects in the scene can be used in many applications, including occlusion handling, label placement, and 3D modeling.

Summary form only given. We report on a disruption in organizational dynamics arising from the introduction of model-driven development tools in General Motors. The introduction altered the balance of collaboration deeply, and the organization is still negotiating with its aftermath. Our report illustrates one consequence of tool adoption in groups, and that these consequences should be understood to facilitate technical change.

This paper investigates possible physical alterations of tracking targets to obtain improved 6DoF pose detection for a camera observing the known targets. We explore the influence of several texture characteristics on the pose detection, by simulating a large number of different target objects and camera poses. Based on statistical observations, we rank the importance of characteristics such as texturedness and feature distribution for a specific implementation of a 6DoF tracking technique. These findings allow informed modification strategies for improving the tracking target objects themselves, in the common case of man-made targets, as for example used in advertising. This fundamentally differs from and complements the traditional approach of leaving the targets unchanged while trying to optimize the tracking algorithms and parameters.

We describe the design and implementation of a physical and virtual model of an imaginary urban scene-the “City of Sights”- that can serve as a backdrop or “stage” for a variety of Augmented Reality (AR) research. We argue that the AR research community would benefit from such a standard model dataset which can be used for evaluation of such AR topics as tracking systems, modeling, spatial AR, rendering tests, collaborative AR and user interface design. By openly sharing the digital blueprints and assembly instructions for our models, we allow the proposed set to be physically replicable by anyone and permit customization and experimental changes to the stage design which enable comprehensive exploration of algorithms and methods. Furthermore we provide an accompanying rich dataset consisting of video sequences under varying conditions with ground truth camera pose. We employed three different ground truth acquisition methods to support a broad range of use cases. The goal of our design is to enable and improve the replicability and evaluation of future augmented reality research.

In augmented reality displays, X-Ray visualization techniques make hidden objects visible through combining the physical view with an artificial rendering of the hidden information. An important step in X-Ray visualization is to decide which parts of the physical scene should be kept and which should be replaced by overlays. The combination should provide users with essential perceptual cues to understand the relationship of depth between hidden information and the physical scene. In this paper we present an approach that addresses this decision in unknown environments by analyzing camera images of the physical scene and using the extracted information for occlusion management. Pixels are grouped into perceptually coherent image regions and a set of parameters is determined for each region. The parameters change the X-Ray visualization for either preserving existing structures or generating synthetic structures. Finally, users can customize the overall opacity of foreground regions to adapt the visualization.

In this work we present a novel idea of evaluating natural feature-point based tracking targets. Our main objective is to evaluate the inherent characteristics of natural feature-point sets with respect to vision-based pose estimation algorithms. Our work attempts to break new ground by concentrating on evaluating complete tracking targets, rather than evaluating tracking methods or single features. This allows deriving indications on how to improve the trackability of natural feature point sets.