Abstract Rendering in real time for virtual reality headsets with high user immersion is challenging due to strict framerate constraints as well as due to a low tolerance for artefacts. Eye tracking‐based foveated rendering presents an opportunity to strongly increase performance without loss of perceived visual quality. To this end, we propose a novel foveated rendering method for virtual reality headsets with integrated eye tracking hardware. Our method comprises recycling pixels in the periphery by spatio‐temporally reprojecting them from previous frames. Artefacts and disocclusions caused by this reprojection are detected and re‐evaluated according to a confidence value that is determined by a newly introduced formalized perception‐based metric, referred to as confidence function. The foveal region, as well as areas with low confidence values, are redrawn efficiently, as the confidence value allows for the delicate regulation of hierarchical geometry and pixel culling. Hence, the average primitive processing and shading costs are lowered dramatically. Evaluated against regular rendering as well as established foveated rendering methods, our approach shows increased performance in both cases. Furthermore, our method is not restricted to static scenes and provides an acceleration structure for post‐processing passes.

Common VR headsets require lenses that increase the field of view and allow the user to focus the display. In most systems, thick lenses are used that generate strong pincushion distortions. To account for this, the content is warped by a corresponding barrel distortion before displaying, resulting in undistorted images for the viewer. This approach assumes that the eye is exactly positioned, typically on the optical axis of the lens. However, in real systems the eye's location deviates from this optimal position even at rest; moreover, the pupil - and thus the optical center of the eye - moves by several millimeters when the user looks around. Thus, eye movement results in additional distortion, which is ignored in current VR headsets. Also in literature on head-mounted displays, the effect is most often not considered, or at least badly documented. The contribution of this paper are experiments that emphasize the importance of this mostly ignored effect. To this end, we have built a simple setup with a camera at variable eye positions in a standard VR headset, that allows us to directly measure the variation of distortion during eye movement. Our experiments show that distortion varies by several dozens of pixels within the full range of eye movements, which emphasizes that the effect is definitely significant. We also demonstrate how in a headset with built-in eye tracker, the knowledge of the eye position can be used to achieve a view-dependent lens correction with only minimal additional effort at run-time.

Abstract Monte‐Carlo path tracing techniques can generate stunning visualizations of medical volumetric data. In a clinical context, such renderings turned out to be valuable for communication, education, and diagnosis. Because a large number of computationally expensive lighting samples is required to converge to a smooth result, progressive rendering is the only option for interactive settings: Low‐sampled, noisy images are shown while the user explores the data, and as soon as the camera is at rest the view is progressively refined. During interaction, the visual quality is low, which strongly impedes the user's experience. Even worse, when a data set is explored in virtual reality, the camera is never at rest, leading to constantly low image quality and strong flickering. In this work we present an approach to bring volumetric Monte‐Carlo path tracing to the interactive domain by reusing samples over time. To this end, we transfer the idea of temporal antialiasing from surface rendering to volume rendering. We show how to reproject volumetric ray samples even though they cannot be pinned to a particular 3D position, present an improved weighting scheme that makes longer history trails possible, and define an error accumulation method that downweights less appropriate older samples. Furthermore, we exploit reprojection information to adaptively determine the number of newly generated path tracing samples for each individual pixel. Our approach is designed for static, medical data with both volumetric and surface‐like structures. It achieves good‐quality volumetric Monte‐Carlo renderings with only little noise, and is also usable in a VR context.

The computation of partial occlusion, as required for ambient occlusion or soft shadows, provides visually important cues but is notoriously expensive. In this paper we propose a novel solution to the ambient occlusion problem, combining signed distance scene representations and machine learning. We demonstrate how to learn and apply mappings which approximate a ray traced ground truth occlusion using only a few nearby samples of a signed distance representation. As representation for our trained mappings we use small feed-forward neural networks which are fast to evaluate, allowing for real-time occlusion queries. Our ambient occlusion approximation outperforms state-of-the-art methods in both quality and performance, yielding temporally stable and smooth results. Since our training data is different from typical machine learning approaches which mostly deal with 2D/3D image data and our techniques are also applicable to other occlusion problems (e.g. soft shadows), we give an in-depth overview of our framework. Furthermore, we discuss arising artifacts and possible extensions of our approach.