The continuous advancement of photorealism in rendering is accompanied by a growth in texture data and, consequently, increasing storage and memory demands. To address this issue, we propose a novel neural compression technique specifically designed for material textures. We unlock two more levels of detail, i.e., 16× more texels, using low bitrate compression, with image quality that is better than advanced image compression techniques, such as AVIF and JPEG XL. At the same time, our method allows on-demand, real-time decompression with random access similar to block texture compression on GPUs, enabling compression on disk and memory. The key idea behind our approach is compressing multiple material textures and their mipmap chains together, and using a small neural network, that is optimized for each material, to decompress them. Finally, we use a custom training implementation to achieve practical compression speeds, whose performance surpasses that of general frameworks, like PyTorch, by an order of magnitude.

Abstract Despite recent advances in Monte Carlo path tracing at interactive rates, denoised image sequences generated with few samples per‐pixel often yield temporally unstable results and loss of high‐frequency details. We present a novel adaptive rendering method that increases temporal stability and image fidelity of low sample count path tracing by distributing samples via spatio‐temporal joint optimization of sampling and denoising. Adding temporal optimization to the sample predictor enables it to learn spatio‐temporal sampling strategies such as placing more samples in disoccluded regions, tracking specular highlights, etc; adding temporal feedback to the denoiser boosts the effective input sample count and increases temporal stability. The temporal approach also allows us to remove the initial uniform sampling step typically present in adaptive sampling algorithms. The sample predictor and denoiser are deep neural networks that we co‐train end‐to‐end over multiple consecutive frames. Our approach is scalable, allowing trade‐off between quality and performance, and runs at near real‐time rates while achieving significantly better image quality and temporal stability than previous methods.

Abstract Temporal Antialiasing (TAA), formally defined as temporally‐amortized supersampling, is the most widely used antialiasing technique in today's real‐time renderers and game engines. This survey provides a systematic overview of this technique. We first review the history of TAA, its development path and related work. We then identify the two main sub‐components of TAA, sample accumulation and history validation, and discuss algorithmic and implementation options. As temporal upsampling is becoming increasingly relevant to today's game engines, we propose an extension of our TAA formulation to cover a variety of temporal upsampling techniques. Despite the popularity of TAA, there are still significant unresolved technical challenges that affect image quality in many scenarios. We provide an in‐depth analysis of these challenges, and review existing techniques for improvements. Finally, we summarize popular algorithms and topics that are closely related to TAA. We believe the rapid advances in those areas may either benefit from or feedback into TAA research and development.

We describe a machine learning technique for reconstructing image sequences rendered using Monte Carlo methods. Our primary focus is on reconstruction of global illumination with extremely low sampling budgets at interactive rates. Motivated by recent advances in image restoration with deep convolutional networks, we propose a variant of these networks better suited to the class of noise present in Monte Carlo rendering. We allow for much larger pixel neighborhoods to be taken into account, while also improving execution speed by an order of magnitude. Our primary contribution is the addition of recurrent connections to the network in order to drastically improve temporal stability for sequences of sparsely sampled input images. Our method also has the desirable property of automatically modeling relationships based on auxiliary per-pixel input channels, such as depth and normals. We show significantly higher quality results compared to existing methods that run at comparable speeds, and furthermore argue a clear path for making our method run at realtime rates in the near future.

Foveated rendering synthesizes images with progressively less detail outside the eye fixation region, potentially unlocking significant speedups for wide field-of-view displays, such as head mounted displays, where target framerate and resolution is increasing faster than the performance of traditional real-time renderers. To study and improve potential gains, we designed a foveated rendering user study to evaluate the perceptual abilities of human peripheral vision when viewing today's displays. We determined that filtering peripheral regions reduces contrast, inducing a sense of tunnel vision. When applying a postprocess contrast enhancement, subjects tolerated up to 2× larger blur radius before detecting differences from a non-foveated ground truth. After verifying these insights on both desktop and head mounted displays augmented with high-speed gaze-tracking, we designed a perceptual target image to strive for when engineering a production foveated renderer. Given our perceptual target, we designed a practical foveated rendering system that reduces number of shades by up to 70% and allows coarsened shading up to 30° closer to the fovea than Guenter et al. [2012] without introducing perceivable aliasing or blur. We filter both pre- and post-shading to address aliasing from undersampling in the periphery, introduce a novel multiresolution- and saccade-aware temporal antialising algorithm, and use contrast enhancement to help recover peripheral details that are resolvable by our eye but degraded by filtering. We validate our system by performing another user study. Frequency analysis shows our system closely matches our perceptual target. Measurements of temporal stability show we obtain quality similar to temporally filtered non-foveated renderings.

We present a novel algorithm for reconstructing high-quality defocus blur from a sparsely sampled light field. Our algorithm builds upon recent developments in the area of sheared reconstruction filters and significantly improves reconstruction quality and performance. While previous filtering techniques can be ineffective in regions with complex occlusion, our algorithm handles such scenarios well by partitioning the input samples into depth layers. These depth layers are filtered independently and then combined together, taking into account inter-layer visibility. We also introduce a new separable formulation of sheared reconstruction filters that achieves real-time preformance on a modern GPU and is more than two orders of magnitude faster than previously published techniques.

We introduce multi-layer alpha blending, a novel solution to real-time order-independent transparency that operates in a single rendering pass and in bounded memory. The main contribution of our method is a new scalable approximation for the compositing equation that makes possible to easily trade off better image quality for more memory and lower performance. We demonstrate improved image quality and performance over previously published methods, while also reducing memory requirements.

We present surface based anti-aliasing (SBAA), a new approach to real-time anti-aliasing for deferred renderers that improves the performance and lowers the memory requirements for anti-aliasing methods that sample sub-pixel visibility. We introduce a novel way of decoupling visibility determination from shading that, compared to previous multi-sampling based approaches, significantly reduces the number of samples stored and shaded per pixel. Unlike post-process anti-aliasing techniques used in conjunction with deferred renderers, SBAA correctly resolves visibility of sub-pixel features, minimizing spatial and temporal artifacts.

Figure 1: A comparison of standard, scene-independent Parallel-Split Shadow Maps (PSSMs, left) with sample distribution shadow maps (SDSMs, right). Even with partitions tweaked to a specific camera view, PSSMs still produce lower quality shadows than SDSMs, which are fully automatic. Furthermore, the tighter SDSM shadow frusta enable more aggressive culling causing the algorithm to run faster than scene-independent schemes for complex geometry, such as this scene from Left for Dead 2 (courtesy of Valve Corporation). This paper introduces Sample Distribution Shadow Maps (SDSMs), a new algorithm for hard and soft-edged shadows that greatly reduces undersampling, oversampling, and geometric aliasing errors compared to other shadow map techniques. SDSMs fall into the space between scene-dependent, variable-performance shadow algorithms and scene-independent, fixed-performance shadow algorithms. They provide a fully automated solution to shadow map aliasing by optimizing the placement and size of a fixed number of Z-partitions using the distribution of the light space samples required by the current frame. SDSMs build on the advantages of current state of the art techniques, including predictable performance and constant memory usage, while removing tedious and ultimately suboptimal parameter tuning. We compare SDSMs to Parallel-Split Shadow Maps (PSSMs, a state of the art Z-partitioning scheme) and show that SDSMs produce higher quality shadows. Finally, we demonstrate that SDSMs outperform PSSMs in a large 2009 game scene at high resolutions, making them suitable for games and other interactive applications.

Abstract We introduce adaptive volumetric shadow maps (AVSM), a real‐time shadow algorithm that supports high‐quality shadowing from dynamic volumetric media such as hair and smoke. The key contribution of AVSM is the introduction of a streaming simplification algorithm that generates an accurate volumetric light attenuation function using a small fixed memory footprint. This compression strategy leads to high performance because the visibility data can remain in on‐chip memory during simplification and can be efficiently sampled during rendering. We demonstrate that AVSM compression closely approximates the ground‐truth correct solution and performs competitively to existing real‐time rendering techniques while providing higher quality volumetric shadows.