Abstract Achieving physically consistent image editing remains a significant challenge in computer vision. Existing image editing methods typically rely on neural networks, which struggle to accurately handle shadows and refractions. Conversely, physics-based inverse rendering often requires multi-view optimization, limiting its practicality in single-image scenarios. In this paper, we propose Materialist , a neural-initialized physically based rendering pipeline for single-image inverse rendering. Unlike previous hybrid methods that use physics to guide neural generation, our method leverages neural networks to predict initial material properties, which are then rigorously optimized via progressive differentiable rendering. Our approach enables a range of applications, including material editing, object insertion, and relighting, while also introducing an effective method for editing material transparency via ray-traced refraction without requiring full scene geometry. Furthermore, our envmap estimation method also achieves competitive performance, further enhancing the accuracy of image editing task. Experiments demonstrate strong performance across synthetic and real-world datasets, excelling even on challenging out-of-domain images.

We introduce a divergence-free nD vector noise defined as the n-dimensional cross product of the gradients of n − 1 noise functions. We show that this vector noise function is divergence-free and hence volume preserving for any dimension n. Our method enables precise integration and extends to new settings by substituting noise functions with implicit surfaces, (hyper)surfaces, or custom functions. We demonstrate applications including image warping, surface texturing, noise bounded by implicit surfaces, anisotropic curl-noise, and high-dimensional point jittering up to 7D.

Abstract Cosmetic products have found their place in various aspects of human life, yet their digital appearance reproduction has received little attention. We present an appearance model for cosmetics, in particular for foundation layers, that reproduces a range of existing appearances of foundation cosmetics: from a glossy to a matte to an almost velvety look. Our model is a multilayered BSDF that reproduces the stacking of multiple layers of cosmetics. Inspired by the microscopic particulates used in cosmetics, we model each individual layer as a stochastic participating medium with two types of scatterers that mimic the most prominent visual features of cosmetics: spherical diffusers, resulting in a uniform distribution of radiance; and platelets, responsible for the glossy look of certain cosmetics. We implement our model on top of the position‐free Monte Carlo framework, that allows us to include multiple scattering. We validate our model against measured reflectance data, and demonstrate the versatility and expressiveness of our model by thoroughly exploring the range of appearances that it can produce.

Abstract Monte Carlo rendering of translucent objects with heterogeneous scattering properties is often expensive both in terms of memory and computation. If the scattering properties are described by a 3D texture, memory consumption is high. If we do path tracing and use a high dynamic range lighting environment, the computational cost of the rendering can easily become significant. We propose a compact and efficient neural method for representing and rendering the appearance of heterogeneous translucent objects. Instead of assuming only surface variation of optical properties, our method represents the appearance of a full object taking its geometry and volumetric heterogeneities into account. This is similar to a neural radiance field, but our representation works for an arbitrary distant lighting environment. In a sense, we present a version of neural precomputed radiance transfer that captures relighting of heterogeneous translucent objects. We use a multi‐layer perceptron (MLP) with skip connections to represent the appearance of an object as a function of spatial position, direction of observation, and direction of incidence. The latter is considered a directional light incident across the entire non‐self‐shadowed part of the object. We demonstrate the ability of our method to compactly store highly complex materials while having high accuracy when comparing to reference images of the represented object in unseen lighting environments. As compared with path tracing of a heterogeneous light scattering volume behind a refractive interface, our method more easily enables importance sampling of the directions of incidence and can be integrated into existing rendering frameworks while achieving interactive frame rates.

Abstract We present a method for capturing the BSSRDF (bidirectional scattering‐surface reflectance distribution function) of arbitrary geometry with a neural network. We demonstrate how a compact neural network can represent the full 8‐dimensional light transport within an object including heterogeneous scattering. We develop an efficient rendering method using importance sampling that is able to render complex translucent objects under arbitrary lighting. Our method can also leverage the common planar half‐space assumption, which allows it to represent one BSSRDF model that can be used across a variety of geometries. Our results demonstrate that we can render heterogeneous translucent objects under arbitrary lighting and obtain results that match the reference rendered using volumetric path tracing.

We propose a novel dictionary-based representation learning model for Bidirectional Texture Functions (BTFs) aiming at compact storage, real-time rendering performance, and high image quality. Our model is trained once, using a small training set, and then used to obtain a sparse tensor containing the model parameters. Our technique exploits redundancies in the data across all dimensions simultaneously, as opposed to existing methods that use only angular information and ignore correlations in the spatial domain. We show that our model admits efficient angular interpolation directly in the model space, rather than the BTF space, leading to a notably higher rendering speed than in previous work. Additionally, the high quality-storage cost tradeoff enabled by our method facilitates controlling the image quality, storage cost, and rendering speed using a single parameter, the number of coefficients. Previous methods rely on a fixed number of latent variables for training and testing, hence limiting the potential for achieving a favorable quality-storage cost tradeoff and scalability. Our experimental results demonstrate that our method outperforms existing methods both quantitatively and qualitatively, as well as achieving a higher compression ratio and rendering speed.

We present a practical method for temporal and stereoscopic filtering that generates stereo-consistent rendering. Existing methods for stereoscopic rendering often reuse samples from one eye for the other or do averaging between the two eyes. These approaches fail in the presence of ray tracing effects such as specular reflections and refractions. We derive a new blending strategy that leverages variance to compute per pixel blending weights for both temporal and stereoscopic rendering. In the temporal domain, our method works well in a low noise context and is robust in the presence of inconsistent motion vectors, where existing methods such as temporal anti-aliasing (TAA) and deep learning super sampling (DLSS) produce artifacts. In the stereoscopic domain, our method provides a new way to ensure consistency between the left and right eyes. The stereoscopic version of our method can be used with our new temporal method or with existing methods such as DLSS and TAA. In all combinations, it reduces the error and significantly increases the consistency between the eyes making it practical for real-time settings such as virtual reality (VR).

We propose a method for implicitly generating blue noise point sets. Our method is based on the observations that curl noise vector fields are volume-preserving and that jittering can be construed as moving points along the streamlines of a vector field. We demonstrate that the volume preservation keeps the points well separated when jittered using a curl noise vector field. At the same time, the anisotropy that stems from regular lattices is significantly reduced by such jittering. In combination, these properties entail that jittering by curl noise effectively transforms a regular lattice into a point set with blue noise properties. Our implicit method does not require computing the point set in advance. This makes our technique valuable when an arbitrarily large set of points with blue noise properties is needed. We compare our method to several other methods based on jittering as well as other methods for blue noise point set generation. Finally, we show several applications of curl noise jittering in two and three dimensions.

Abstract Image denoising based on deep learning has become a powerful tool to accelerate Monte Carlo rendering. Deep learning techniques can produce smooth images using a low sample count. Unfortunately, existing deep learning methods are biased and do not converge to the correct solution as the number of samples increase. In this paper, we propose a progressive denoising technique that aims to use denoising only when it is beneficial and to reduce its impact at high sample counts. We use Stein's unbiased risk estimate (SURE) to estimate the error in the denoised image, and we combine this with a neural network to infer a per‐pixel mixing parameter. We further augment this network with confidence intervals based on classical statistics to ensure consistency and convergence of the final denoised image. Our results demonstrate that our method is consistent and that it improves existing denoising techniques. Furthermore, it can be used in combination with existing high quality denoisers to ensure consistency. In addition to being asymptotically unbiased, progressive denoising is particularly good at preserving fine details that would otherwise be lost with existing denoisers.

Abstract Most models for bidirectional surface scattering by arbitrary explicitly defined microgeometry are either based on geometric optics and include multiple scattering but no diffraction effects or based on wave optics and include diffraction but no multiple scattering effects. The few exceptions to this tendency are based on rigorous solution of Maxwell's equations and are computationally intractable for surface microgeometries that are tens or hundreds of microns wide. We set up a measurement equation for combining results from single scattering scalar diffraction theory with multiple scattering geometric optics using Monte Carlo integration. Since we consider an arbitrary surface microgeometry, our method enables us to compute expected bidirectional scattering of the metasurfaces with increasingly smaller details seen more and more often in production. In addition, we can take a measured microstructure as input and, for example, compute the difference in bidirectional scattering between a desired surface and a produced surface. In effect, our model can account for both diffraction colors due to wavelength‐sized features in the microgeometry and brightening due to multiple scattering. We include scalar diffraction for refraction, and we verify that our model is reasonable by comparing with the rigorous solution for a microsurface with half ellipsoids.

Abstract The outset of realistic rendering is a desire to reproduce the appearance of the real world. Rendering techniques therefore operate at a scale corresponding to the size of objects that we observe with our naked eyes. At the same time, rendering techniques must be able to deal with objects of nearly arbitrary shapes and materials. These requirements lead to techniques that oftentimes leave the task of setting the optical properties of the materials to the user. Matching the appearance of real objects by manual adjustment of optical properties is however nearly impossible. We can render objects with a plausible appearance in this way but cannot compare the appearance of a manufactured item to that of its digital twin. This is especially true in the case of translucent objects, where we need more than a goniometric measurement of the optical properties. In this survey, we provide an overview of forward and inverse models for acquiring the optical properties of translucent materials. We map out the efforts in graphics research in this area and describe techniques available in related fields. Our objective is to provide a better understanding of the tools currently available for appearance specification when it comes to digital representations of real translucent objects.

Abstract Digital light processing stereolithography is a promising technique for 3D printing. However, it offers little control over the surface appearance of the printed object. The printing process is typically layered, which leads to aliasing artefacts that affect surface appearance. An antialiasing option is to use greyscale pixel values in the layer images that we supply to the printer. This enables a kind of subvoxel growth control. We explore this concept and use it for editing surface microstructure. In other words, we modify the surface appearance of a printed object by applying a greyscale pattern to the surface voxels before sending the cross‐sectional layer images to the printer. We find that a smooth noise function is an excellent tool for varying surface roughness and for breaking the regularities that lead to aliasing. Conversely, we also present examples that introduce regularities to produce controlled anisotropic surface appearance. Our hope is that subvoxel growth control in stereolithography can lead 3D printing towards customizable surface appearance. The printing process adds what we call ground noise to the printed result. We suggest a way of modelling this ground noise to provide users with a tool for estimating a printer's ability to control surface reflectance.

Interactive 3D modeling in VR is both aided by immersive 3D input and hampered by model disjunct, tool-based or selection-action user interfaces. We propose a direct, signifier-based approach to the popular interactive technique of creating 3D models through a sequence of extrusion operations. Motivated by handles and signifiers that communicate the affordances of everyday objects, we define a set of design principles for an immersive, signifier-based modeling interface. We then present an interactive 3D modeling system where all modeling affordances are modelessly reachable and signified on the model itself.

The appearance of a transparent object is determined by a combination of refraction and reflection, as governed by a complex function of its shape as well as the surrounding environment. Prior works on 3D reconstruction have largely ignored transparent objects due to this challenge, yet they occur frequently in real-world scenes. This paper presents an approach to estimate depths and normals for transparent objects using a single image acquired under a distant but otherwise arbitrary environment map. In particular, we use a deep convolutional neural network (CNN) for this task. Unlike opaque objects, it is challenging to acquire ground truth training data for refractive objects, thus, we propose to use a large-scale synthetic dataset. To accurately capture the image formation process, we use a physically-based renderer. We demonstrate that a CNN trained on our dataset learns to reconstruct shape and estimate segmentation boundaries for transparent objects using a single image, while also achieving generalization to real images at test time. In experiments, we extensively study the properties of our dataset and compare to baselines demonstrating its utility.

Abstract We present a photon splatting technique which reduces noise and blur in the rendering of caustics. Blurring of illumination edges is an inherent problem in photon splatting, as each photon is unaware of its neighbours when being splatted. This means that the splat size is usually based on heuristics rather than knowledge of the local flux density. We use photon differentials to determine the size and shape of the splats such that we achieve adaptive anisotropic flux density estimation in photon splatting. As compared to previous work that uses photon differentials, we present the first method where no photons or beams or differentials need to be stored in a map. We also present improvements in the theory of photon differentials, which give more accurate results and a faster implementation. Our technique has good potential for GPU acceleration, and we limit the number of parameters requiring user adjustment to an overall smoothing parameter and the number of photons to be traced.

Rendering translucent materials using Monte Carlo ray tracing is computationally expensive due to a large number of subsurface scattering events. Faster approaches are based on analytical models derived from diffusion theory. While such analytical models are efficient, they miss out on some translucency effects in the rendered result. We present an improved analytical model for subsurface scattering that captures translucency effects present in the reference solutions but remaining absent with existing models. The key difference is that our model is based on ray source diffusion, rather than point source diffusion. A ray source corresponds better to the light that refracts through the surface of a translucent material. Using this ray source, we are able to take the direction of the incident light ray and the direction toward the point of emergence into account. We use a dipole construction similar to that of the standard dipole model, but we now have positive and negative ray sources with a mirrored pair of directions. Our model is as computationally efficient as existing models while it includes single scattering without relying on a separate Monte Carlo simulation, and the rendered images are significantly closer to the references. Unlike some previous work, our model is fully analytic and requires no precomputation.

Abstract Glare is a consequence of light scattered within the human eye when looking at bright light sources. This effect can be exploited for tone mapping since adding glare to the depiction of high‐dynamic range (HDR) imagery on a low‐dynamic range (LDR) medium can dramatically increase perceived contrast. Even though most, if not all, subjects report perceiving glare as a bright pattern that fluctuates in time, up to now it has only been modeled as a static phenomenon. We argue that the temporal properties of glare are a strong means to increase perceived brightness and to produce realistic and attractive renderings of bright light sources. Based on the anatomy of the human eye, we propose a model that enables real‐time simulation of dynamic glare on a GPU. This allows an improved depiction of HDR images on LDR media for interactive applications like games, feature films, or even by adding movement to initially static HDR images. By conducting psychophysical studies, we validate that our method improves perceived brightness and that dynamic glare‐renderings are often perceived as more attractive depending on the chosen scene.

This paper introduces a theoretical model for computing the scattering properties of participating media and translucent materials. The model takes as input a description of the components of a medium and computes all the parameters necessary to render it. These parameters are the extinction and scattering coefficients, the phase function, and the index of refraction, Our theory is based on a robust generalization of the Lorenz-Mie theory. Previous models using Lorenz-Mie theory have been limited to non-absorbing media with spherical particles such as paints and clouds. Our generalized theory is capable of handling both absorbing host media and non-spherical particles, which significantly extends the classes of media and materials that can be modeled. We use the theory to computer optical properties for different types of ice and ocean water, and we derive a novel appearance model for milk parameterized by the fat and protein contents. Our results show that we are able to match measured scattering properties in cases where the classical Lorez-Mie theory breaks down, and we can compute properties for media that cannot be measured using existing techniques in computer graphics.

A novel method for real-time simulation of indirect illumination is presented in this paper. The method, which we call direct radiance mapping (DRM), is based on basal radiance calculations and does not impose any restrictions on scene geometry or dynamics. This makes the method tractable for real-time rendering of arbitrary dynamic environments and for interactive preview of feature animations. Through DRM we simulate two diffuse reflections of light, but can also, in combination with traditional real-time methods for specular reflections, simulate more complex light paths. DRM is a GPU-based method, which can draw further advantages from upcoming GPU functionalities. The method has been tested for moderately sized scenes with close to real-time frame rates and it scales with interactive frame rates for more complex scenes.