We present a novel generative model that synthesizes photorealistic, biophysically plausible faces by capturing the intricate relationships between facial geometry and biophysical attributes. Our approach models facial appearance in a biophysically grounded manner, allowing for the editing of both high‐level attributes such as age and gender, as well as low‐level biophysical properties such as melanin level and blood content. This enables continuous modeling of physical skin properties that correlate changes in skin properties with shape changes. We showcase the capabilities of our framework beyond its role as a generative model through two practical applications: editing the texture maps of 3D faces that have already been captured, and serving as a strong prior for face reconstruction when combined with differentiable rendering. Our model allows for the creation of physically‐based relightable, editable faces with consistent topology and uv layout that can be integrated into traditional computer graphics pipelines.

In recent years, reconstruction methods have been developed that can recover strand-level hair geometry from images. However, these methods recover a vast number of individual hair strands that are difficult to edit and simulate. Many methods also rely on neural priors to infer non-visible inner hair, which can result in poor inner hair structure for complex hairstyles, such as curly hair. We propose an inverse hair grooming pipeline that transforms the imperfect 3D strands from these reconstruction methods into procedural hair grooms that consist of a small set of guide strands and hair grooming operators, inspired by pipelines used by artists in popular 3D modeling tools such as Blender and Houdini. We take a probabilistic view of these hair grooms and design various optimization strategies and loss functions to optimize for the guide strands and operator parameters. Due to the proceduralism, our resulting grooms can naturally represent challenging hairstyles, have structurally sound inner hair, and are easily editable.

We propose a novel method for high-quality facial texture reconstruction from RGB images using a novel capturing routine based on a single smartphone which we equip with an inexpensive polarization foil. Specifically, we turn the flashlight into a polarized light source and add a polarization filter on top of the camera. Leveraging this setup, we capture the face of a subject with cross-polarized and parallel-polarized light. For each subject, we record two short sequences in a dark environment under flash illumination with different light polarization using the modified smartphone. Based on these observations, we reconstruct an explicit surface mesh of the face using structure from motion. We then exploit the camera and light colocation within a differentiable renderer to optimize the facial textures using an analysis-by-synthesis approach. Our method optimizes for high-resolution normal textures, diffuse albedo, and specular albedo using a coarse-to-fine optimization scheme. We show that the optimized textures can be used in a standard rendering pipeline to synthesize high-quality photo-realistic 3D digital humans in novel environments.

Abstract We present a method for estimating the main properties of human skin, leveraging a hyperspectral dataset of skin tones synthetically generated through a biophysical layered skin model and Monte Carlo light transport simulations. Our approach learns the mapping between the skin parameters and diffuse skin reflectance in such space through an encoder‐decoder network. We assess the performance of RGB and spectral reflectance up to 1 μm, allowing the model to retrieve visible and near‐infrared. Instead of restricting the parameters to values in the ranges reported in medical literature, we allow the model to exceed such ranges to gain expressiveness to recover outliers like beard, eyebrows, rushes and other imperfections. The continuity of our albedo space allows to recover smooth textures of skin properties, enabling reflectance manipulations by meaningful edits of the skin properties. The space is robust under different illumination conditions, and presents high spectral similarity with the current largest datasets of spectral measurements of real human skin while expanding its gamut.

Abstract Efficiently and accurately rendering hair accounting for multiple scattering is a challenging open problem. Path tracing in hair takes long to converge while other techniques are either too approximate while still being computationally expensive or make assumptions about the scene. We present a technique to infer the higher order scattering in hair in constant time within the path tracing framework, while achieving better computational efficiency. Our method makes no assumptions about the scene and provides control over the renderer's bias & speedup. We achieve this by training a small multilayer perceptron (MLP) to learn the higher‐order radiance online, while rendering progresses. We describe how to robustly train this network and thoroughly analyze our resulting renderer's characteristics. We evaluate our method on various hairstyles and lighting conditions. We also compare our method against a recent learning based & a traditional real‐time hair rendering method and demonstrate better quantitative & qualitative results. Our method achieves a significant improvement in speed with respect to path tracing, achieving a run‐time reduction of 40%‐70% while only introducing a small amount of bias.

Abstract Differentiable rendering of translucent objects with respect to their shapes has been a long‐standing problem. State‐of‐the‐art methods require detecting object silhouettes or specifying change rates inside translucent objects—both of which can be expensive for translucent objects with complex shapes. In this paper, we address this problem for translucent objects with no refractive or reflective boundaries. By reparameterizing interior components of differential path integrals, our new formulation does not require change rates to be specified in the interior of objects. Further, we introduce new Monte Carlo estimators based on this formulation that do not require explicit detection of object silhouettes.

Traditional fiber scattering models, based on ray optics, are missing some important visual aspects of fiber appearance. Previous work [Xia et al. 2020] on wave scattering from ideal extrusions demonstrated that diffraction produces strong forward scattering and colorful effects that are missing from ray-based models. However, that work was unable to include some important surface characteristics such as surface roughness and tilted cuticle scales, which are known to be important for fiber appearance. In this work, we take an important step to study wave effects from rough fibers with arbitrary 3D microgeometry. While the full-wave simulation of realistic 3D fibers remains intractable, we developed a 3D wave optics simulator based on a physical optics approximation, using a GPU-based hierarchical algorithm to greatly accelerate the calculation. It simulates surface reflection and diffractive scattering, which are present in all fibers and typically dominate for darkly pigmented fibers. The simulation provides a detailed picture of first order scattering, but it is not practical to use for production rendering as this would require tabulation per fiber geometry. To practically handle geometry variations in the scene, we propose a model based on wavelet noise, capturing the important statistical features in the simulation results that are relevant for rendering. Both our simulation and practical model show similar granular patterns to those observed in optical measurement. Our compact noise model can be easily combined with existing scattering models to render hair and fur of various colors, introducing visually important colorful glints that were missing from all previous models.

Abstract To increase diversity and realism, surface bidirectional scattering distribution functions (BSDFs) are often modelled as consisting of multiple layers, but accurately evaluating layered BSDFs while accounting for all light transport paths is a challenging problem. Recently, Guo et al. [GHZ18] proposed an accurate and general position‐free Monte Carlo method, but this method introduces variance that leads to longer render time compared to non‐stochastic layered models. We improve the previous work by presenting two new sampling strategies, pair‐product sampling and multiple‐product sampling. Our new methods better take advantage of the layered structure and reduce variance compared to the conventional approach of sequentially sampling one BSDF at a time. Our pair‐product sampling strategy importance samples the product of two BSDFs from a pair of adjacent layers. We further generalize this to multiple‐product sampling, which importance samples the product of a chain of three or more BSDFs. In order to compute these products, we developed a new approximate Gaussian representation of individual layer BSDFs. This representation incorporates spatially varying material properties as parameters so that our techniques can support an arbitrary number of textured layers. Compared to previous Monte Carlo layering approaches, our results demonstrate substantial variance reduction in rendering isotropic layered surfaces.

Many applications in rendering rely on integrating functions over spherical polygons. We present a new numerical solution for computing the integral of spherical harmonics (SH) expansions clipped to polygonal domains. Our solution, based on zonal decompositions of spherical integrands and discrete contour integration, introduces an important numerical operating for SH expansions in rendering applications. Our method is simple, efficient, and scales linearly in the bandlimited integrand’s harmonic expansion. We apply our technique to problems in rendering, including surface and volume shading, hierarchical product importance sampling, and fast basis projection for interactive rendering. Moreover, we show how to handle general, nonpolynomial integrands in a Monte Carlo setting using control variates. Our technique computes the integral of bandlimited spherical functions with performance competitive to (or faster than) more general numerical integration methods for a broad class of problems, both in offline and interactive rendering contexts. Our implementation is simple, relying only on self-contained SH evaluation and discrete contour integration routines, and we release a full source CPU-only and shader-based implementations (<750 lines of commented code).

Abstract Facial appearance capture is now firmly established within academic research and used extensively across various application domains, perhaps most prominently in the entertainment industry through the design of virtual characters in video games and films. While significant progress has occurred over the last two decades, no single survey currently exists that discusses the similarities, differences, and practical considerations of the available appearance capture techniques as applied to human faces. A central difficulty of facial appearance capture is the way light interacts with skin—which has a complex multi‐layered structure—and the interactions that occur below the skin surface can, by definition, only be observed indirectly. In this report, we distinguish between two broad strategies for dealing with this complexity. “Image‐based methods” try to exhaustively capture the exact face appearance under different lighting and viewing conditions, and then render the face through weighted image combinations. “Parametric methods” instead fit the captured reflectance data to some parametric appearance model used during rendering, allowing for a more lightweight and flexible representation but at the cost of potentially increased rendering complexity or inexact reproduction. The goal of this report is to provide an overview that can guide practitioners and researchers in assessing the tradeoffs between current approaches and identifying directions for future advances in facial appearance capture.

Abstract Environment‐mapped rendering of Lambertian isotropic surfaces is common, and a popular technique is to use a quadratic spherical harmonic expansion. This compact irradiance map representation is widely adopted in interactive applications like video games. However, many materials are anisotropic, and shading is determined by the local tangent direction, rather than the surface normal. Even for visualization and illustration, it is increasingly common to define a tangent vector field, and use anisotropic shading. In this paper, we extend spherical harmonic irradiance maps to anisotropic surfaces, replacing Lambertian reflectance with the diffuse term of the popular Kajiya‐Kay model. We show that there is a direct analogy, with the surface normal replaced by the tangent. Our main contribution is an analytic formula for the diffuse Kajiya‐Kay BRDF in terms of spherical harmonics; this derivation is more complicated than for the standard diffuse lobe. We show that the terms decay even more rapidly than for Lambertian reflectance, going as l–3, where l is the spherical harmonic order, and with only 6 terms (l = 0 and l = 2) capturing 99.8% of the energy. Existing code for irradiance environment maps can be trivially adapted for real‐time rendering with tangent irradiance maps. We also demonstrate an application to offline rendering of the diffuse component of fibers, using our formula as a control variate for Monte Carlo sampling.

We present an automated approach for high-quality preview of feature-film rendering during lighting design. Similar to previous work, we use a deep-framebuffer shaded on the GPU to achieve interactive performance. Our first contribution is to generate the deep-framebuffer and corresponding shaders automatically through data-flow analysis and compilation of the original scene. Cache compression reduces automatically-generated deep-framebuffers to reasonable size for complex production scenes and shaders. We also propose a new structure, the indirect framebuffer , that decouples shading samples from final pixels and allows a deep-framebuffer to handle antialiasing, motion blur and transparency efficiently. Progressive refinement enables fast feedback at coarser resolution. We demonstrate our approach in real-world production.