We introduce a single-view reconstruction technique of volumetric fields in which multiple light scattering effects are omnipresent, such as in clouds. We model the unknown distribution of volumetric fields using an unconditional diffusion model trained on a novel benchmark dataset comprising 1,000 synthetically simulated volumetric density fields. The neural diffusion model is trained on the latent codes of a novel, diffusion-friendly, monoplanar representation. The generative model is used to incorporate a tailored parametric diffusion posterior sampling technique into different reconstruction tasks. A physically-based differentiable volume renderer is employed to provide gradients with respect to light transport in the latent space. This stands in contrast to classic NeRF approaches and makes the reconstructions better aligned with observed data. Through various experiments, we demonstrate single-view reconstruction of volumetric clouds at a previously unattainable quality.

Abstract Accurate subsurface scattering solutions require the integration of optical material properties along many complicated light paths. We present a method that learns a simple geometric approximation of random paths in a homogeneous volume with translucent material. The generated representation allows determining the absorption along the path as well as a direct lighting contribution, which is representative of all scatter events along the path. A sequence of conditional variational auto‐encoders (CVAEs) is trained to model the statistical distribution of the photon paths inside a spherical region in the presence of multiple scattering events. A first CVAE learns how to sample the number of scatter events, occurring on a ray path inside the sphere, which effectively determines the probability of this ray to be absorbed. Conditioned on this, a second model predicts the exit position and direction of the light particle. Finally, a third model generates a representative sample of photon position and direction along the path, which is used to approximate the contribution of direct illumination due to in‐scattering. To accelerate the tracing of the light path through the volumetric medium toward the solid boundary, we employ a sphere‐tracing strategy that considers the light absorption and can perform a statistically accurate next‐event estimation. We demonstrate efficient learning using shallow networks of only three layers and no more than 16 nodes. In combination with a GPU shader that evaluates the CVAEs’ predictions, performance gains can be demonstrated for a variety of different scenarios. We analyze the approximation error that is introduced by the data‐driven scattering simulation and shed light on the major sources of error.