Accurate illumination is crucial for many imaging and vision applications, and skies are the dominant source of lighting in many scenes. Most existing work for representing sky illumination has focused on clear skies or more recently generative approaches for synthesizing clouds. However, these are very limited in that they assume distant illumination and do not capture the 3D properties of clouds. This paper presents a novel and principled approach to extract 3D whole-sky volumetric representations of clouds which can be used for imaging applications. Our approach extracts clouds from a single fisheye capture of the sky via an iterative optimization process. We achieve this by exploiting the physical properties of light scattering in clouds and use these to drive a domain-specific light transport simulation algorithm to render the images required for optimization. Results for this method provide high accuracy when re-rendering with our reconstructed clouds compared to real captures, and also enable novel uses of environment maps such as inclusion of captured clouds in renderings, cloud shadows, and more accurate aerial perspective and lighting.

Sky illumination is a core source of lighting in rendering, and a substantial amount of work has been developed to simulate lighting from clear skies. However, in reality, clouds substantially alter the appearance of the sky and subsequently change the scene illumination. While there have been recent advances in developing sky models which include clouds, these all neglect cloud movement which is a crucial component of cloudy sky appearance. In any sort of video or interactive environment, it can be expected that clouds will move, sometimes quite substantially in a short period of time. Our work proposes a solution to this which enables whole-sky dynamic cloud synthesis for the first time. We achieve this by proposing a multi-timescale sky appearance model which learns to predict the sky illumination over various timescales, and can be used to add dynamism to previous static, cloudy sky lighting approaches. • A framework for dynamic cloud lighting synthesis from a single static sky image • Input images from hemispherical sky capture or from recent static generative methods • An learned optical flow based method for hemispherical image cloud movement. • A multi-timescale method for smooth, coherent cloud movement across the sky • Results show smooth cloud movement synthesis applied in various rendering scenarios

Abstract Image editing is a commonly studied problem in computer graphics. Despite the presence of many advanced editing tools, there is no satisfactory solution to controllably update the position of the sun using a single image. This problem is made complicated by the presence of clouds, complex landscapes, and the atmospheric effects that must be accounted for. In this paper, we tackle this problem starting with only a single photograph. With the user clicking on the initial position of the sun, our algorithm performs several estimation and segmentation processes for finding the horizon, scene depth, clouds, and the sky line. After this initial process, the user can make both fine‐ and large‐scale changes on the position of the sun: it can be set beneath the mountains or moved behind the clouds practically turning a midday photograph into a sunset (or vice versa). We leverage a precomputed atmospheric scattering algorithm to make all of these changes not only realistic but also in real‐time. We demonstrate our results using both clear and cloudy skies, showing how to add, remove, and relight clouds, all the while allowing for advanced effects such as scattering, shadows, light shafts, and lens flares.