Photos taken in unconstrained tourist environments often present challenges for accurate 3D scene reconstruction due to variable appearances and transient occlusions, which can introduce artifacts in novel view synthesis. Recently, in-the-wild 3D scene reconstruction has been achieved realistic rendering with Neural Radiance Fields (NeRFs). With the advancement of 3D Gaussian Splatting (3DGS), some methods also attempt to reconstruct 3D scenes from unconstrained photo collections and achieve real-time rendering. However, the rapid convergence of 3DGS is misaligned with the slower convergence of neural network-based appearance encoder and transient mask predictor, hindering the reconstruction efficiency. To address this, we propose a novel sky-aware framework for scene reconstruction from unconstrained photo collection using 3DGS. Firstly, we observe that the learnable per-image transient mask predictor in previous work is unnecessary. By introducing a simple yet efficient greedy supervision strategy, we directly utilize the pseudo mask generated by a pretrained semantic segmentation network as the transient mask, thereby achieving more efficient and higher quality in-the-wild 3D scene reconstruction. Secondly, we find that separately estimating appearance embeddings for the sky and building significantly improves reconstruction efficiency and accuracy. We analyze the underlying reasons and introduce a neural sky module to generate diverse skies from latent sky embeddings extract from unconstrained images. Finally, we propose a mutual distillation learning strategy to constrain sky and building appearance embeddings within the same latent space, further enhancing reconstruction efficiency and quality. Extensive experiments on multiple datasets demonstrate that the proposed framework outperforms existing methods in novel view and appearance synthesis, offering superior rendering quality with faster convergence and rendering speed.

Background Physics-based differentiable rendering (PBDR) aims to propagate gradients from scene parameters to image pixels or vice versa. The physically correct gradients obtained can be used in various applications, including inverse rendering and machine learning. Currently, two categories of methods are prevalent in the PBDR community: reparameterization and boundary sampling methods. The state-of-the-art boundary sampling methods rely on a guiding structure to calculate the gradients efficiently. They utilize the rays generated in traditional path-tracing methods and project them onto the object silhouette boundary to initialize the guiding structure. Methods In this study, we propose an augmentation of previous projective-sampling-based boundary-sampling methods in a bidirectional manner. Specifically, we utilize the rays spawned from the sensors and also employ the rays emitted by the emitters to initialize the guiding structure. Results To demonstrate the benefits of our technique, we perform a comparative analysis of differentiable rendering and inverse rendering performance. We utilize a range of synthetic scene examples and evaluate our method against state-of-the-art projective-sampling-based differentiable rendering methods. Conclusions The experiments show that our method achieves lower variance gradients in the forward differentiable rendering process and better geometry reconstruction quality in the inverse-rendering results.