We present Image Sculpting, a new framework for editing 2D images by incorporating tools from 3D geometry and graphics. This approach differs markedly from existing methods, which are confined to 2D spaces and typically rely on textual instructions, leading to ambiguity and limited control. Image Sculpting converts 2D objects into 3D, enabling direct interaction with their 3D geometry. Post-editing, these objects are re-rendered into 2D, merging into the original image to produce high-fidelity results through a coarse-to-fine enhancement process. The framework supports precise, quantifiable, and physically-plausible editing options such as pose editing, rotation, translation, 3D composition, carving, and serial addition. It marks an initial step towards combining the creative freedom of generative models with the precision of graphics pipelines.

We present NeX, a new approach to novel view synthesis based on enhancements of multiplane images (MPI) that can reproduce view-dependent effects in real time. Unlike traditional MPI, our technique parameterizes each pixel as a linear combination of spherical basis functions learned from a neural network to model view-dependent effects and uses a hybrid implicit-explicit modeling strategy to improve fine detail. Moreover, we also present an extension to NeX, which leveragesknowledge distillationto train multiple MPIs for unbounded 360$^\circ$scenes. Our method is evaluated on several benchmark datasets: NeRF-Synthetic dataset, Light Field dataset, Real Forward-Facing dataset, Space dataset, as well asShiny, our new dataset that contains significantly more challenging view-dependent effects, such as the rainbow reflections on the CD. Our method outperforms other real-time rendering approaches on PSNR, SSIM, and LPIPS and can renderunbounded360$^\circ$scenes in real time.

We present NeX, a new approach to novel view synthesis based on enhancements of multiplane image (MPI) that can reproduce next-level view-dependent effects—in real time. Unlike traditional MPI that uses a set of simple RGBα planes, our technique models view-dependent effects by instead parameterizing each pixel as a linear combination of basis functions learned from a neural network. Moreover, we propose a hybrid implicit-explicit modeling strategy that improves upon fine detail and produces state-of-the-art results. Our method is evaluated on benchmark forwardfacing datasets as well as our newly-introduced dataset designed to test the limit of view-dependent modeling with significantly more challenging effects such as the rainbow reflections on a CD. Our method achieves the best overall scores across all major metrics on these datasets with more than 1000× faster rendering time than the state of the art. For real-time demos, visit https://nex-mpi.github.io/