Extracting high-fidelity RGBD information from two-dimensional (2D) images is essential for various visual computing applications. Stereo imaging, as a reliable passive imaging technique for obtaining three-dimensional (3D) scene information, has benefited greatly from deep learning advancements. However, existing stereo depth estimation algorithms struggle to perceive high-frequency information and resolve high-resolution depth maps in realistic camera settings with large depth variations. These algorithms commonly neglect the hardware parameter configuration, limiting the potential for achieving optimal solutions solely through software-based design strategies.This work presents a hardware-software co-designed RGBD imaging framework that leverages both stereo and focus cues to reconstruct texture-rich color images along with detailed depth maps over a wide depth range. A pair of rank-2 parameterized diffractive optical elements (DOEs) is employed to encode perpendicular complementary information optically during stereo acquisitions. Additionally, we employ an IGEV-UNet-fused neural network tailored to the proposed rank-2 encoding for stereo matching and image reconstruction. Through prototyping a stereo camera with customized DOEs, our deep stereo imaging paradigm has demonstrated superior performance over existing monocular and stereo imaging systems in both image PSNR by 2.96 dB gain and depth accuracy in high-frequency details across distances from 0.67 to 8 meters.

Differentiable optics, as an emerging paradigm that jointly optimizes optics and (optional) image processing algorithms, has made many innovative optical designs possible across a broad range of imaging and display applications. Many of these systems utilize diffractive optical components for holography, PSF engineering, or wavefront shaping. Existing approaches have, however, mostly remained limited to laboratory prototypes, owing to a large quality gap between simulation and manufactured devices. We aim at lifting the fundamental technical barriers to the practical use of learned diffractive optical systems. To this end, we propose a fabrication-aware design pipeline for diffractive optics fabricated by direct-write grayscale lithography followed by replication with nano-imprinting, which is directly suited for inexpensive mass-production of large area designs. We propose a super-resolved neural lithography model that can accurately predict the 3D geometry generated by the fabrication process. This model can be seamlessly integrated into existing differentiable optics frameworks, enabling fabrication-aware, end-to-end optimization of computational optical systems. To tackle the computational challenges, we also devise tensor-parallel compute framework centered on distributing large-scale FFT computation across many GPUs. As such, we demonstrate large scale diffractive optics designs up to 32.16 mm × 21.44 mm, simulated on grids of up to 128,640 by 85,760 feature points. We find adequate agreement between simulation and fabricated prototypes for applications such as holography and PSF engineering. We also achieve high image quality from an imaging system comprised only of a single diffractive optical element, with images processed only by a one-step inverse filter utilizing the simulation PSF. We believe our findings lift the fabrication limitations for real-world applications of diffractive optics and differentiable optical design.

Modeling surface reflectance is central to connecting optical theory with real-world rendering and fabrication. While analytic BRDFs remain standard in rendering, recent advances in geometric and wave optics have expanded the design space for complex reflectance effects. However, existing wave-optics-based methods are limited to controlling reflectance intensity only, lacking the ability to design full-spectrum, color-dependent BRDFs. In this work, we present the first method for designing and fabricating color BRDFs using a fully differentiable wave optics framework. Our differentiable and memory-efficient simulation framework supports end-to-end optimization of microstructured surfaces under scalar diffraction theory, enabling joint control over both angular intensity and spectral color of reflectance. We leverage grayscale lithography with a feature size of 1.5–2.0 μ m to fabricate 15 BRDFs spanning four representative categories: anti-mirrors, pictorial reflections, structural colors, and iridescences. Compared to prior work, our approach achieves significantly higher fidelity and broader design flexibility, producing physically accurate and visually compelling results. By providing a practical and extensible solution for full-color BRDF design and fabrication, our method opens up new opportunities in structural coloration, product design, security printing, and advanced manufacturing.

Monocular RGBD imaging, also known as simultaneous all-in-focus (AiF) imaging and monocular depth estimation (MDE), represents a significant yet challenging task in computer vision. The crux lies in devising optical coding techniques to maximize the modulation transfer function (MTF) across various depths while minimizing their cross-correlation, all while aligning with the capabilities of image processing algorithms. End-to-end design of optics and algorithms offers a promising avenue towards achieving this holistic objective, but these approaches require solving non-convex inverse problems with millions of parameters. In this paper, we introduce a lattice-focal shape capable of nearly achieving the MTF bound as an initial solution, followed by employing B-spline parameterization for surface geometry representation to reduce the number of optimization variables. Further integration with the Restormer-based neural network, which possesses a global perspective, achieves high-performance RGBD imaging quality. Compared against state-of-the-art monocular RGBD imaging methods, our proposed approach improves the imaging peak signal-to-noise ratio (PSNR) by 3.0 dB and reduces the depth mean absolute error (MAE) by 39%. Experiments in real indoor and outdoor scenes validate the effectiveness of our method. The proposed approach paves the way for the development of monocular RGBD imaging.

While conventional cameras offer versatility for applications ranging from amateur photography to autonomous driving, computational cameras allow for domain-specific adaption. Cameras with co-designed optics and image processing algorithms enable high-dynamic-range image recovery, depth estimation, and hyperspectral imaging through optically encoding scene information that is otherwise undetected by conventional cameras. However, this optical encoding creates a challenging inverse reconstruction problem for conventional image recovery, and often lowers the overall photographic quality. Thus computational cameras with domain-specific optics have only been adopted in a few specialized applications where the captured information cannot be acquired in other ways. In this work, we investigate a method that combines two optical systems into one to tackle this challenge. We split the aperture of a conventional camera into two halves: one which applies an application-specific modulation to the incident light via a diffractive optical element to produce a coded image capture, and one which applies no modulation to produce a conventional image capture. Co-designing the phase modulation of the split aperture with a dual-pixel sensor allows us to simultaneously capture these coded and uncoded images without increasing physical or computational footprint. With an uncoded conventional image alongside the optically coded image in hand, we investigate image reconstruction methods that are conditioned on the conventional image, making it possible to eliminate artifacts and compute costs that existing methods struggle with. We assess the proposed method with 2-in-1 cameras for optical high-dynamic-range reconstruction, monocular depth estimation, and hyperspectral imaging, comparing favorably to all tested methods in all applications.

Unwanted camera obstruction can severely degrade captured images, including both scene occluders near the camera and partial occlusions of the camera cover glass. Such occlusions can cause catastrophic failures for various scene understanding tasks such as semantic segmentation, object detection, and depth estimation. Existing camera arrays capture multiple redundant views of a scene to see around thin occlusions. Such multi-camera systems effectively form a large synthetic aperture, which can suppress nearby occluders with a large defocus blur, but significantly increase the overall form factor of the imaging setup. In this work, we propose a monocular single-shot imaging approach that optically cloaks obstructions by emulating a large array. Instead of relying on different camera views, we learn a diffractive optical element (DOE) that performs depth-dependent optical encoding, scattering nearby occlusions while allowing paraxial wavefronts to be focused. We computationally reconstruct unobstructed images from these superposed measurements with a neural network that is trained jointly with the optical layer of the proposed imaging system. We assess the proposed method in simulation and with an experimental prototype, validating that the proposed computational camera is capable of recovering occluded scene information in the presence of severe camera obstruction.