On-chip spectral filter arrays enable compact, snapshot hyperspectral cameras that could be useful for applications ranging from agriculture to medical imaging and robotics. The spectral range and resolution of these cameras are determined by the number of spectral filters on the sensor, which are usually grouped together in repeating super-pixels, similar to the structure of a Bayer filter. Many tasks require dozens to hundreds of spectral channels, but this usually comes at the cost of a proportional decrease in spatial resolution and light throughput. We propose a multi-shot approach that leverages lens defocus to encode light before it is filtered by an on-chip spectral filter. Using this approach, we capture a burst of spatio-spectral encoded measurements at different focus planes, then solve an inverse problem to recover a high-resolution spatio-spectral datacube. Since most lens-based cameras already have a focusing mechanism, our approach is practical and could be broadly used by spectral filter array-based hyperspectral cameras. We provide analysis of our proposed system and comparisons against alternative methods. Finally, we demonstrate a prototype system and present experimental results of challenging dense, daylight scenes, demonstrating greater than $3 \times$ sub-super-pixel super-resolution.

Imaging systems have traditionally been designed to mimic the human eye and produce visually interpretable measurements. Modern imaging systems, however, process raw measurements computationally before or instead of human viewing. As a result, the information content of raw measurements matters more than their visual interpretability. Despite the importance of measurement information content, current approaches for evaluating imaging system performance do not quantify it: they instead either use alternative metrics that assess specific aspects of measurement quality or assess measurements indirectly with performance on secondary tasks. We developed the theoretical foundations and a practical method to directly quantify mutual information between noisy measurements and unknown objects. By fitting probabilistic models to measurements and their noise characteristics, our method estimates information by upper bounding its true value. By applying gradient-based optimization to these estimates, we also developed a technique for designing imaging systems called Information-Driven Encoder Analysis Learning (IDEAL). Our information estimates accurately captured system performance differences across four imaging domains (color photography, radio astronomy, lensless imaging, and microscopy). Systems designed with IDEAL matched the performance of those designed with end-to-end optimization, the prevailing approach that jointly optimizes hardware and image processing algorithms. These results establish mutual information as a universal performance metric for imaging systems that enables both computationally efficient design optimization and evaluation in real-world conditions. A video summary of this work can be found at: https://waller-lab.github.io/EncodingInformationWebsite/

Computer generated holography has long been touted as the future of augmented and virtual reality (AR/VR) displays, but has yet to be realized in practice. Previous high-quality, color holographic displays have made either a 3 × sacrifice on frame rate by using a sequential color illumination scheme or used more than one spatial light modulator (SLM) and/or bulky, complex optical setups. The reduced frame rate of sequential color introduces distracting judder and color fringing in the presence of head motion while the form factor of current simultaneous color systems is incompatible with a head-mounted display. In this work, we propose a framework for simultaneous color holography that allows the use of the full SLM frame rate while maintaining a compact and simple optical setup. Simultaneous color holograms are optimized through the use of a perceptual loss function, a physics-based neural network wavefront propagator, and a camera-calibrated forward model. We measurably improve hologram quality compared to other simultaneous color methods and move one step closer to the realization of color holographic displays for AR/VR.