Neural Radiance Field (NeRF)-based segmentation methods focus on object semantics and rely solely on RGB data, lacking intrinsic material properties. This limitation restricts accurate material perception, which is crucial for robotics, augmented reality, simulation, and other applications. We introduce UnMix-NeRF, a framework that integrates spectral unmixing into NeRF, enabling joint hyperspectral novel view synthesis and unsupervised material segmentation. Our method models spectral reflectance via diffuse and specular components, where a learned dictionary of global endmembers represents pure material signatures, and per-point abundances capture their distribution. For material segmentation, we use spectral signature predictions along learned endmembers, allowing unsupervised material clustering. Additionally, UnMix-NeRF enables scene editing by modifying learned endmember dictionaries for flexible material-based appearance manipulation. Extensive experiments validate our approach, demonstrating superior spectral reconstruction and material segmentation to existing methods. Project page: https://www.factral.co/UnMix-NeRF.

Passive hyperspectral longwave infrared measurements are remarkably informative about the surroundings. Remote object material and temperature determine the spectrum of thermal radiance, and range, air temperature, and gas concentrations determine how this spectrum is modified by propagation to the sensor. We introduce a passive range imaging method based on computationally separating these phenomena. Previous methods assume hot and highly emitting objects; ranging is more challenging when objects' temperatures do not deviate greatly from air temperature. Our method jointly estimates range and intrinsic object properties, with explicit consideration of air emission, though reflected light is assumed negligible. Inversion being underdetermined is mitigated by using a parametric model of atmospheric absorption and regularizing for smooth emissivity estimates. To assess where our estimate is likely accurate, we introduce a technique to detect which scene pixels are significantly influenced by reflected downwelling. Monte Carlo simulations demonstrate the importance of regularization, temperature differentials, and availability of many spectral bands. We apply our method to longwave infrared (8-13 $\mathrm{\mu }\mathrm{m}$μm) hyperspectral image data acquired from natural scenes with no active illumination. Range features from 15 m to 150 m are recovered, with good qualitative match to lidar data for pixels classified as having negligible reflected downwelling.

Compressive multispectral imaging systems comprise a new generation of spectral imagers that capture coded projections of a scene where spectral data cubes are reconstructed computationally. Separately, time-of-flight (ToF) cameras obtain 2D range images where each pixel records the distance from the camera sensor to the target surface. The demand for these imaging modalities is rapidly increasing, and thus, there is strong interest in developing new image sensors that can simultaneously acquire multispectral-color-and-depth imagery (MS+D) using a single aperture. Work in this path has been mainly developed via RGB+D imaging. However, in RGB+D, the multispectral image is limited to three spectral channels, and the imaging system often relies on two image sensors. We recently proposed a compressive MS+D imaging device that used a digital-micromirror-device, requiring a bulky double imaging-and-relay path. To overcome the bulkiness and other difficulties of our previous imaging system, this work presents a more-compact MS+D imaging device with snapshot capabilities. It provides better spectral sensing, relying on a static color-coded-aperture (CCA) and a ToF sensor. To guarantee good quality in the recovery, we develop an optimization method for CCA based-on blue-noise-multitoning, solved via the direct-binary-search algorithm. A testbed-setup is reported along with simulated and real experiments that demonstrate the MS+D capabilities of the proposed system over static and dynamic scenes.

In recent years there has been an increasing interest in sensing devices that capture multidimensional information such as the spectral light field (SLF) images, which are 5-dimensional (5D) representations of a scene including 2D spatial, 2D angular and 1D spectral information. Spatio-spectral and angular information plays an important role in modern applications spanning from microscopy to computer vision. However, SLF sensors use expensive beam-splitters or cameras arrays placed in tandem, which split the sensing problem in two time consuming and independent tasks: spectral and light field imaging tasks. This work proposes a compressive spectral light field imaging architecture that builds on the principles of the compressive imaging framework, to capture multiplexed representations of the multidimensional information, so that, less measurements are required to capture the SLF data cube. Alongside, we propose a computational algorithm to recover the 5D information from the compressed measurements, exploiting the inherent high correlations within the SLF by treating them as 3D tensors. Furthermore, exploiting the geometry properties of the proposed optical architecture, the Tucker decomposition is applied to the set of compressed measurements, so that, an ad-hoc dictionary-like image representation basis is calculated online. This in turn, entails a more accurate reconstruction of the SLF since the dictionary fits the specific characteristics of the image itself. We demonstrate through simulations over three SLF datasets captured in our laboratory, and an experimental proof-of-concept implementation, that the proposed compressive imaging device together with the proposed computational algorithm represent an efficient alternative to capture SLF, compared to conventional methods that employ either side-information or multiple sensors. Also, we show that the tensor-based proposed algorithm exhibits a lower computational complexity than the matrix-based state of the art counterparts, thus enabling fast processing of multidimensional images.

Compressive spectral imaging (CSI) is a framework that captures coded-and-multiplexed low-dimensional projections of spectral data-cubes. In general, the sensing process in many CSI architectures is described using binary matrices, so-called sensing/projection matrices, whose elements can be either random or designed. However, some characteristics of the spectral data, such as the ℓ2-norm or the second moment statistics, can be lost when this dimensionality reduction is performed. Similarly, principal component analysis (PCA) is a data dimensionality reduction technique that minimizes the least-squared error between the spectral data and its low-dimensional projection, but preserving its structure or variance. Thus, PCA can be used to guide the CSI acquisition process by designing the binary sensing matrix. Nonetheless, PCA requires to know the spectral image a-priori, and also, its associated projection matrix is not binary, as required by CSI optical architectures. Therefore, in this paper, an algorithm to design CSI sensing matrices by exploiting the structure-preserving property of the PCA projection is proposed. First, a set of compressive measurements obtained with random sensing matrices is used to rapidly estimate the covariance matrix associated with the spectral data. Then, a new sensing matrix is designed by solving a non-convex optimization problem that finds a set of binary vectors that approximate the principal components of the covariance matrix, thus maximizing the explanation of the data variance. Experimental results show an improvement of up to 3 dB in image reconstruction quality, in terms of the peak signal to noise ratio (PSNR), when the binary PCA-based sensing matrices are used and compared with conventional random sensing matrices and state-of-art designed matrices based on PCA.

Compressive spectral imaging (CSI) has demonstrated to be a feasible technique for capturing the 3D spatio spectral information of a scene through less measurements than the Nyquist rate. The coded aperture snapshot spectral imaging (CAS-SI) is an example of a CSI optical architecture, which has been proposed to work on the visible electromagnetic spectrum. Due to the rich information contained in the infrared spectrum, in this paper, we mathematically model and demonstrate the implementation of a broadband CASSI system covering the visible and the near infrared spectra between 448 nm to 1436 nm. Particularly, we developed a Digital Micromirror Device-based CSI system, which implements dual-band CS measurement processes of 3D spatio-spectral scenes.

Compressive spectral imaging (CSI) systems capture the 3D spatio-spectral information of a scene by measuring 2D focal plane array (FPA) coded projections. A reconstruction algorithm exploiting the sparsity of the signal is then used to recover the underlying hyperspectral scene. CSI systems use a set of binary coded apertures, commonly realized through photomasks, to modulate the spatial characteristics of the scene. The reconstruction image quality in CSI is determined by the design of a 2D coded aperture binary set which block or unblock light from the scene onto the detector. This work extends the framework of CSI by replacing the traditional block-unblock photomasks by colored coded apertures which modulate the source not only spatially but spectrally as well. Simulations show a significant improvement in the quality of spectral image reconstructions.

Coded aperture snapshot spectral imaging systems (CASSI) measure the 3D spatio-spectral information of a scene using several compressive 2D focal plane array (FPA) snapshots. The image reconstruction algorithms utilized in CASSI use a first-order approximation of the underlying analog sensing phenomena. A calibration method is then used to compensate for the coarse approximation - an approach not well suited for multishot CASSI systems. This paper develops a more accurate computational model for CASSI which provides a higher quality of image reconstruction. Several simulations are shown to illustrate the performance improvement attained by the new model.