Martian terrain recognition is pivotal for advancing our understanding of topography, geomorphology, paleoclimate, and habitability. While deep clustering methods have shown promise in learning semantically homogeneous feature embeddings from Martian rover imagery, the natural variations in intensity, scale, and rotation pose significant challenges for accurate terrain classification. To address these limitations, we propose Deep Constrained Clustering with Metric Learning (DCCML), a novel algorithm that leverages multiple constraint types to guide the clustering process. DCCML incorporates soft must-link constraints derived from spatial and depth similarities between neighboring patches, alongside hard constraints from stereo camera pairs and temporally adjacent images. Experimental evaluation on the Curiosity rover dataset (using 150 clusters) demonstrates that DCCML increases homogeneous clusters by 16.7% while reducing the Davies–Bouldin Index from 3.86 to 1.82 and boosting retrieval accuracy from 86.71% to 89.86%. This improvement enables more precise classification of Martian geological features, advancing our capacity to analyze and understand the planet’s landscape.

Homogeneity within a terrain image is crucial for the scientific categorization of the image. Images consisting of more than one terrain class are irrelevant to geologic tasks such as classification, and novelty detection which require granular terrain categories. Further, images containing far-away geological objects are less relevant for scientists interested in studying rock types for Martian paleoclimate and habitability. In this work, we use image segmentation to identify and eliminate images that show multiple terrains, and depth estimation to exclude images taken from a distance beyond a certain range. We then show an improvement in the performance of deep clustering of Martian terrain images qualitatively and discuss the resulting retrieval performance that helps scientists rapidly categorize geologic terrain images.

In this work, we present a novel clustering method that jointly learns a triplet model and the cluster assignments of the resulting representations of image patches from data acquired by the mast cameras on the MSL Curiosity rover. Deep clustering using metric learning (DCML) iteratively clusters the features using standard K - means clustering algorithm and uses the subsequent assignments as pseudo-labels to train a triplet network with online triplet mining method. The resulting model performs better than our baseline model [1] according to visual inspection of the cluster quality and simple clustering performance measures.

In the last decade, the sparse regression approach was established as a new paradigm in hyperspectral unmixing. This paper reviews various directions in sparse unmixing, starting from the initial formulation proposed by Prof. José Bioucas-Dias: Sparse Unmixing via variable Splitting and Augmented Lagrangian (SUnSAL). SUnSAL has paved the path towards algorithms accounting for spatial homogeneity, data collaborativity, structured dictionaries, among others. Despite being the first sparse regression algorithm widely exploited in hyperspectral unmixing, SUnSAL can be still considered competitive and its legacy lies in the plethora of subsequent algorithms that it inspired.

This study exploits recent advances in image calibration and feature extraction techniques for analysis of hyperspectral images acquired by the Compact Reconnaissance Imaging Spectrometer for Mars (CRISM) to characterize subtle geologic outcrops at the border of phyllosilicate-bearing and sulfate-bearing regions of Mars. Specifically, a unique spectral “doublet” feature at 2.21-2.23 and 2.26-2.28 μm is isolated to characterize salty regions that may represent a changing climate on Mars. The martian locations exhibiting these spectral features are identified and compared with terrestrial settings with similar geologic compositions.

In this work, we adapt a clustering method that jointly learns the parameters of a neural network and the cluster assignments of the resulting features to the unsupervised training of image patches from data acquired by the mast cameras on the MSL Curiosity rover. The method iteratively groups the features with a k-means clustering algorithm, and uses the subsequent assignments as supervision to update the weights of the network. The resulting model performs reasonably according to visual inspection of the cluster quality and simple clustering performance measures. The results however highlight the need for a strong validation via expert assessment of the morphological features within the scenes.

In this work we introduce two novel products obtained from CRISM hyperspectral data, a new generation of denoised and atmospherically-corrected cubes and refined mineral identification maps. The corrected spectral cubes exhibit more resolved subtle absorption features that are crucial for mineral identification. The improved mineral maps, produced with a technique based on generative adversarial models, was successful in more accurately characterizing the mineralogy of the Mars2020 landing site area at Jezero crater and helping address several scientific questions along a suggested traverse about the habitability of the environments on the crater floor, rim and delta.

In optical remote sensing, phenomena such as multiple scattering, shadowing, and spatial neighbor effects generate spectral reflectances that are nonlinear mixtures of the reflectances of the surface materials. Using hyperspectral images, the obtained spectral reflectances can be unmixed. We present a general method for creating nonlinear mixing models, based on a ray-based approximation of light and a graph-based description of the optical interactions. This results in a stochastic process which can be used to calculate path probabilities and contributions, and their weighted sum. In many cases, a closed-form equation can be obtained. We illustrate the approach by deriving several existing mixing models, such as linear, bilinear, and multilinear mixing (MLM) models popular in remote sensing, layered models for vegetation canopies, and intimate mineral mixtures. Furthermore, we use the proposed technique to derive a new mixing model, which extends the MLM model with shadowing. Experiments on artificial and real data show the positive traits of this model, which also demonstrates the power of the graphical model approach.

Spectra from a single intimate (nonlinear) mixture can be modeled as data points drawn from a smooth manifold. Spectral data sets containing hyperspectral observations of multiple intimate mixtures with some constituent materials in common can, therefore, be modeled as data clouds, in which each point is drawn from a union of manifolds that share a boundary. Two important steps in the processing of such data are to: 1) identify the different mixture manifolds present in the data and 2) invert the nonlinear mixing function by mapping each mixture manifold into some low-dimensional Euclidean space (manifold embedding). The present state-of-the-art algorithms for joint manifold clustering and embedding perform poorly for hyperspectral data, particularly in the embedding task. We propose a novel reconstruction-based algorithm for the improved clustering and the embedding of mixture manifolds. The algorithm attempts to reconstruct each target point as an affine combination of its nearest neighbors with an additional rank penalty on the neighborhood to ensure that only the neighbors on the same manifold as the target point are used in the reconstruction. The reconstruction matrix generated by this technique is both block diagonal and neighborhood-based, leading to improved clustering and embedding. The improved performance of the algorithm against its competitors is exhibited on a variety of simulated and real mixture data sets.

Several classes of endmember (EM) extraction algorithms based on the pure pixel assumption exist. Most of these algorithms employ some geometrical interpretation of the spectral mixing process, and use orthogonal projections, random projections, or some combination of them. Random projection based algorithms, such as pixel purity index, often find clusters of EM candidates which show high correlation, requiring a manual post-processing. Pure orthogonal projection based methods such as the simplex growing algorithm always yield a single, identical set of EMs, as the iteration process is fully deterministic. Mixed methods, such as VCA, can be highly random, and produce different sets of EMs each run. In this work, we present a new EM extraction algorithm which combines the positive aspects of orthogonal projection and random projection-based methods, resulting in a method which does not require manual intervention, possesses much less randomness than VCA, and is more flexible than fixed iterative methods. These properties are illustrated on a real data set, and compared with several different types of popular EM extraction algorithms.

We propose a new hyperspectral sparse unmixing method under the assumption of the availability of a spectral library. Hyperspectral signals inevitably possess non-linearity or distortion caused by the presence of endmembers outside of the collection, inaccurate measurement of atmosphere, and endmember mismatches. Since the spectral signals in the library are usually extremely coherent to each other, even a small distortion is problematic for sparse unmixing based on a linear model. To overcome this difficulty, we propose a new sparse unmixing method that simultaneously models a “background” that represents the contribution of these non-linearity and distortion. Experimental results show that our method not only improves the performance in mineral detection but also significantly promotes the sparsity of the solution without losing the quality of fit to observed spectral curves.

Spectral datasets acquired for unmixing are noisy and largely unlabeled (with unknown abundances). As a result the ability to accurately predict endmember abundances of surface samples is as important as the capacity to generate spectra from hypothetical abundances, e.g. endmembers from abundances sampled from the corner of a simplex. We construct a deep (semi-supervised) generative model to accomplish both these tasks while making use of the readily available unlabeled spectra and being able to encode environmental and instrumental nuisances. Our main technical contribution is that we train our model both forward and in reverse. The algorithm successfully identifies endmember spectra while isolating photometric effects and imaging errors as nuisances in a real dataset of intimately mixed samples.

High spectral resolution brings hyperspectral images with large amounts of information, which makes these images more useful in many applications than images obtained from traditional multispectral scanners with low spectral resolution. However, the high data dimensionality of hyperspectral images increases the burden on data computation, storage, and transmission; fortunately, the high redundancy in the spectral domain allows for significant dimensionality reduction. Band selection provides a simple dimensionality reduction scheme by discarding bands that are highly redundant, thereby preserving the structure of the data set. This paper proposes a new criterion for pointwise-ranking-based band selection that uses a nonhomogeneous hidden Markov chain (NHMC) model for redundant wavelet coefficients of each hyperspectral signature. The model provides a binary multiscale label that encodes semantic features that are useful to discriminate spectral types. A band ranking score considers the average correlation among the average NHMC labels for each band. We also test richer discrete-valued label vectors that provide a more finely grained quantization of spectral fluctuations. In addition, since band selection methods based on band ranking often ignore correlations in selected bands, we study the effect of redundancy elimination, applied on the selected features, on the performance of an example classification problem. Our experimental results also include an optional redundancy elimination step and test their effect on classification performance that is based on the selected bands. The experimental results also include a comparison with several relevant supervised band selection techniques.

Estimation of the number of endmembers (NOE) is an important first step in many hyperspectral unmixing applications. We present a new method for solving this problem, based on the statistics of the indegree distribution (IDD) of the data nearest neighbor graph. It is known that this IDD shows a high dependence on the intrinsic dimensionality (ID) of the data, and becomes skewed for increasing dimensionality. This effect is known as the hubness phenomenon, and we propose a technique that exploits this effect to derive an estimate for the NOE in a hyperspectral data set. While this number should have a trivial relation with the ID of the data set, this relation is often obscured by the large correlations that exist between endmember spectra and adjacent spectral bands. The proposed technique circumvents this problem by building representative statistics based on simulated hyperspectral data sets, and therefore performs much better than alternative techniques. Also several types of nonlinearly mixed data sets can be treated by the proposed technique, which is illustrated with bilinear data sets.

This paper proposes a new hyperspectral unmixing method for nonlinearly mixed hyperspectral data using a semantic representation in a semisupervised fashion, assuming the availability of a spectral reference library. Existing semisupervised unmixing algorithms select members from an endmember library that are present at each of the pixels; most such methods assume a linear mixing model. However, those methods will fail in the presence of nonlinear mixing among the observed spectra. To address this issue, we develop an endmember selection method using a recently proposed semantic spectral representation obtained via nonhomogeneous hidden Markov chain model for a wavelet transform of the spectra. The semantic representation can encode spectrally discriminative features for any observed spectrum, and therefore, our proposed method can perform endmember selection without any assumption on the mixing model. The experimental results show that in the presence of sufficiently nonlinear mixing, our proposed method outperforms dictionary-based sparse unmixing approaches based on linear models.

This paper explores the performance of sparse unmixing (SU) on non-linear mixtures. We consider SU as an endmember selection method from a spectral library and measure its performance using recently proposed approximately perfect recovery condition for sparse unmixing, comparing with non-negative least squares (NNLS). We also further explore the effect of thresholding on SU and NNLS. Simulations on various kinds of non-linear mixtures and an experiment on real hyperspectral data show that thresholding greatly improves the performance in endmember selection and NNLS could outperform SU when combined with thresholding.

Superpixel segmentation algorithms attempt to group contiguous image pixels which are in homogeneous regions into segments (superpixels). Superpixel segmentation maps have proven successful in improving the performance of unmixing algorithms on hyperspectral images. For hyperspectral images (HSIs), segment members must contain spectrally similar pixels, a requirement we refer to as segment uniformity. Existing superpixel segmentation algorithms which have been applied to HSIs provide no guarantees on the uniformity inside segments. In the absence of such guarantees, the only viable option is to make the segments small enough that uniformity is always ensured; this leads to an oversegmentation of the image. An accurate uniformity measure would lead to a more accurate segmentation. We propose a graph-based agglomerative approach that enforces segment uniformity by setting a threshold for maximum variability inside segments. The threshold is computed by a statistical analysis of the within-class and between-class spectral divergences of several mineral families of interest. We show that the proposed algorithm can be used to generate parsimonious segmentations and facilitate the computation of accurate mineralogical summaries for several simulated and real HSIs of terrestrial and planetary geological surfaces.

Classical unmixing algorithms focus primarily on scenarios with a single mixture. These techniques are easily extensible in the case of images with multiple discrete mixtures (i.e. no shared endmembers). Unmixing in scenarios with multiple mixtures with shared or common endmembers is significantly harder. Manifold clustering and embedding seem tailor-made for such a scenario, but generally these algorithms focus on intersecting manifolds (i.e. manifolds that pass through each other) rather than adjoining manifolds (i.e. manifolds that share a boundary) as is the case with mixtures. In this paper we propose a NNMF based technique for simultaneous manifold clustering and embedding of adjoining manifolds. The algorithm is based on including a clustering term in the objective for finding an appropriate reconstruction matrix. The performance of the new algorithm is tested on a toy dataset made of a couple of simulated manifolds which share a boundary and a simulated dataset made up of two ternary Hapke mixtures with two shared endmembers. The algorithm shows improvements on the state-of-the-art manifold clustering algorithms in terms of both clustering and embedding.

We consider the application of non-homogeneous hidden Markov chain (NHMC) models to the problem of hyperspectral signature classification. It has been previously shown that the NHMC model enables the detection of several semantic structural features of hyperspectral signatures. However, there are some aspects of the spectral data that are not fully captured by the proposed NHMC models such as the relatively smooth but fluctuating regions and the fluctuation orientations. In order to address these limitations, we propose an improved NHMC model based on Daubechies-1 wavelets in conjunction with an increased the model complexity. Experimental results show that the revised approach outperforms existing approaches relevant in classification tasks.

In this paper, we propose a new nonlinear approach which uses graphs for detecting endmembers in hyperspectral images. Endmembers are defined as the purest points of the image and lie on the boundary of the data cloud. The image is modeled by a graph and in order to reduce the effects of noise and artifacts existent in the image, the superpixel representation is used instead of pixel representation. Superpixels are the image segments with locally contiguous pixels. The nodes of the graph are the mean spectra of the superpixels and some similarity between each pair of the nodes defines their connectivity to each other. Since the endmembers are the extreme points of the data cloud, we use graph theoretic quantities to discriminate them from the central points in the data cloud. For the validation of the introduced method, we apply this approach to some real hyperspectral images and present the results of applying the method to one image.

Planetary missions such as the Compact Reconnaissance Imaging Spectrometer for Mars (CRISM) can benefit from the use of automatic approaches and statistical learning techniques due to the amount of data involved. Thanks to its high sensor resolution, CRISM data volumes overwhelm scientists capacity for exhaustive manual analysis. Planetary investigations would benefit from an automated process that could identify the unique spectral signatures present in a CRISM scene and store them for further examination or interpretation. If installed aboard an orbital system, such a tool could relieve transmission constraints for high-bandwidth hyper spectral datasets by giving priority to the most informative data products. This paper introduces an algorithm that extracts image endmembers of a CRISM scene, which can be used as the scene concise mineralogical representation for cataloging purposes, in addition to existing browse products and parameter maps. The approach uses robust techniques, resilient to CRISM noise. This work benefits from the results of previous efforts [6] and it is currently being extended to other hyperspectral datasets.

The simulation of remote-sensing hyperspectral images is a useful tool for a variety of tasks such as the design of systems, the understanding of the image formation process, and the development and validation of data processing algorithms. The lack of ground truth and the incomplete knowledge of the Martian environment make simulation studies of Mars hyperspectral images a useful tool for automated analysis of Mars data. Hyperspectral near-infrared scenes of mineral mixtures have been simulated to analyze the contributions of surface minerals, atmosphere, and sensor noise on images of Mars. Modeling the remote-sensing process creates a means for the independent analysis of the influence of the environment and instruments on the detection accuracy of the surface composition (e.g., the scene endmembers). The end-to-end model builds surface reflectance scenes based on laboratory sample spectra, creates atmospheric effects using radiative transfer routines, simulates the instrument response function using CRISM data files, and adds instrument noise from thermal and other sources. The purpose of this paper is to understand the hyperspectral remote-sensing process to eventually enable the elevated detection accuracy of minerals on the surface of Mars. The viability of a linear approximation of the complete model is also investigated. The approximation is compared to the complete model in an image classification task.