The fusion of hyperspectral images (HSI) and light detection and ranging (LiDAR) data reflects strong complementary advantages, significantly enhancing the accuracy and robustness of land cover classification. However, inherent differences in sensor mechanisms and imaging principles lead to substantial heterogeneity in feature representation and scene characterization between these two modalities, posing challenges for effective multisource remote sensing (RS) fusion. To address this issue, a shared-specific graph construction oriented cross-modal fusion network (S2GCCFN) is proposed for HSI and LiDAR data joint classification. Our framework facilitates deep feature interaction between heterogeneous HSI and LiDAR data to achieve modal homogenization, yielding a high-performance joint representation. The core motivation of this study is to capture one assimilation modality (AM) by discovering latent crossmodal mapping strategies from both HSI and LiDAR data simultaneously. Notably, S2GCCFN mitigates modal heterogeneity while enhancing complementary information exchange. S2GCCFN considers one dynamic graph mapping mechanism that adaptively integrates intramodal and crossmodal features comprehensively to align dual RS modalities and promote RS interaction. By establishing shared feature spaces, S2GCCFN enhances intermodal consistency while preserving modality-specific characteristics, ultimately generating highly discriminative AMs for joint classification. Furthermore, S2GCCFN incorporates an AM reconstruction decoder to jointly address modality reconstruction and classification. Experimental results demonstrate that S2GCCFN outperforms state-of-the-art techniques, improving overall accuracy by 2.66% and 1.34% on average on both datasets, respectively.

Combining their complementary properties, using hyperspectral image (HSI) and light detection and ranging (LiDAR) data improves classification performance. Nevertheless, the heterogeneous capturing instruments and distribution characteristics of these two remote sensing (RS) modalities always limit their application scopes in on-ground observation-related domains. This heterogeneity hinders capturing the crossmodal connection for discriminant information extraction and exchange. In this letter, we propose an assimilation modality mapping guided crossmodal fusion network (AM2CFN) for HSI and LiDAR data joint classification. Our motivation is to explore one RS assimilation modality (RSAM) by exploiting one latent crossmodal mapping strategy from HSI and LiDAR data simultaneously to remove the effect of modality heterogeneity and contribute to information exchange. AM2CFN constructs one level-wise assimilating encoder to simulate modality heterogeneity and enhance regional consistency. Modality intrinsic features are captured in this encoder to provide knowledge for modality assimilation. Furthermore, one RSAM balancing HS and LiDAR properties is explored. AM2CFN constructs one RSAM reconstruction decoder for modality reconstruction and classification. Dual constraints based on solid angle and Kullback-Leibler divergence are considered to restrain the information exchange process toward the optimal direction. Experiments show that AM2CFN outperforms several state-of-the-art techniques qualitatively and quantitatively. AM2CFN increases the overall accuracy (OA) by 2.46% and 1.62% on average on the Houston and MUUFL datasets. The codes will be available athttps://github.com/GEOywb/AM2CFN

Hyperspectral images (HSIs) and digital surface models (DSMs) derived from light detection and ranging (LiDAR) data are packed with rich on-ground object characteristics, making their joint classification crucial for accurate remote sensing (RS) identification. However, variations in illumination properties hinder efficient and precise crossmodal information interaction. This phenomenon further exacerbates the difficulty in correctly identifying corresponding spectral curves, ultimately resulting in suboptimal performance. In this paper, a hybrid updating graph motivated illumination-wise property unification network (HugIpuNet) is proposed to solve these challenges. Our core motivation is to unify global illumination descriptions with assistance from elevations for capturing precise multimodal properties from HSIs and DSMs derived from LiDAR data. HS spectral curves and LiDAR elevations interact constantly, considering their geological connection to strengthen their coupling capability. HugIpuNet further considers several illumination-wise mechanisms to address intricate and variable environmental conditions. The whole structure utilizes various hybrid updating graphs to enhance the attribute unification ability in complex environmental scenarios. Dynamic adjacent matrices are constructed to introduce uncertainty to limit the effect caused by description distortions. Experiments show an average 2.86% improvement in overall accuracy (OA) compared with multiple state-of-the-art approaches. The detailed datasets and codes will be available at https://github.com/weixinttt/HugIpuNet.

Hyperspectral (HS) and light detection and ranging (LiDAR) fusion for land-cover identification in multimodal tasks is always restricted owing to modality heterogeneity, especially when annotated samples are unavailable. Clustering techniques become pivotal in these unsupervised scenarios, yet their inability to offer interpretable fusion mechanisms poses challenges in pursuing enhanced identification performance. In this article, an Intuitive Assimilation Modality-driven Crossmodal Subspace Clustering (IamCSC) method is proposed to exploit intuitive crossmodal information. Different from existing multiview clustering approaches, our core innovation is that IamCSC distills an intuitive assimilation modality (IAM) with distinguishing geological characteristics of HS and LiDAR images simultaneously. The IAM with a small number of IAM channels still preserves the spectral similarity and diversity of neighboring samples. Meanwhile, it reflects sample elevations by peak regions along the IAM channel dimension. Specifically, the IAM connects and balances these two RS modalities to learn an assimilation cluster structure from modality-specific subspace representations. Several matrix-wise constraints are considered for capturing the optimal modality-shared representation. Given that IamCSC does not exhibit over-reliance on spatial neighborhood contributions, it achieves satisfactory performance particularly in image pairs with larger ground sampling distances (GSDs). Multiple experiments clearly prove the effectiveness of IamCSC over state-of-the-art techniques qualitatively and quantitatively. The codes are provided inhttps://github.com/GEOywb/IamCSC.

Numerous deep learning (DL) studies have indicated that fusing hyperspectral (HS) and light detection and ranging (LiDAR) data is effective for land-cover classification. However, the sequential characteristics (seqCHAs) in the spatial domain are always ambiguous and neglected. In this letter, we propose a deep crossmodal sequential interaction network (CsiNet) for HS and LiDAR data joint classification. We aim to verify the contributions of crossmodal seqCHAs in multimodal joint classification tasks and present an effective crossmodal sequential flattening (SF) strategy. Specifically, CsiNet sorts the neighboring samples in terms of the spectral and 3-D spatial diversities between the corresponding samples and the central one. Notably, the 3-D spatial diversity considers the shared sample positions in both modalities and the sample elevations in LiDAR data simultaneously. CsiNet is capable of extracting crossmodal sequential features comprehensively by long and short-term memory (LSTM) layers and better simulating sequential properties of samples compared with convolutional layer based networks. Experiments conducted on the Muufl Gulfport (MUUFL) and Houston 2013 datasets prove that CsiNet outperforms several state-of-the-art techniques qualitatively and quantitatively. When using 1% training samples per category, the overall accuracies of CsiNet on both datasets achieve 90.27% and 92.41% and are increased by 0.27% and 0.17% than the best comparison technique, respectively. Ablation experiments verify the effectiveness of CsiNet by replacing the crossmodal SF strategy with several alternative ones. All codes are available athttps://github.com/GEOywb/CsiNet.

Fusing hyperspectral (HS) and light detection and ranging (LiDAR) data is capable of enhancing land-cover interpretation capability in multimodal remote sensing (RS) tasks. Nevertheless, nonuniform surrounding illumination in real-world capturing scenarios tends to distort the spectral curves of on-ground objects, inevitably resulting in poor classification performance. This nonuniform illumination further interferes with the cross-modal RS information interaction procedure. In this article, an Illumination Insensitive Modalitywise Assimilation guided Multistage Interaction network (MAMInet II) is proposed to tackle this challenge. Our primary motivation is to eliminate complicated illumination interference (CII) and capture high-level modalitywise assimilation information (MAI) from HS and LiDAR data simultaneously. Notably, the geological connection between these two RS modalities is emphasized in MAMInet II, making the joint classification framework interpretable. Specifically, MAMInet II presents one channelwise illumination removal mechanism by simulating real-world illumination conditions and associating HS spectral curves with LiDAR elevations. One generative switching mechanism contributes to closing the cross-modal RS distribution gap and enhancing the cross-modal capturing ability. These mechanisms are extended to multiscale and multistage versions for aligning local and global RS properties. Experiments illustrate that MAMInet II outperforms several state-of-the-art techniques qualitatively and quantitatively. All source codes are available athttps://github.com/weixinttt/MAMInet-II.

In multimodal data fusion and land-cover interpretation tasks, the fusion interpretability between hyperspectral image (HSI) and light detection and ranging (LiDAR) data is always nontrivial to be clarified. Furthermore, the heterogeneous sample and distribution variances of these two remote sensing (RS) modalities impede the joint classification performance. In this paper, a Hyperspectral Intrinsic Image Decomposition guided Data Fusion Network (HI2D2FNet) is proposed. Generally, classic hyperspectral intrinsic image decomposition (HIID) performs well in image enhancement and shadow removal. It decomposes one HSI into one reflectance component and one shading component. Inspired by the core mechanism of HIID, our motivation is to preliminarily exploit its potential for multimodal RS data fusion and explore the inherent modality connection between HSI and LiDAR data from the intrinsic perspective. Specifically, compared with existing techniques, HI2D2FNet is capable of fusing the horizontal geometry information in the shading component with the vertical geometry information in the LiDAR data from both sample and distribution perspectives in the spatial domain. The generated cross-modal geometry feature contributes to guiding the reflectance stream optimization. This unique fusion framework connects both modalities with respect to geometry information and enhances the specific fusion interpretability. The decomposition and fusion modules in HI2D2FNet are optimized simultaneously in a novel alternative optimization pattern. Furthermore, several unique cross-modal constraints in terms of prior RS properties are presented. Experiments conducted on three widely available datasets prove the superiority of HI2D2FNet over state-of-the-art techniques. The source codes will be available at https://github.com/GEOywb/HI2D2FNet.

Remote sensing (RS) modalities from multi-sensor platforms, including hyperspectral (HS) and light detection and ranging (LiDAR) data, have been garnering increasing attention in overcoming the lack of information diversity. However, the specific modality correlation is always weakened and neglected. By reducing the spectral uncertainty, hyperspectral intrinsic image decomposition (HIID) has been proven to be effective in enhancing the HS image quality. It is an ill-posed problem and is challenged to capture the inherent modality connection. This paper proposes a shadow mask driven multimodal intrinsic image decomposition (smMIID) for HS and LiDAR data fusion. Notably, smMIID creatively clarifies and activates the potential of the crossmodal information in RS modalities and builds on the strength of the LiDAR elevation information when constructing HIID constraints. Our motivation is to overcome the deficiency of information diversity and modality correlation in existing IID based frameworks for better data fusion performance. Classic IID methods decompose the HS image into one reflectance component (RC) and one shading component (SC). There are three constraints in smMIID: 1) the fundamental constraint on RC and SC, 2) the HS-LiDAR hybrid gradient based constraint on RC and 3) the LiDAR gradient based constraint on SC. A shadow mask is obtained and utilized to guarantee spectral consistency when combining neighboring samples. Experiments on HS and LiDAR datasets prove that smMIID outperforms other techniques in terms of visualization and classification. The analyses are provided theoretically and experimentally to demonstrate that smMIID is robust to shadow mask generation.

Hyperspectral images (HSIs) provide abundant high-quality spectral information through an immense number of spectral channels, which can be used to classify on-ground objects for Earth observation accurately. However, these highly correlated channels and complex informative features always limit the application of HSIs. In this letter, we propose a multilevel dual-direction modifying variational autoencoder (MD2MVAE) for hyperspectral feature extraction. Its architecture is inspired by the spectral–spatial coherence in HSIs. Our motivation is to modify spectral sequential features by spatial sequential features in a multilevel dual-direction network. The dual-direction strategy has two implications: 1) the spatial continuity is captured by flattening neighboring samples in dual classic directions and 2) the spectral–spatial continuities are captured in the forward and backward directions. This multilevel dual-direction network provides a feasible way to avoid the loss of spatial information when flattening samples in the spatial domain, without using any convolutional layers. Inspired by our previous work, a variational autoencoder (VAE)-based network is used to enhance the noise immunity of latent features. To preserve the consistency of spectral–spatial sequential features, a combined loss function based on the solid angle on a unit sphere is proposed for parameter optimization. Two typical datasets are selected as benchmarks to show the effectiveness of MD2MVAE, compared with the state-of-the-art methods.

Tridimensional empirical mode decomposition (TEMD) has been successfully employed as a powerful tool for hyperspectral classification. It achieves good performance to analyze nonlinear/non-stationary data and capture abstract features of hyperspectral images (HSIs). However, the major challenge is how to adequately exploit and balance decomposed edge-wise (EW) and trend-wise (TW) features. To address this issue, we propose an accelerated adaptive feature balance technique (A2FBT) based on TEMD. Due to complex discriminant information, A2FBT initially decomposes the HSI into varying oscillations adequately, including several tridimensional intrinsic mode functions (TIMFs) and one residual (RES). They contain high frequency EW and low frequency TW features, respectively. To further accelerate the decomposition, a k-d tree based search strategy is proposed for filter size selection. To bridge the gaps between the RES and all TIMFs, a novel adaptive feature balance strategy is presented by assigning adaptive trade-offs. We aim to balance multiple features adequately according to the intrinsic discriminant information. A combined metric based strategy is provided for trade-off calculation, measuring the mutual similarities between oscillations comprehensively. A2FBT pays more attention to these similarities, reinforcing the respective contributions of EW and TW features. Experiments on four typical hyperspectral datasets illustrate the effectiveness of A2FBT compared with several state-of-the-art techniques.

Hyperspectral image processing is faced with difficulties considering its redundant features and complex information. Studies on hyperspectral feature extraction in the deep learning domain have become increasingly popular. The mainstream techniques fully consider the spatial information in local neighborhoods when extracting spectral features by constructing deep neural networks. Deep generative models simulate the intrinsic structure of samples by adequately training, showing their potential values for signal processing. In this article, a convolutional two-stream network (cs2GAN-FE) based on the improved Wasserstein generative adversarial network (WGAN) is proposed for unsupervised hyperspectral spatial–spectral feature extraction. The improved WGAN is composed of one generator and one discriminator; the former perceives real data distributions, and the latter determines the attribution of generated data. The designed two-stream strategy is not a simple extension of a one-stream strategy and considers both the static spectral–spatial information and the dynamic spectral reflectance variation in multiple bands. Intrinsic spatial–spectral features are extracted by the trained discriminator considering sample distributions and feature relationships. The loss function is also improved for the unique structure of cs2GAN-FE. Various state-of-the-art techniques are chosen for comparison. Experimental results show the feasibility and potential of this network. Besides, experiments with the random split and the disjointed split both show that the proposed method can outperform other comparison techniques.

Hyperspectral image with high dimensionality always increases the computational consumption, which challenges image processing. Deep learning models have achieved extraordinary success in various image processing domains, which are effective to improve classification performance. There remain considerable challenges in fully extracting abundant spectral information, such as the combination of spatial and spectral information. In this article, a novel unsupervised hyperspectral feature extraction architecture based on spatial revising variational autoencoder (AE) (UHfeSRVAE) is proposed. The core concept of this method is extracting spatial features via designed networks from multiple aspects for the revision of the obtained spectral features. Multilayer encoder extracts spectral features, and then, latent space vectors are generated from the obtained means and standard deviations. Spatial features based on local sensing and sequential sensing are extracted using multilayer convolutional neural networks and long short-term memory networks, respectively, which can revise the obtained mean vectors. Besides, the proposed loss function guarantees the consistency of the probability distributions of various latent spatial features, which obtained from the same neighbor region. Several experiments are conducted on three publicly available hyperspectral data sets, and the experimental results show that UHfeSRVAE achieves better classification results compared with comparison methods. The combination of spatial feature extraction models and deep AE models is designed based on the unique characteristics of hyperspectral images, which contributes to the performance of this method.

Feature selection, which is called band selection for hyperspectral data, is widely used for hyperspectral images. A novel hyperspectral band selection method based on combined fast and adaptive tridimensional empirical mode decomposition (cFATEMD) is proposed in this study. The hyperspectral data is decomposed into a set of tridimensional intrinsic mode functions (TIMFs) and a residual (RES) by FATEMD, which can reduce high‐frequency noise and signal. A stop condition of the decomposition is proposed based on the k‐means clustering algorithm and the Dunn validity index, which can prevent excessive decomposition and make generated RES contain as much useful information as possible. In consideration of the useful information in decomposition results, these TIMFs and the RES are combined into a new data based on the spectral similarity between themselves and the original data. Four state‐of‐the‐art band selection methods, cooperating with the proposed cFATEMD, are used to select bands by the new combined data. Several experiments are conducted on three publicly available hyperspectral datasets and the results are compared with corresponding methods’ results using the original data. Experimental results demonstrate that the proposed method yields great classification appearance.

Hyperspectral technology is a huge leap of remote sensing technology, however, how to make full use of its rich information is a difficult problem. A novel hyperspectral band selection method based on 3D empirical mode decomposition (3D-EMD) is proposed by constructing the upper approximation and the lower approximation of the whole image. The hyperspectral image (HSI) is decomposed into a set of tridimensional intrinsic mode functions (TIMFs) and bands are selected from the HSI using each TIMF based on optimum index factor (OIF). These bands are combined into a band combination. Experimental results demonstrate that the proposed method yields improved decomposition performance on the HSI and increases classification accuracy.

Hyperspectral data with high dimensionality always needs more storage space and increases the computational consumption, manifold learning is extensively used in dimensionality reduction. A novel dimensionality reduction method based on manifold learning is proposed through learning a stable local manifold representation. Four adjacency graphs are constructed to model the interclass similarity, interclass diversity, intraclass similarity and intraclass diversity respectively, and then merge these graphs into the discriminant objective function for linear dimensionality reduction. The classification results in the use of different methods are compared with and experimental results show that the proposed method is effective and it is superior to comparison methods.

A novel deep learning classification method for hyperspectral data based on convolutional neural network is proposed in this paper. Deep learning means bringing multiple layers instead of one to the structure. Through convolution layers and pooling layers, the features in different layers are extracted from original spectral feature images. The key of this method is to restructure spectral feature images and choose convolution filters with a reasonable size, so that the spectral features of different land coverings in high dimensions can be extracted properly. In our experiments, proposed method was applied for hyperspectral data in several different situations, and preferable classification performance were obtained through relative parameters adjustment, which were given recommended scope during our comparative experiments.