Optical satellites, with their diverse band layouts and ground sampling distances, supply indispensable evidence for tasks ranging from ecosystem surveillance to emergency response. However, significant discrepancies in band composition and spatial resolution across different optical sensors present major challenges for existing Remote Sensing Foundation Models (RSFMs). These models are typically pretrained on fixed band configurations and resolutions, making them vulnerable to real world scenarios involving missing bands, cross sensor fusion, and unseen spatial scales, thereby limiting their generalization and practical deployment. To address these limitations, we propose Any Optical Model (AOM), a universal RSFM explicitly designed to accommodate arbitrary band compositions, sensor types, and resolution scales. To preserve distinctive spectral characteristics even when bands are missing or newly introduced, AOM introduces a spectrum-independent tokenizer that assigns each channel a dedicated band embedding, enabling explicit encoding of spectral identity. To effectively capture texture and contextual patterns from sub-meter to hundred-meter imagery, we design a multi-scale adaptive patch embedding mechanism that dynamically modulates the receptive field. Furthermore, to maintain global semantic consistency across varying resolutions, AOM incorporates a multi-scale semantic alignment mechanism alongside a channel-wise self-supervised masking and reconstruction pretraining strategy that jointly models spectral-spatial relationships. Extensive experiments on over 10 public datasets, including those from Sentinel-2, Landsat, and HLS, demonstrate that AOM consistently achieves state-of-the-art (SOTA) performance under challenging conditions such as band missing, cross sensor, and cross resolution settings. These results highlight AOM as a crucial step toward building truly general-purpose RSFMs.

Hyperspectral images (HSIs) and multispectral images (MSIs) fusion is a hot topic in the remote sensing society. A high-resolution HSI (HR-HSI) can be obtained by fusing a low-resolution HSI (LR-HSI) and a high-resolution MSI (HR-MSI) or RGB image. However, most deep learning-based methods require a large amount of HR-HSIs for supervised training, which is very rare in practice. In this paper, we propose a coupled diffusion posterior sampling (CDPS) method for HSI and MSI fusion in which the HR-HSIs are no longer required in the training process. Because the LR-HSI contains the spectral information and HR-MSI contains the spatial information of the captured scene, we design an unsupervised strategy that learns the required diffusion priors directly and solely from the input test image pair (the LR-HSI and HR-MSI themselves). Then, a coupled diffusion posterior sampling method is proposed to introduce the two priors in the diffusion posterior sampling which leverages the observed LR-HSI and HR-MSI as fidelity terms. Experimental results demonstrate that the proposed method outperforms other state-of-the-art unsupervised HSI and MSI fusion methods. Additionally, this method utilizes smaller networks that are simpler and easier to train without other data.

Pan-sharpening aims to generate a spatially and spectrally enriched multi-spectral image by integrating complementary cross-modality information from low-resolution multi-spectral image and texture-rich panchromatic counterpart. In this work, we propose a WKV-sharing embraced random shuffle RWKV high-order modeling paradigm for pan-sharpening from Bayesian perspective, coupled with random weight manifold distribution training strategy derived from Functional theory to regularize the solution space adhering to the following principles: 1) Random-shuffle RWKV. Recently, the Vision RWKV model, with its inherent linear complexity in global modeling, has inspired us to explore its untapped potential in pan-sharpening tasks. However, its attention mechanism, relying on a recurrent bidirectional scanning strategy, suffers from biased effects and demands significant processing time. To address this, we propose a novel Bayesian-inspired scanning strategy called Random Shuffle, complemented by a theoretically-sound inverse shuffle to preserve information coordination invariance, effectively eliminating biases associated with fixed sequence scanning. The Random Shuffle approach mitigates preconceptions in global 2D dependencies in mathematical expectation, providing the model with an unbiased prior. In line with similar spirit of Dropout, we introduce a testing methodology based on Monte Carlo averaging to ensure the model’s output aligns more closely with expected results. 2) WKV-sharing high-order. Regarding KV’s attention score calculation in spatial mixer of RWKV, we leverage WKV-sharing mechanism to transfer KV activations across RWKV layers, achieving lower latency and improved trainability, and revisit the channel mixer in RWKV, originally a first-order weighting function, and redevelop its high-order potential by sharing the gate mechanism across RWKV layer. Comprehensive experiments across pan-sharpening benchmarks demonstrate our model’s effectiveness, consistently outperforming state-of-the-art alternatives

Panchromatic (PAN) and multi-spectral (MS) remote satellite image fusion, known as pan-sharpening, aims to produce high-resolution MS images by combining the complementary information from the high-resolution, texture-rich PAN and the low-resolution but high spectral-resolution MS counterparts. Despite notable advancements in this field, the current state-of-the-art pan-sharpening techniques do not explicitly address the spatial resolution mismatching problem between the two modalities of PAN and MS images. This mismatching issue can lead to misalignment in feature representation and the creation of blurry artifacts in the model output, ultimately hindering the generation of high-frequency textures and impeding the performance improvement of such methods. To address the aforementioned spatial resolution mismatching problem in pan-sharpening, we propose a novel modality-aware feature-aligned pan-sharpening framework in this paper. The framework comprises three primary stages: modality-aware feature extraction, modality-aware feature aligning, and context integrated image reconstruction. First, we introduce the half-instance normalization strategy as the backbone to filter out the inconsistent features and promote the learning of consistent features between the PAN and MS modalities. Second, a learnable modality-aware feature interpolation is devised to effectively address the misalignment issue. Specifically, the extracted features from the backbone are integrated to predict the transformation offsets of each pixel, which allows for the adaptive selection of custom contextual information and enables the modality-aware features to be more aligned. Finally, within the context of the interactive offset correction, multi-stage information is aggregated to generate the feasible pan-sharpened model output. Extensive experimental results over multiple satellite datasets demonstrate that the proposed algorithm outperforms other state-of-the-art methods both qualitatively and quantitatively, exhibiting great generalization ability to real-world scenes.

Multimodal image fusion involves tasks like pan-sharpening and depth super-resolution. Both tasks aim to generate high-resolution target images by fusing the complementary information from the texture-rich guidance and low-resolution target counterparts. They are inborn with reconstructing high-frequency information. Despite their inherent frequency domain connection, most existing methods only operate solely in the spatial domain and rarely explore the solutions in the frequency domain. This study addresses this limitation by proposing solutions in both the spatial and frequency domains. We devise a Spatial-Frequency Information Integration Network, abbreviated as SFINet for this purpose. The SFINet includes a core module tailored for image fusion. This module consists of three key components: a spatial-domain information branch, a frequency-domain information branch, and a dual-domain interaction. The spatial-domain information branch employs the spatial convolution-equipped invertible neural operators to integrate local information from different modalities in the spatial domain. Meanwhile, the frequency-domain information branch adopts a modality-aware deep Fourier transformation to capture the image-wide receptive field for exploring global contextual information. In addition, the dual-domain interaction facilitates information flow and the learning of complementary representations. We further present an improved version of SFINet, SFINet++, that enhances the representation of spatial information by replacing the basic convolution unit in the original spatial domain branch with the information-lossless invertible neural operator. We conduct extensive experiments to validate the effectiveness of the proposed networks and demonstrate their outstanding performance against state-of-the-art methods in two representative multimodal image fusion tasks: pan-sharpening and depth super-resolution.

Multi-modal image fusion aims to generate a fused image by integrating and distinguishing the cross-modality complementary information from multiple source images. While the cross-attention mechanism with global spatial interactions appears promising, it only captures second-order spatial interactions, neglecting higher-order interactions in both spatial and channel dimensions. This limitation hampers the exploitation of synergies between multi-modalities. To bridge this gap, we introduce a Synergistic High-order Interaction Paradigm (SHIP), designed to systematically investigate spatial fine-grained and global statistics collaborations between the multi-modal images across two fundamental dimensions: 1) Spatial dimension: we construct spatial fine-grained interactions through element-wise multiplication, mathematically equivalent to global interactions, and then foster high-order formats by iteratively aggregating and evolving complementary information, enhancing both efficiency and flexibility. 2) Channel dimension: expanding on channel interactions with first-order statistics (mean), we devise high-order channel interactions to facilitate the discernment of inter-dependencies between source images based on global statistics. We further introduce an enhanced version of the SHIP model, called SHIP++ that enhances the cross-modality information interaction representation by the cross-order attention evolving mechanism, cross-order information integration, and residual information memorizing mechanism. Harnessing high-order interactions significantly enhances our model's ability to exploit multi-modal synergies, leading in superior performance over state-of-the-art alternatives, as shown through comprehensive experiments across various benchmarks in two significant multi-modal image fusion tasks: pan-sharpening, and infrared and visible image fusion.

Object detection is a fundamental task in computer vision that involves accurately locating and classifying objects within images or video frames. In remote sensing, this task is particularly challenging due to the high resolution, multi-scale features, and diverse ground object characteristics inherent in satellite and UAV imagery. These challenges necessitate more advanced approaches for effective object detection in such environments. While deep learning methods have achieved remarkable success in remote sensing object detection, they typically rely on large amounts of labeled data. Acquiring sufficient labeled data, particularly for novel or rare objects, is both challenging and time-consuming in remote sensing scenarios, limiting the generalization capabilities of existing models. To address these challenges, few-shot learning (FSL) has emerged as a promising approach, aiming to enable models to learn new classes from limited labeled examples. Building on this concept, few-shot object detection (FSOD) specifically targets object detection challenges in data-limited conditions. However, the generalization capability of FSOD models, particularly in remote sensing, is often constrained by the complex and diverse characteristics of the objects present in such environments. In this paper, we propose the Generalization-Enhanced Few-Shot Object Detection (GE-FSOD) model to improve the generalization capability in remote sensing FSOD tasks. Our model introduces three key innovations: the Cross-Level Fusion Pyramid Attention Network (CFPAN) for enhanced multi-scale feature representation, the Multi-Stage Refinement Region Proposal Network (MRRPN) for more accurate region proposals, and the Generalized Classification Loss (GCL) for improved classification performance in few-shot scenarios. GE-FSOD demonstrates superior robustness and accuracy in remote sensing FSOD tasks through these enhancements. Extensive experiments on the DIOR and NWPU VHR-10 datasets show that our model achieves state-of-the-art performance, significantly advancing the field of few-shot object detection in remote sensing. The source code is available at (https://github.com/leenamx/GE-FSOD).

Nowadays, various graph convolutional networks (GCNs) to process graph-structured data have been proposed for hyperspectral image (HSI) classification. Nevertheless, most GCN-based HSI classification methods emphasize graph node feature aggregation instead of graph pooling, resulting in them being shallow networks and unable to extract deep discriminative features. Besides, to obtain the new graph after the pooling layer, current graph pooling methods used for HSI classification just consider node feature information to select important nodes and directly discard unselected nodes, which could be a subjective process and may cause information loss. To solve this issue, we propose a novel graph U-Net with topology-feature awareness pooling (the so-called TFAP graph U-Net) for HSI classification considering a deep network to extract compelling features and automatically selecting nodes beneficial to classification. More specifically, to establish a more precise pooled graph, the graph’s topology structure and node feature information are taken into account, making the node selection process more convincing and objective. Furthermore, to allow that graph nodes preserve more useful graph information, our method aggregates node features from neighboring nodes that may not be selected, which can alleviate the loss of information during the pooling process. Moreover, a cross-attention module (CAM) is used to filter out irrelevant or noisy features. Finally, we evaluate the proposed method on three public HSI datasets, i.e., Indian Pines, University of Pavia (PaviaU), and University of Houston. The experimental results demonstrate the superiority of the proposed approach compared with other state-of-the-art methods.

Hyperspectral image (HSI) clustering is a fundamental yet challenging task that typically operates without training labels. Recent advancements in deep graph clustering methods have shown promise for HSI due to their ability to effectively encode spatial structural information. However, limitations such as inadequate utilization of structural information, poor feature representation, and weak graph update capabilities hinder their performance. In this article, we propose an adaptive homophily structure graph clustering (AHSGC) method for HSI. Our approach begins with the generation of homogeneous regions to process HSI and construct the initial graph. Next, we design an adaptive filter graph encoder that captures both high and low-frequency features for subsequent processing. We then develop a graph embedding clustering self-training decoder using KL Divergence to generate pseudo-labels for network training. To enhance graph learning, we introduce homophily-enhanced structure learning, which updates the graph based on the clustering task. This involves estimating node connections through orient correlation estimation and dynamically adjusting graph edges via graph edge sparsification. Finally, we implement joint network optimization to facilitate self-training and graph updates, with K-means used to express latent features. The clustering accuracy on three datasets is 83.60%, 63.65%, and 86.03%, the FLOPs are 3.57G, 30.62G, and 2.95G. The source code will be available athttps://github.com/DY-HYX.

Pansharpening is the critical technology for generating high-resolution (HR) multispectral (MS) images by learning the cross-modality complementary representations between the panchromatic (PAN) images and low-resolution (LR) MS images. Though methods based on convolutional neural networks (CNNs) have dominated the pansharpening community, they still suffer from the limited global modeling capability due to the inherent property of the convolutional operator. To remedy this common limitation, the transformer family has recently gained great popularity in this field. However, existing cascaded transformer designs inevitably introduce a heavy memory footprint and computational cost due to the dense dot-product self-attention (SA) computation. More importantly, these paradigms simply ignore the innate sparsity of remote sensing images, leading to information redundancy and a challenging optimization process. To alleviate these issues, we propose the bilateral adaptive evolution transformer (BAEFormer), which is built upon two core mechanisms: bilateral attention computation and adaptive attention evolution. Specifically, we first decompose the conventional quadratic complexity SA into linear-degree height and width computing at the first stage, respectively, which significantly reduces the computational complexity. Given the data-specific properties, furthermore, we devise a novel yet effective neighboring layer-dependent strategy to adaptively update the attention map of two spatial dimensions, thereby avoiding the repetitive SA computation while taking into account the dynamics toward the evolution of attention weights. Our model, called BAEFormer, outperforms other state-of-the-art pansharpening methods on various remote sensing datasets while showing fewer network parameters and computational requirements. The code is available athttps://github.com/coder-JMHou/BAEFormer.

Pan-sharpening essentially to boost the spatial resolution of a multispectral (MS) image guided by its paired panchromatic (PAN) image. In other words, this process intricately integrates the high-frequency components extracted from texture-rich PAN images into the low-resolution (LR) MS images, resulting in texture-rich MS images. Though existing deep learning (DL)-based techniques have made impressive performance compared with traditional algorithms, they still face challenges in accurately restoring high-frequency details in MS images, thus limiting overall pan-sharpening performance. In addition, reference high-resolution (HR) MS images are often underutilized, typically serving only as training labels. In this work, we present a general high-frequency enhancement framework for pan-sharpening, which is implemented through a cooperative optimization strategy using mutual information (MI) maximization and contrastive learning. Specifically, our model comprises two fundamental modules: the high-frequency feature alignment (HFFA) module and the high-frequency detail calibration (HFDC) module. The first employs MI maximization to align the high-frequency semantic statistical distribution between PAN images and reference HRMS images. The latter is designed to calibrate the high-frequency components of MS modality under the guidance of the PAN counterparts through the contrastive learning constraint, thereby producing more accurate high-frequency information on MS modality. By integrating the calibrated high-frequency features of MS modality and those of PAN modality, we can obtain a more comprehensive and precise high-frequency feature representation of these two modalities, facilitating the reconstruction of LRMS images. Our model, incorporating the aforementioned key elements, significantly surpasses other state-of-the-art (SOTA) techniques across multiple satellite datasets in both quantitative and qualitative experiments. Moreover, the real-world full-resolution and cross-sensor assessments testify to its exceptional generalization capabilities. The code is available athttps://github.com/Vcocoi/CONet.

In this work, we reveal an interesting observation within the multi-spectral modality: high-frequency components exhibit a long-tailed distribution, in contrast to the Gaussian distribution of dominant low-frequency components. This dual-distribution characteristic presents a challenge for network optimization, leading to overfitting on low-frequency information while neglecting essential high-frequency details. Addressing this issue from a causal inference perspective, we identify optimizer momentum as a confounding factor that biases models towards focusing on the head part of the high-frequency distribution during training. To counteract this effect, we propose a novel optimization strategy and supplement the global-modeling network architecture to balance the frequencies learning. In the training stage, we employ the Recurrent Weighted Key-Value (RWKV) architecture, which features a global receptive field, to effectively learn the long-tailed distribution of high-frequency components and quantify the cumulative direction of feature bias. During the testing stage, we apply counterfactual reasoning to adjust feature distributions based on the quantified bias. To our knowledge, this is the first time to investigate the imbalance of frequency learning within pan-sharpening from the causal inference perspective. Extensive experiments on three benchmark datasets demonstrate that our method significantly outperforms state-of-the-art approaches, showcasing its effectiveness and robustness in pan-sharpening tasks.

Pan-sharpening aims to enhance the spatial resolution of low-resolution multispectral (LRMS) images by integrating high-frequency information from a corresponding texture-rich panchromatic (PAN) image, while maintaining the spectral integrity of the LRMS image. Although text-guided multi-modal learning has made considerable strides in the natural image domain, its potential to pan-sharpening remains underexplored, primarily due to the limited availability of multi-modal remote sensing datasets. To this end, we construct an entirely new pan-sharpening framework by making efforts from three key aspects: (1) text-equipped multi-modal data collection through chain-of-reasoning, (2) large model prior-driven multi-modal information fusion, and (3) visual information interaction through prompt engineering, leveraging textual information to guide the pan-sharpening process within a multi-modal fusion framework. We initially utilize the generic large language model priors to generate descriptive captions for MS images, forming a multi-modal pan-sharpening dataset. By integrating super-resolved imagery and segmentation maps generated by segment anything, we apply Chain-of-Thought (CoT) prompting to generate spatially focused captions across diverse satellite datasets. These captions enhance visual features and provide high-level contextual information, improving semantic understanding for pan-sharpening. Building on the aforementioned multi-modal data, we tailor two text-guided information fusion modules: Textual Enhancement Block (TEB) standing on large model prior and Textual Modulated Block (TMB) utilizing text information to effectively guide and refine the pan-sharpening fusion process. Extensive experiments on multiple satellite datasets demonstrate that our proposed framework outperforms state-of-the-art methods, highlighting its effectiveness and superior performance in pan-sharpening.

Remote sensing image captioning (RSIC) presents unique challenges and plays a critical role in applications such as environmental monitoring, urban planning, and disaster management. Traditional RSIC methods often struggle to produce rich and diverse descriptions. Recently, with significant advancements in vision-language models (VLMs), efforts have emerged to integrate these models into the remote sensing domain and to introduce richly descriptive datasets specifically designed to enhance VLM training. However, most current RSIC models generally apply only fine-tuning to these datasets without developing models tailored to the unique characteristics of remote sensing imagery. This article proposes RS-MoE, the first mixture of expert (MoE)-based VLM specifically customized for remote sensing domain. Unlike traditional MoE models, the core of RS-MoE is the MoE block, which incorporates a novel instruction router and multiple lightweight large language models (LLMs) as expert models. The instruction router is designed to generate specific prompts tailored for each corresponding LLM, guiding them to focus on distinct aspects of the RSIC task. This design not only allows each expert LLM to concentrate on a specific subset of the task, thereby enhancing the specificity and accuracy of the generated captions, but also improves the scalability of the model by facilitating parallel processing of subtasks. In addition, we present a two-stage training strategy for tuning our RS-MoE model to prevent performance degradation due to sparsity. We fine-tuned our model on the RSICap dataset using our proposed training strategy. Experimental results on the RSICap dataset, along with evaluations on other traditional datasets where no additional fine-tuning was applied, demonstrate that our model achieves state-of-the-art performance in generating precise and contextually relevant captions. Notably, our RS-MoE-1B variant achieves performance comparable to 13B VLMs, demonstrating the efficiency of our model design. Moreover, our model demonstrates promising generalization capabilities by consistently achieving state-of-the-art performance on the remote sensing visual question answering (RSVQA) task.

Local and global spectral and spatial information is crucial for hyperspectral image (HSI) classification. However, modeling the global context has been challenging due to the limitations of receptive fields and quadratic complexity. Mamba’s ability to leverage long-range dependencies with linear computational complexity offers an effective approach to alleviate this issue; however, it does lead to the loss of local detail information. To address this challenge, we propose a novel local-to-global Mamba for HSI classification, termed MambaLG. MambaLG consists of a dual-branch strategy, comprising two core modules: a local and global spatial modeling module (SpaM) and a short- and long-range spectral dynamic perception module (SpeM). In the SpaM, the local and global spatial information is sequentially extracted and integrated, aiming to capture global spatial semantics while preserving the integrity of local 2-D spatial structures. In the SpeM, we utilize local spectral extraction, spectral grouping, and spectral dynamic correlation clustering (SDCC) modules, leveraging Mamba’s strengths in exploring long-range dependencies for more precise short- and long-range spectral feature modeling. Additionally, we introduce a gate attention unit into MambaLG and design a more efficient and interpretable manner for merging spatial and spectral features. Experimental results across multiple datasets (encompassing urban and agricultural scenes) indicate that MambaLG surpasses state-of-the-art algorithms regarding classification accuracy (CA) and inference speed. Comprehensive ablation studies substantiate the advantages of MambaLG in modeling local and global spatial context, enhancing short- and long-range spectral perception, and fusing spatial and spectral information. The codes will be openly available athttps://github.com/danfenghong/IEEE_TGRS_MambaLGto facilitate the reproduction of experimental results.

Unsupervised domain adaptation semantic segmentation for high spatial resolution remote sensing imagery aims to utilize a large amount of labeled data from the source domain to train a supervised model. This model is then adapted for pixel-dense prediction in the unlabeled target domain. Current methods in remote sensing unsupervised domain adaptation mainly focus on mitigating pseudo-label noise. However, they overlook the commonalities between pixel representations across domains. This oversight results in scattered semantic feature representations, thereby limiting segmentation accuracy. To address this issue, this study proposes a novel method prioritizes the extraction of learning universal features, domain-invariant representations spanning low-dimensional discriminative attributes and globally consistent spatial patterns. Specifically, a low-dimensional feature enhancement module is designed to increase feature diversity in shallow layers by employing two learnable scaling and shifting parameters. Additionally, a high-dimensional feature smoothing module uses Gaussian filtering to balance the distribution of global spatial features, enhancing feature learning in deep layers. Comprehensive evaluations on two remote sensing domain adaptation semantic segmentation datasets demonstrate that the superiority of our method, achieving state-of-the-art performance against contemporary baselines in terms of segmentation accuracy and feature alignment.

Semi-supervised learning offers an appealing solution for remote sensing (RS) image segmentation to relieve the burden of labor-intensive pixel-level labeling. However, RS images pose unique challenges, including rich multi-scale features and high inter-class similarity. To address these problems, this paper proposes a novel semi-supervised Multi-Scale Uncertainty and Cross-Teacher-Student Attention (MUCA) model for RS image semantic segmentation tasks. Specifically, MUCA constrains the consistency among feature maps at different layers of the network by introducing a multi-scale uncertainty consistency regularization. It improves the multi-scale learning capability of semi-supervised algorithms on unlabeled data. Additionally, MUCA utilizes a Cross-Teacher-Student attention mechanism to guide the student network, guiding the student network to construct more discriminative feature representations through complementary features from the teacher network. This design effectively integrates weak and strong augmentations (WA and SA) to further boost segmentation performance. To verify the effectiveness of our model, we conduct extensive experiments on ISPRS-Potsdam and LoveDA datasets. The experimental results show the superiority of our method over state-of-the-art semi-supervised methods. Notably, our model excels in distinguishing highly similar objects, showcasing its potential for advancing semi-supervised RS image segmentation tasks.

Integrating low-resolution hyperspectral images (LR-HSIs) with corresponding high-resolution multispectral images (HR-MSIs) for the reconstruction of HR-HSI using deep learning techniques represents a critical area of research. Although convolutional neural networks (CNNs) are widely used for HR-HSI reconstruction, their small receptive fields hinder effective global feature extraction, which limits their potential. Fusion methods that rely on traditional attention mechanisms also lack feature interaction, failing to integrate and harmonize feature information from hyperspectral image (HSI) and MSI effectively. To address these issues, this article develops a novel progressive fusion of multiscale axial attention network (PMA2-Net), which combines multiscale convolutions with axial attention (AA) and employs a progressive interaction approach to reconstruct high-resolution images. Specifically, PMA2-Net extracts the spatial and spectral information from HSI through spatial feature extraction (Spatial-FE) and spectral feature extraction (Spectral-FE). Concurrently, a feature injection module (FIM) is introduced, employing multiscale convolution to capture local features and integrating AA to enhance global feature association. Moreover, a progressive fusion module (PFM) is employed to enhance multidimensional feature collaboration and hierarchical integration. Extensive studies conducted using five significant HSI datasets verify the effectiveness of PMA2-Net, demonstrating its superior performance compared to current state-of-the-art (SOTA) fusion techniques.

Deep learning models exhibit promising potential in multimodal remote sensing image semantic segmentation (MRSISS). However, the constrained access to labeled samples for training deep learning networks significantly influences the performance of these models. To address that, self-supervised learning (SSL) methods have garnered significant interest in the remote sensing community. Accordingly, this article proposes a novel multimodal contrastive learning framework based on tuple perturbation, which includes the pretraining and fine-tuning stages. First, a tuple perturbation-based multimodal contrastive learning network (TMCNet) is designed to better explore shared and different feature representations across modalities during the pretraining stage and the tuple perturbation module is introduced to improve the network’s ability to extract multimodal features by generating more complex negative samples. In the fine-tuning stage, we develop a simple and effective multimodal semantic segmentation network (MSSNet), which can reduce noise by using complementary information from various modalities to integrate multimodal features more effectively, resulting in better semantic segmentation performance. Extensive experiments have been carried out on two published multimodal image datasets including optical and synthetic aperture radar (SAR) pairs, and the results show that the proposed framework can obtain more superior performance of semantic segmentation than the current state-of-the-art methods in cases of limited labeled samples. The source code is available athttps://github.com/yeyuanxin110/TMCNet-MSSNet.

Self-supervised hyperspectral image (HSI) clustering remains a fundamental yet challenging task due to the absence of labeled data and the inherent complexity of spatial-spectral interactions. While recent advancements have explored innovative approaches, existing methods face critical limitations in clustering accuracy, feature discriminability, computational efficiency, and robustness to noise, hindering their practical deployment. In this paper, a self-supervised efficient low-pass contrastive graph clustering (SLCGC) is introduced for HSIs. Our approach begins with homogeneous region generation, which aggregates pixels into spectrally consistent regions to preserve local spatial-spectral coherence while drastically reducing graph complexity. We then construct a structural graph using an adjacency matrix A and introduce a low-pass graph denoising mechanism to suppress high-frequency noise in the graph topology, ensuring stable feature propagation. A dual-branch graph contrastive learning module is developed, where Gaussian noise perturbations generate augmented views through two multilayer perceptrons (MLPs), and a cross-view contrastive loss enforces structural consistency between views to learn noise-invariant representations. Finally, latent embeddings optimized by this process are clustered via K-means. Extensive experiments and repeated comparative analysis have verified that our SLCGC contains high clustering accuracy, low computational complexity, and strong robustness. The code source will be available athttps://github.com/DY-HYX.

Multihead self-attention and cross-attention mechanisms often suffer from computational inefficiencies, limited scalability, and suboptimal contextual understanding, particularly in hyperspectral image (HSI) classification. These mechanisms struggle to effectively capture long-range dependencies while maintaining computational feasibility due to the quadratic complexity of self-attention. To address these challenges, this work proposes PolicyMamba, a spectral-spatial mamba model enhanced with a localized policy attention mechanism. This mechanism reduces computational overhead by restricting attention to nonoverlapping localized regions and enforcing sparsity constraints, ensuring that only the most informative interactions are retained. A hierarchical aggregation strategy further integrates patch-wise attention outputs, preserving spectral-spatial correlations across scales. In addition, a sliding window patch process enhances local feature continuity while mitigating information loss. The PolicyMamba framework integrates spectral-spatial token generation, token enhancement, localized attention, and state transition modules, significantly improving HSI feature representation. Extensive experiments demonstrate that PolicyMamba achieves superior classification accuracy, outperforming conventional and state-of-the-art methods in land cover classification (LCC) by efficiently modeling intricate dependencies in HSI data.

Accurately distinguishing between background and anomalous objects within hyperspectral images poses a significant challenge. The primary obstacle lies in the inadequate modeling of prior knowledge, leading to a performance bottleneck in hyperspectral anomaly detection (HAD). In response to this challenge, we put forth a groundbreaking coupling paradigm that combines model-driven low-rank representation (LRR) methods with data-driven deep learning techniques by learning disentangled priors (LDP). LDP seeks to capture complete priors for effectively modeling the background, thereby extracting anomalies from hyperspectral images more accurately. LDP follows a model-driven deep unfolding architecture, where the prior knowledge is separated into the explicit low-rank prior formulated by expert knowledge and implicit learnable priors by means of deep networks. The internal relationships between explicit and implicit priors within LDP are elegantly modeled through a skip residual connection. Furthermore, we provide a mathematical proof of the convergence of our proposed model. Our experiments, conducted on multiple widely recognized datasets, demonstrate that LDP surpasses most of the current advanced HAD techniques, exceling in both detection performance and generalization capability.

In the expansive domain of computer vision, a myr-iad of pretrained models are at our disposal. However, most of these models are designed for natural RGB images and prove inadequate for spectral remote sensing (RS) images. Spectral RS images have two main traits: (1) multiple bands capturing diverse feature information, (2) spatial alignment and consistent spectral sequencing within the spatial-spectral dimension. In this paper, we introduce Spatial-SpectralMAE (S2MAE), a specialized pretrained architecture for spectral RS imagery. S2MAE employs a 3D transformer for masked autoencoder modeling, inte-grating learnable spectral-spatial embeddings with a 90% masking ratio. The model efficiently captures local spec-tral consistency and spatial invariance using compact cube tokens, demonstrating versatility to diverse input characteristics. This adaptability facilitates progressive pretraining on extensive spectral datasets. The effectiveness of S2MAE is validated through continuous pretraining on two sizable datasets, totaling over a million training images. The pretrained model is subsequently applied to three dis-tinct downstream tasks, with in-depth ablation studies conducted to emphasize its efficacy.

Pan-sharpening is a super-resolution problem that essentially relies on spectra fusion of panchromatic (PAN) images and low-resolution multi-spectral (LRMS) images. The previous methods have validated the effectiveness of information fusion in the Fourier space of the whole image. However, they haven't fully explored the Fourier relationships at different hierarchies between PAN and LRMS images. To this end, we propose a Hierarchical Frequency Integration Network (HFIN) to facilitate hierarchical Fourier information integration for pan-sharpening. Specifically, our network consists of two designs: information stratification and information integration. For information stratification, we hierarchically decompose PAN and LRMS information into spatial, global Fourier and local Fourier information, and fuse them independently. For information integration, the above hierarchical fused information is processed to further enhance their relationships and undergo comprehensive integration. Our method extend a new space for exploring the relationships of PAN and LRMS images, enhancing the integration of spatial-frequency information. Extensive experiments robustly validate the effectiveness of the proposed network, showcasing its superior performance compared to other state-of-the-art methods and generalization in real-world scenes and other fusion tasks as a general image fusion framework. Code is available at https://github.com/JosephTiTan/HFIN.

The annotation of polarimetric synthetic aperture radar (PolSAR) images is a labor-intensive and time-consuming process. Therefore, classifying PolSAR images with limited labels is a challenging task in remote sensing domain. In recent years, self-supervised learning approaches have proven effective in PolSAR image classification with sparse labels. However, we observe a lack of research on generative self-supervised learning in the studied task. Motivated by this, we propose a dual-branch classification model based on generative self-supervised learning in this paper. The first branch is a superpixel-branch, which learns superpixel-level polarimetric representations using a generative self-supervised graph masked autoencoder. To acquire finer classification results, a convolutional neural networks-based pixel-branch is further incorporated to learn pixel-level features. Classification with fused dual-branch features is finally performed to obtain the predictions. Experimental results on the benchmark Flevoland dataset demonstrate that our approach yields promising classification results.

Due to its capacity to recognize detailed spectral differences, hyperspectral data have been extensively used for precise Land Use Land Cover (LULC) mapping. However, recent multi-modal methods have shown their superior classification performance over the algorithms that use single data sets. On the other hand, Convolutional Neural Networks (CNNs) are models extensively utilized for the hierarchical extraction of features. Vision transformers (ViTs), through a self-attention mechanism, have recently achieved superior modeling of global contextual information compared to CNNs. However, to harness their image classification strength, ViTs require substantial training datasets. In cases where the available training data is limited, current advanced multi-layer perceptrons (MLPs) can provide viable alternatives to both deep CNNs and ViTs. In this paper, we developed the SGU-MLP, a deep learning algorithm that effectively combines MLPs and spatial gating units (SGUs) for precise Land Use Land Cover (LULC) mapping using multi-modal data from multi-spectral, LiDAR, and hyperspectral data. Results illustrated the superiority of the developed SGU-MLP classification algorithm over several CNN and CNN-ViT-based models, including HybridSN, ResNet, iFormer, EfficientFormer, and CoAtNet. The SGU-MLP classification model consistently outperformed the benchmark CNN and CNN-ViT-based algorithms. The code will be made publicly available at https: //github.com/aj1365/SGUMLP.

Convolutional Neural Networks (CNNs) are employed extensively in remote sensing due to their capacity to capture intricate features from a broad range of object patterns, irrespective of object size, shape or color. These networks excel at extracting high-frequency spectral information such as angles, edges and outlines. The classification boundary zone, however, becomes hazy for CNNs because they learn characteristics by means of a fixed shape kernel concentrated on the central pixel, and can perform poorly in image classification at class boundaries. Additionally, CNNs are not designed to capture global relations. Thus, in this letter, we propose an Attention Graph Convolutional Network (Attention-GCN) as a solution to the aforementioned shortcomings. The developed model illustrated a high level of superiority over several CNN and ViT-based models. For example, in the Augsburg data benchmark, the developed algorithm exhibited an average accuracy of 61.11%, substantially outperforming other models such as HybridSN, iFormer, Efficient Former, GCN, CoAtNet, 2D-CNN, 3D-CNN, and ResNet by approximately 9, 13, 14, 15, 18, 24, 25 and 29 percentage points, respectively. The code will be made publicly available at https://github.com/aj1365/AGCN.

Incorrect labeling (i.e., noisy label learning) in HSI classification has attracted so much attention in recent years, which holds the assumption that the given pixels of an HSI may be incorrectly labeled and only one candidate label is required to provide for a typical pixel. However, instead of offering only one candidate label that may be incorrect, partial label learning often provides a candidate label set that contains the ground-truth label for each pixel in an HSI, which is also an essential problem of great practical value and has recently started to attract attention. This paper proposes a novel framework for partial label learning in HSI classification, namely unlabeled data guided partial label learning (UPLL). The proposed framework is an iterative process that can fully exploit the benefits of unlabeled data. Specifically, during each iteration, we conduct the semi-supervised label propagation; the resulting labeling confidence matrices of the original training samples and the unlabeled testing samples are further enhanced by exploiting the spatial information. Then, we select qualified original training samples and unlabeled testing samples with high confident predictions to disambiguate and expand the original training set, leading to a more robust representation of training data. Such phases are repeated until convergence. The comprehensive experiments show the superiority of the proposed UPLL method over the existing state-of-the-art methods. Especially, the classification accuracy improves more than 5% with very few training samples than the second best comparing method.

The foundation model has recently garnered significant attention due to its potential to revolutionize the field of visual representation learning in a self-supervised manner. While most foundation models are tailored to effectively process RGB images for various visual tasks, there is a noticeable gap in research focused on spectral data, which offers valuable information for scene understanding, especially in remote sensing (RS) applications. To fill this gap, we created for the first time a universal RS foundation model, named SpectralGPT, which is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT). Compared to existing foundation models, SpectralGPT 1) accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS Big Data; 2) leverages 3D token generation for spatial-spectral coupling; 3) captures spectrally sequential patterns via multi-target reconstruction; and 4) trains on one million spectral RS images, yielding models with over 600 million parameters. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS Big Data applications within the field of geoscience across four downstream tasks: single/multi-label scene classification, semantic segmentation, and change detection.

Multilabel remote sensing scene classification (MLRSSC) has garnered growing attention in recent years, owing to its more comprehensive description of land covers compared to its single-label counterpart. However, challenges arise inevitably. First, the relations among multiple scene labels are sophisticated. How to excavate the interclass dependencies is, therefore, a key challenge for the MLRSSC task. Second, extracting discriminative semantic features is essential, yet challenging for scene prediction of remote sensing images. Another issue is that the multilabel dataset usually shows twofold sample imbalances, that is, class imbalance and positive-negative imbalance, which have not been explored in MLRSSC tasks so far. To overcome the above hurdles, we put forward a cross-attention-driven adaptive graph relational network for the MLRSSC task. Different from the chain-like long short-term memory (LSTM) or static label co-occurrence matrices, we propose to use image-specific relational graphs to dynamically model the interclass dependencies. We innovatively devise a cross-attention-driven representation learning approach, which uses learnable label embeddings to query the class-wise semantic features, explicitly establishing the feature-label connections. Moreover, we design a balanced focal loss (BFL) function, where the loss contributions of positive and negative samples are rebalanced based on the respective imbalance degrees of diverse classes. Extensive experiments were performed on UCM, AID, and DFC15 multilabel datasets. Experimental results demonstrated that our proposed method achieves state-of-the-art performance in the studied task.

In recent years, multimodal remote sensing data classification (MMRSC) has evoked growing attention due to its more comprehensive and accurate delineation of Earth’s surface compared to its single-modal counterpart. However, it remains challenging to capture and integrate local and global features from single-modal data. Moreover, how to fully excavate and exploit the interactions between different modalities is still an intricate issue. To this end, we propose a novel dual-branch transformer-based framework named deep symmetric fusion transformer (DSymFuser). Within the framework, each branch contains a stack of local-global mixture (LGM) blocks, to extract hierarchical and discriminative single-modal features. In each LGM block, a local-global feature mixer with learnable weights is specifically devised to adaptively aggregate the local and global features extracted with a convolutional neural network (CNN)–transformer network. Furthermore, we innovatively design a symmetric fusion transformer (SFT) that trails behind each LGM block. The elaborately designed SFT symmetrically facilitates cross-modal correlation excavation, comprehensively exploiting the complementary cues underlying heterogeneous modalities. The hierarchical construction of the LGM and SFT blocks enables feature extraction and fusion in a multilevel manner, further promoting the completeness and descriptiveness of the learned features. We conducted extensive ablation studies and comparative experiments on three benchmark datasets, and the experimental results validated the effectiveness and superiority of the proposed method. The source code of the proposed method will be available publicly athttps://github.com/HaixiaBi1982/DSymFuser.

With the increasing availability and openness of remote sensing (RS) data collected from diverse sensors, there has been a growing interest in multimodal RS data classification. Nowadays, in the area of deep learning, there is a paradigm shift with the rise of foundation models, which are trained on large-scale datasets and are adaptable to a wide range of downstream tasks. In this study, the potential and effectiveness of foundation models for multimodal RS data classification is investigated. The training datasets of foundation models and multimodal RS datasets are quite different, and therefore, it is difficult to use a pretrained foundation model for multimodal RS data classification directly. To mitigate this difficulty, this article proposes a foundation model adaptation (FMA) framework for multimodal RS data classification without fine-tuning the parameters. Specifically, two learnable modules, i.e., cross-spatial interaction module and cross-channel interaction module, are proposed to add to the foundation model for extracting multimodal-specific representations. The cross-spatial and cross-channel interaction modules extract the characteristics of unimodal features along the spatial dimension and channel dimension, respectively. To effectively tackle the disparities among various RS modalities, an alignment approach (FMA2) is further explored based on the FMA. The FMA2 describes dependencies between different modalities by establishing a coupling score function, which can further enhance classification performance. To demonstrate the effectiveness and superiority of the FMA framework, comprehensive experiments are conducted on three multimodal RS datasets, showing improvement over the advanced multimodal RS data classification image methods.

Current hyperspectral anomaly detection (HAD) benchmark datasets suffer from low resolution, simple background, and small size of the anomalies. These factors also limit the performance of the well-known low-rank representation (LRR) models in terms of robustness on the separation of background and target features and the reliance on manual parameter selection. To this end, we build a new HAD benchmark dataset for improving the robustness in complex scenarios, AIR-HAD for short, and propose an interpretable network with deep unfolding a binary subspace learning, named LRR-Net+, which is capable of spectrally decoupling the background structure and object properties in a more generalized fashion and eliminating the bias introduced by vital interference targets simultaneously. In addition, LRR-Net+ integrates the solution process of the alternating direction method of multipliers (ADMM) optimizer with the deep network, guiding its search process and imparting a level of interpretability to parameter optimization. Additionally, the integration of physical models with DL techniques eliminates the need for manual parameter tuning. The manually tuned parameters are seamlessly transformed into trainable parameters for deep neural networks, facilitating a more efficient and automated optimization process. Extensive experiments conducted on the AIR-HAD dataset show the superiority of our LRR-Net+ in terms of detection performance and generalization ability, compared to top-performing competitors. Furthermore, our AIR-HAD benchmark datasets will be made available freely and openly athttps://github.com/danfenghong/IEEE_TGRS_LRR-Net.

Infrared small target detection (ISTD) is a challenging task in the computer vision field due to factors such as target scale variations and strong clutter. The existing infrared patch tensor (IPT) models achieve good detection performance but still have several limitations, such as inaccurate background modeling results and poor robustness against noise. To alleviate these issues, in this article, we propose a new IPT model (dubbed as IPT-TCTV) by fully exploiting prior background knowledge. We construct an improved spatial-temporal (STT) model by sliding a 3-D window, which could better preserve the spatial correlation and temporal continuity of multiframe infrared images in the constructed tensor. Specifically, a joint low-rank and local smoothness regularization, i.e., tensor correlated total variation (TCTV), is utilized to characterize the background since the background exhibits not only the low-rank property but also the local smoothness property, without introducing additional trade-off parameters. Furthermore, considering the effect of edge structures, the${l} _{2,1}$norm is adopted as a noise constraint to eliminate strong residuals, which can help to extract real targets from the background with more precision. Finally, we design an efficient alternating direction method of multipliers (ADMMs) approach to solve the proposed model. Experimental results on some benchmark datasets illustrate that our IPT-TCTV model can achieve better detection performance than other state-of-the-art (SOTA) methods in various real scenes. The source code is released athttps://github.com/AuroraPei/IPT-TCTV.

Multimodal image fusion has recently garnered increasing interest in the field of remote sensing. By leveraging the complementary information in different modalities, the fused results may be more favorable in characterizing objects of interest, thereby increasing the chance of a more comprehensive and accurate perception of the scene. Unfortunately, most existing fusion methods tend to extract modality-specific features independently without considering intermodal alignment and complementarity, leading to a suboptimal fusion process. To address this issue, we propose a novel interactive generative adversarial network (IG-GAN), for the task of multimodal image fusion. IG-GAN comprises guided dual streams tailored for enhanced learning of details and content, as well as cross-modal consistency. Specifically, a details-guided interactive running-in module (GIR1) and a content-guided interactive running-in module (GIR2) are developed, with the stronger modality serving as guidance for detail richness or content integrity, and the weaker one assisting. To fully integrate multigranularity features from dual-modality, a hierarchical fusion and reconstruction branch is established. Specifically, a shallow interactive fusion (SIF) module followed by a multilevel interactive fusion (MIF) module is designed to aggregate multilevel local and long-range features. Concerning feature decoding and fused image generation, a high-level interactive fusion and reconstruction module (HRM) is further developed. In addition, to empower the fusion network to generate fused images with complete content, sharp edges, and high fidelity without supervision, a loss function facilitating the mutual game between the generator and two discriminators is also formulated. Comparative experiments with 14 state-of-the-art methods are conducted on three datasets. Qualitative and quantitative results indicate that IG-GAN exhibits obvious superiority in terms of both visual effect and quantitative metrics. Moreover, experiments on two RGB-IR object detection datasets are also conducted, which demonstrate that IG-GAN can enhance the accuracy of object detection by integrating complementary information from different modalities. The code will be available athttps://github.com/flower6top.

Semantic segmentation of high spatial resolution remote sensing images presents unique challenges due to the imbalanced foreground-background distribution and large intra-class variance. This study proposes a novel semantic segmentation algorithm based on non-learnable prototypes, named RSProtoSeg. This approach optimizes the spatial relationship between foreground-background prototypes and intra-class prototypes. Specifically, we propose a foreground-background distance optimization loss function to enhance sparsity between these phototypes, effectively mitigating foreground-background distribution imbalances. Moreover, we introduce an online discrete clustering module that represents each class with a set of prototypes and adds an adaptive regular term penalty to promote sparse structure and reduce the variance issue. Evaluation on three remote sensing datasets (iSAID, ISPRS Potsdam, and Vaihingen) demonstrates significant accuracy improvements, aligning our approach with state-of-the-art methods. Our non-learnable prototype-based approach offers a promising solution for semantic segmentation in high spatial resolution remote sensing images.

Cloud contamination greatly limits the potential utilization of optical images for geoscience applications. An effective alternative is to extract data from synthetic aperture radar (SAR) images to remove clouds due to the strong penetration ability of microwaves. In this article, we propose a novel unified spatial–spectral residual network that utilizes SAR images as auxiliary data to remove clouds from optical images. The method can better establish the relationship between SAR and optical images and be divided into two modules: feature extraction and fusion module and reconstruction module. In the feature extraction and fusion module, a gated convolutional layer is introduced to discriminate cloud pixels from clean pixels, which makes up for the lack of distinguishing ability of vanilla convolutional layers and avoids the error of cloud areas in feature extraction. In the reconstruction module, spatial and channel attention mechanisms are introduced to obtain global spatial and spectral information. The network is tested on three datasets with different spatial resolutions and compositions of land covers to verify the effectiveness and applicability of the method. The results show that the method outperforms other mainstream algorithms that simultaneously use SAR images as auxiliary data with a gain of about 2.3 dB in terms of peak signal-to-noise ratio PSNR on the SEN12MS-CR dataset.

In hyperspectral images (HSIs), both local and non-local features play crucial roles in classification tasks. Vision Transformer (VIT) can extract non-local features through attention mechanisms, while Convolutional Neural Networks (CNN) excel at handling local components. However, in traditional dual-branch models based on VIT and CNN, there is a lack of interaction during feature processing, leading to potential compatibility issues when merging the two types of features. In this article, we propose HyperSINet, a Synergetic Interaction Network that combines VIT and CNN to establish interaction between the two branches, enabling mutual compensation between local and non-local features during the training process and ultimately enhancing the performance of classification tasks. Specifically, we devise a pair of interactors, namely Conv2Trans and Trans2Conv, which serve as intermediaries between the two branches, enabling the VIT branch to refine its local details, while allowing the CNN branch to process larger receptive field non-local features. Typical feature Maps are implemented to visualize the function of the interactors. Furthermore, within the VIT branch, a VIT Encoder with the local mask is developed to strike a balance between emphasizing non-local features and preserving local details, while a lightweight CNN block is designed to process spectral and spatial features in the CNN branch. Extensive experiments conducted on four real-world datasets demonstrate that, under a reasonable count of parameters, HyperSINet surpasses several current state-of-the-art methods.

Pan-sharpening is essentially a panchromatic (PAN)-guided super-resolution process, primarily focused on enhancing multi-spectral image quality. This methodology intricately incorporates the high-frequency derived from texture-rich PAN images into the lower-resolution multi-spectral (LRMS) counterparts. However, current spatial domain techniques frequently face challenges in accurately restoring texture details, while frequency domain methods lack efficient interaction with spatial domains, thus restricting the overall model performance. In response to these challenges, we introduce a novel High-frequency Wavelet Network that capitalizes on the spatial-frequency interaction and frequency division capabilities inherent in wavelet transform. In particular, our approach consists of two fundamental modules: the Wavelet-Inspired Fusion Block and the High-Frequency Enhancement Block. The former is inspired by wavelet lifting schemes, enabling the fusion of frequencies and facilitating information exchange across various subbands. The latter harnesses wavelet’s frequency division attributes to enhance high-frequency information learning. Comprehensive experiments over multiple satellite datasets demonstrate that our approach outperforms state-of-the-art techniques in both quantitative and qualitative assessments. Moreover, our model showcases exceptional generalization capabilities in real-world scenarios. Code is available at https://github.com/alexhe101/WINet.

It is generally known that pan-sharpening is fundamentally a PAN-guided multispectral (MS) image super-resolution problem that involves learning the nonlinear mapping from low-resolution (LR) to high-resolution (HR) MS images. Since an infinite number of HR-MS images can be downsampled to produce the same corresponding LR-MS image, learning the mapping from LR-MS to HR-MS image is typically ill-posed and the space of the possible pan-sharpening functions can be extremely large, making it difficult to estimate the optimal mapping solution. To address the above issue, we propose a closed-loop scheme that learns the two opposite mapping including the pan-sharpening and its corresponding degradation process simultaneously to regularize the solution space in a single pipeline. More specifically, an invertible neural network (INN) is introduced to perform a bidirectional closed-loop: the forward operation for LR-MS pan-sharpening and the backward operation for learning the corresponding HR-MS image degradation process. In addition, given the vital importance of high-frequency textures for the Pan-sharpened MS images, we further strengthen the INN by designing a specified multiscale high-frequency texture extraction module. Extensive experimental results demonstrate that the proposed algorithm performs favorably against state-of-the-art methods qualitatively and quantitatively with fewer parameters. Ablation studies also verify the effectiveness of the closed-loop mechanism in pan-sharpening. The source code is made publicly available at https://github.com/manman1995/pan-sharpening-Team-zhouman/.

Due to the difficulty of obtaining manual annotations for polarimetric synthetic aperture radar (PolSAR) images, the problem of analyzing these images without or with few labels has become a current challenge. Considering the scarcity of labels, this paper proposes a noise-tolerant deep clustering-based PolSAR image classification that mainly uses autoencoders to learn discriminative features. In addition, in order to improve the performance and noise resistance of this method, we adopt Markov Random Field (MRF) to enhance the smoothness of class labels. We conducted experiments on a real benchmark PolSAR image, and the results show that our method achieves state-of-the-art PolSAR image classification results without any manual annotations.

Abundant spectral signatures and spatial characteristics embedded in hyperspectral (HS) images enable the fine identification of land covers, attracting plenty of studies on feature extraction and feature utilization. Nevertheless, the high representative spectral and spatial features in the HS cube are unevenly distributed, which is failed to consider by many current methods. To conquer this shortcoming, we rethink the feature extraction of HS images from an anisotropic perspective and propose a novel model called grid network (GNet) for HS image classification. Beyond representing spectral-spatial features in three classic paradigms (simultaneously, hierarchically, and separately), GNet is capable of learning them in two new processes: multi-stage and multi-path. In this way, spectral and spatial features can be fully and balanced explored. More significantly, to make full use of low- and high-level features and avoid the existing semantic gap, we devise a spectral-spatial cross-level feature fusion module to model the relation between them. Extensive experiments, implemented on three HS datasets, demonstrate that the proposed GNet enables to acquire promising classification performance compared to state-of-the-art methods. The codes of this work will be available at https://github.com/zhonghaochen/GNet_Master for the sake of reproducibility.

Insufficient prior knowledge of a captured hyperspectral image (HSI) scene may lead the experts or the automatic labeling systems to offer incorrect labels or ambiguous labels (i.e., assigning each training sample to a group of candidate labels, among which only one of them is valid; this is also known as partial label learning) during the labeling process. Accordingly, how to learn from such data with ambiguous labels is a problem of great practical importance. In this letter, we propose a novel superpixelwise low-rank approximation (LRA)-based partial label learning method, namely SLAP, which is the first to take into account partial label learning in HSI classification. SLAP is mainly composed of two phases: disambiguating the training labels and acquiring the predictive model. Specifically, in the first phase, we propose a superpixelwise LRA-based model, preparing the affinity graph for the subsequent label propagation process while extracting the discriminative representation to enhance the following classification task of the second phase. Then to disambiguate the training labels, label propagation propagates the labeling information via the affinity graph of training pixels. In the second phase, we take advantage of the resulting disambiguated training labels and the discriminative representations to enhance the classification performance. The extensive experiments validate the advantage of the proposed SLAP method over state-of-the-art methods.

Hyperspectral image change detection (HSI-CD) has recently become a research hotspot. Current methods rely heavily on a huge amount of training samples to perform the change detection tasks. While acquiring data from the same region of bi-temporal HSIs is extraordinarily time-consuming and laborious. Therefore, this letter proposes an unsupervised method based on three dimensional (3D) depthwise separable convolutional autoencoders (DSConvAE). First, the dual-branch symmetrical 3D DSConvAE is pre-trained with limited samples to obtain the optimal weights, which facilitates extracting discriminative spatial and spectral features subsequently. Second, we adopt the temporal-specific feature concatenation strategy to acquire comprehensive characteristics from bi-temporal HSIs. Third, the general autoencoders are employed at the end of the model to further explore the high-level and abstract feature vectors. Finally, we compare the mean square loss calculated from the spatial-spectral branches and apply threshold judgement to generate the ultimate detection maps. Experimental results on three public HSI datasets demonstrate that the proposed framework outperforms other comparative methods by significant improvements.

Recently, low-rank representation (LRR) methods have been widely applied for hyperspectral anomaly detection, due to their potentials in separating the backgrounds and anomalies. However, existing LRR models generally convert 3-D hyperspectral images (HSIs) into 2-D matrices, inevitably leading to the destruction of intrinsic 3-D structure properties in HSIs. To this end, we propose a novel tensor low-rank and sparse representation (TLRSR) method for hyperspectral anomaly detection. A 3-D TLR model is expanded to separate the LR background part represented by a tensorial background dictionary and corresponding coefficients. This representation characterizes the multiple subspace property of the complex LR background. Based on the weighted tensor nuclear norm and the$L_{F,1}$sparse norm, a dictionary is designed to make its atoms more relevant to the background. Moreover, a principal component analysis (PCA) method can be assigned as one preprocessing step to exact a subset of HSI bands, retaining enough the HSI object information and reducing computational time of the postprocessing tensorial operations. The proposed model is efficiently solved by the well-designed alternating direction method of multipliers (ADMMs). A comparison with the existing algorithms via experiments establishes the competitiveness of the proposed method with the state-of-the-art competitors in the hyperspectral anomaly detection task.

Deep learning (DL) has recently risen to prominence in hyperspectral target detection (HTD). Nevertheless, how to tackle the extreme training sample imbalance together with achieving target highlighting and background suppression is challenging. Additionally, due to the spectral redundancy of hyperspectral imagery (HSI), it is a new course for HTD through band selection (BS) to retain crucial bands thereupon improving the subsequent detection performance. Accordingly, we propose a DL-based BS-HTD (DLBSTD) algorithm, incorporating DL-based BS with DL-based HTD for the first time. Most significantly, a multi-depth and multi-branch network (MDBN) for HTD based on a novel BS method is proposed. First of all, the BS method including an alternating local-global reconstruction network (ALGRN) and a correlation measurement strategy provides representative bands containing key target information for MDBN. For the training sample imbalance of MDBN, we develop a BS-based method to select multifarious representative background training samples and propose a target band random substitution (TBRS) strategy to augment an ample target training set. Lastly, the MDBN composed of a multi-depth feature extraction (MDFE) module, three fusion strategies, and the parallel local convolution and gated recurrent unit (Conv-GRU) fully taps the spectral feature relationships to highlight targets and suppress backgrounds. Compared with nine competitive HTD algorithms, we carry out plentiful experiments on four classical datasets exhibiting that the proposed DLBSTD has strong generalization and salient detection performance of target highlighting and background suppression.

Although convolutional neural networks (CNNs) have shown superior performance to traditional machine learning algorithms for hyperspectral image classification tasks, the ability of traditional CNNs to model remote dependencies in the spatial orientation of HSIs is still limited, and they always extract similar low-level features, leading to feature redundancy. To cope with this limitation, this paper proposes a novel multi-order statistical representation-guided graph convolution and continuous context threshold-aware network for the classification of hyperspectral images with limited training samples. Initially, the spectral spatial information is separately modeled using first-order features and second-order pooling operators. Secondly, we propose graph-structuring the patch’s features. By employing a random walk transition probability matrix, graph-structured convolution can mine more discriminative direction features. In addition, we design a continuous context threshold-aware network to model multidimensional spatial relationships, thereby enhancing the representation of graph features. Specifically, the cross-attention mechanism is used to calculate the attention weights in the vertical and horizontal directions, and the features are divided into two levels—important and secondary—by solving the cosine distance between feature vectors, and the former is retained and the latter is punished. Extensive experiments on multiple HSIs datasets demonstrated that the proposed method delivers competitive performance. The code will be available at: https://github.com/vivitsai/GSC-CCTA.

A Transformer has received a lot of attention in computer vision. Because of global self-attention, the computational complexity of Transformer is quadratic with the number of tokens, leading to limitations for practical applications. Hence, the computational complexity issue can be efficiently resolved by computing the self-attention in groups of smaller fixed-size windows. In this article, we propose a novel pyramid Shuffle-and-Reshuffle Transformer (PSRT) for the task of multispectral and hyperspectral image fusion (MHIF). Considering the strong correlation among different patches in remote sensing images and complementary information among patches with high similarity, we design Shuffle-and-Reshuffle (SaR) modules to consider the information interaction among global patches in an efficient manner. Besides, using pyramid structures based on window self-attention, the detail extraction is supported. Extensive experiments on four widely used benchmark datasets demonstrate the superiority of the proposed PSRT with a few parameters compared with several state-of-the-art approaches. The related code is available athttps://github.com/Deng-shangqi/PSRT.

Hyperspectral target detection (HTD) is a pixel-wise detection method based on limited prior targets and spectral differences, which has been widely studied and applied in many fields. Recently, deep learning (DL) plays an important role in hyperspectral imagery (HSI) processing. However, for HTD, the severe lack of class-balanced training sets is an enormous challenge. Meanwhile, it is difficult to suppress backgrounds while highlighting targets through the deep network. To address these issues, we propose a spectral aggregation and separation network (SASN) with a target band random mask (TBRM) for HTD in this paper. For the training sets of SASN, a multifarious representative background selection strategy (MRBS) is first proposed to obtain a multifarious and representative background training set. Next, aiming at the notorious class imbalance, a data augmentation (DA) method, TBRM, is proposed to generate adequate target training set by repeating randomly zero-masking the spectral bands of a prior target. Subsequently, in the training of SASN, residual connection and squeeze-and-excitation (SE) channel attention mechanism are applied to fully extract high discriminative features and nonlinear ones in the spectra. Besides, to better separate the targets and backgrounds, a triplet-soft loss function is presented, which makes the training in the direction of spectral separation of background samples from both the prior target and target samples. During testing, the trained SASN distinguishes the spectral similarities and differences simultaneously for highlighting targets and suppressing backgrounds. Moreover, extensive experimental results validate that the proposed method has superior detection performances, background suppression capacity, and separability compared with ten cutting-edge HTD algorithms on six benchmark HSI datasets.

The goal of pan-sharpening is to produce a high-spatial-resolution multi-spectral (HRMS) image from a low-spatial-resolution multi-spectral (LRMS) counterpart by super-resolving the LRMS one under the guidance of a texture-rich panchromatic (PAN) image. Existing research has concentrated on using spatial information to generate HRMS images, but has neglected to investigate the frequency domain, which severely restricts the performance improvement. In this work, we propose a novel pan-sharpening approach, named Multi-Scale Dual-Domain Guidance Network (MSDDN) by fully exploring and exploiting the distinguished information in both the spatial and frequency domains. Specifically, the network is inborn with multi-scale U-shape manner and composed by two core parts: a spatial guidance sub-network for fusing local spatial information and a frequency guidance sub-network for fusing global frequency domain information and encouraging dual-domain complementary learning. In this way, the model can capture multi-scale dual-domain information to help it generate high-quality pan-sharpening results. Employing the proposed model on different datasets, the quantitative and qualitative results demonstrate that our method performs appreciatively against other state-of-the-art approaches and comprises a strong generalization ability for real-world scenes. The source code is available at https://github.com/alexhe101/MSDDN.

Enormous efforts have been recently made to super-resolve hyperspectral (HS) images with the aid of high spatial resolution multispectral (MS) images. Most prior works usually perform the fusion task by means of multifarious pixel-level priors. Yet the intrinsic effects of a large distribution gap between HS-MS data due to differences in the spatial and spectral resolution are less investigated. The gap might be caused by unknown sensor-specific properties or highly-mixed spectral information within one pixel (due to low spatial resolution). To this end, we propose a subpixel-level HS super-resolution framework by devising a novel decoupled-and-coupled network, called DC-Net, to progressively fuse HS-MS information from the pixel- to subpixel-level, from the image- to feature-level. As the name suggests, DC-Net first decouples the input into common (or cross-sensor) and sensor-specific components to eliminate the gap between HS-MS images before further fusion, and then thoroughly blends them by a model-guided coupled spectral unmixing (CSU) net. More significantly, we append a self-supervised learning module behind the CSU net by guaranteeing material consistency to enhance the detailed appearance of the restored HS product. Extensive experimental results show the superiority of our method both visually and quantitatively and achieve a significant improvement in comparison with the state-of-the-art.

Considerable endeavors have been expended towards enhancing the representation performance for Hyperspectral Anomaly Detection (HAD) through physical model-based methods and recent deep learning-based approaches. Of these methods, the Low-Rank Representation (LRR) model is widely adopted for its formidable separation capabilities for background and target features, however, its practical applications are limited due to the reliance on manual parameter selection and subpar generalization performance. To this end, this paper presents a new HAD baseline network, referred to as LRR-Net, which synergizes the LRR model with deep learning techniques. LRR-Net leverages the alternating direction method of multipliers (ADMM) optimizer to solve the LRR model efficiently and incorporates the solution as prior knowledge into the deep network to guide the optimization of parameters. Moreover, LRR-Net transforms the regularized parameters into trainable parameters of the deep neural network, thus alleviating the need for manual parameter tuning. Additionally, this paper proposes a sparse neural network embedding to demonstrate the scalability of the LRR-Net framework. Empirical evaluations on eight distinct datasets illustrate the efficacy and superiority of the proposed approach compared to state-of-the-art methods.

International audience

Hyperspectral unmixing aims at estimating pure spectral signatures and their proportions in each pixel. In practice, the atmospheric effects, intrinsic variation of the spectral signatures of the materials, illumination, and topographic changes cause what is known as spectral variability resulting in significant estimation errors being propagated throughout the unmixing task. To this end, we developed a new method, called the orthogonal subspace unmixing (OSU), to address spectral variability by utilizing the orthogonal subspace projection. The proposed OSU method jointly performs orthogonal subspace learning and the unmixing process to find a more suitable subspace for unmixing. The orthogonal subspace projection encourages the representation held in the subspace to be more distinct from each other to remove the complex spectral variability in the subspace. Furthermore, an alternating minimization (AM) was designed to solve the resulting optimization problem. An efficient and convergent symmetric Gauss–Seidel alternating direction method of multipliers (sGS-ADMM), essentially a special case of the semiproximal alternating direction method of multipliers (SPADMM), was developed to solve the subproblem. Experiments conducted on one synthetic data and two real data demonstrate the effectiveness and superiority of the proposed framework in mitigating the effects of spectral variability with respect to classical linear unmixing methods or variability accounting approaches.

Vision transformers (ViTs) have been trending in image classification tasks due to their promising performance when compared to convolutional neural networks (CNNs). As a result, many researchers have tried to incorporate ViTs in hyperspectral image (HSI) classification tasks. To achieve satisfactory performance, close to that of CNNs, transformers need fewer parameters. ViTs and other similar transformers use an external classification (CLS) token which is randomly initialized and often fails to generalize well, whereas other sources of multimodal datasets, such as light detection and ranging (LiDAR) offer the potential to improve these models by means of a CLS. In this paper, we introduce a new multimodal fusion transformer (MFT) network which comprises a multihead cross patch attention (mCrossPA) for HSI land-cover classification. Our mCrossPA utilizes other sources of complementary information in addition to the HSI in the transformer encoder to achieve better generalization. The concept of tokenization is used to generate CLS and HSI patch tokens, helping to learn a distinctive representation in a reduced and hierarchical feature space. Extensive experiments are carried out on widely used benchmark datasets i.e., the University of Houston, Trento, University of Southern Mississippi Gulfpark (MUUFL), and Augsburg. We compare the results of the proposed MFT model with other state-of-the-art transformers, classical CNNs, and conventional classifiers models. The superior performance achieved by the proposed model is due to the use of multihead cross patch attention. The source code will be made available publicly at https://github.com/AnkurDeria/MFT.

Efforts to enhance the detection accuracy of hyperspectral (HS) anomaly detection (AD) have been significant, but the impact of noise resulting from HS data acquisition and transmission has not been well studied. Furthermore, the separation of denoising and subsequent interpretation makes it challenging to evaluate and control the influence of noise on the detection results. To this end, we proposed a joint anomaly detection and noise removal (ADNR) paradigm called DSR-ADNR, which develops a double subspace representation method to obtain both denoised and detection results simultaneously. DSR-ADNR uses a low-dimensional orthogonal basis to represent HS images and extract distinctive features for AD. The feature matrix is represented by a dictionary-based low-rank subspace that captures the complex nature of the low-dimensional features. In each iteration, DSR-ADNR utilizes the nonlocal self-similarity of the feature matrix to remove noise and improve intermediate detection performance. Meanwhile, the progressive LR representation of the background and anomalies for the feature matrix upgrades the explicit LR expression of nonlocal self-similar patches for better denoising. The well-designed linearized alternating direction method of multipliers with an adaptive penalty (LADMAP) is utilized to solve the proposed DSR-ADNR. Extensive experiments on simulated and real-world data sets demonstrate the effectiveness of DSR-ADNR in the HS AD task under different noise cases.

Transformer has been widely used in classification tasks for hyperspectral images (HSI) in recent years. Because it can mine spectral sequence information to establish long-range dependence, its classification performance can be comparable with the convolutional neural network (CNN). However, both CNN and Transformer focus excessively on spatial or spectral domain features, resulting in an insufficient combination of spatial-spectral domain information from HSI for modeling. To solve this problem, we propose a new end-to-end graph convolutional network (GCN) and Transformer fusion network with the spatial-spectral feature extraction (GTFN) in this paper, which combines the strengths of GCN and Transformer in both spatial and spectral domain feature extraction, taking full advantage of the contextual information of classified pixels while establishing remote dependencies in the spectral domain compared with previous approaches. In addition, GTFN uses Follow Patch as an input to the GCN and effectively solves the problem of high model complexity while mining the relationship between pixels. It is worth noting that the spectral attention module is introduced in the process of GCN feature extraction, focusing on the contribution of different spectral bands to the classification. More importantly, to overcome the problem that Transformer is too scattered in the frequency domain feature extraction, a neighborhood convolution module is designed to fuse the local spectral domain features. On Indian Pines, Salinas, and Pavia University datasets, the overall accuracies (OAs) of our GTFN are 94.00%, 96.81%, and 95.14%, respectively. The core code of GTFN is released at https://github.com/1useryang/GTFN.

The emerging research line of cross-modal learning focuses on the issue of transferring feature representation manner learned from limited multimodal data with labelings to the testing phase with partial modalities. This is essentially common and practical in the remote sensing community when only modal-incomplete data are in users’ hands due to inevitable imaging or access restrictions under large-scale observation scenarios. However, most of the existing cross-modal learning methods have been designed with exclusive reliance on labeling, which can be either limited or noisy due to their costly production. To address this issue, we explore in this paper the possibility to learn cross-modal feature representation in an unsupervised fashion. By integrating the multimodal data into a fully recombined matrix form, we propose 1) the use of common subspace representation as the regression target instead of conventionally adopted binary labels, and 2) the orthogonality and manifold alignment regularization terms to shrink the solution space whilst preserving the pairwise manifold correlations. Through this manner, the modality-specific and mutual latent representations in this common subspace as well as their corresponding projections can be learned simultaneously and their optimums can be efficiently reached through a nearly one-step computation with the help of Eigen decomposition. Finally, we show the superiority of our method through extensive image classification experiments on three multimodal datasets with four remotely sensed modalities involved (i.e., hyperspectral, multispectral, synthetic aperture radar, and light detection and ranging data). The code and dataset will be made freely available at https://github.com/jingyao16/UCSL after a possible publication to encourage the reproduction of our method and further use.

The recent success of attention mechanism-driven deep models, like Vision Transformer (ViT) as one of the most representative, has intrigued a wave of advanced research to explore their adaptation to broader domains. However, current Transformer-based approaches in the remote sensing (RS) community pay more attention to single-modality data, which might lose expandability in making full use of the ever-growing multimodal Earth observation data. To this end, we propose a novel multimodal deep learning framework by extending conventional ViT with minimal modifications, abbreviated as ExViT, aiming at the task of land use and land cover classification. Unlike common stems that adopt either linear patch projection or deep regional embedder, our approach processes multimodal RS image patches with parallel branches of position-shared ViTs extended with separable convolution modules, which offers an economical solution to leverage both spatial and modality-specific channel information. Furthermore, to promote information exchange across heterogeneous modalities, their tokenized embeddings are then fused through a cross-modality attention module by exploiting pixel-level spatial correlation in RS scenes. Both of these modifications significantly improve the discriminative ability of classification tokens in each modality and thus further performance increase can be finally attained by a full tokens-based decision-level fusion module. We conduct extensive experiments on two multimodal RS benchmark datasets, i.e., the Houston2013 dataset containing hyperspectral and light detection and ranging (LiDAR) data, and Berlin dataset with hyperspectral and synthetic aperture radar (SAR) data, to demonstrate that our ExViT outperforms concurrent competitors based on Transformer or convolutional neural network (CNN) backbones, in addition to several competitive machine learning-based models. The source codes and investigated datasets of this work will be made publicly available at https://github.com/jingyao16/ExViT.

Recently, there has been growing interest in hyperspectral images (HSIs) classification tasks, with both Graph Neural Networks (GNN) and Convolutional Neural Networks (CNN) proving to be effective means of analysis. GNN can better capture the spatial structure of HSIs in large target irregular regions through superpixel segmentation, while CNN can refine classification tasks by processing pixel-level features in small target regular regions. However, neither GNN nor CNN models alone can simultaneously consider superpixel-level and pixel-level features to cover both large and small target regions. To fully utilize the strengths of GNN and CNN, we propose a novel model called the Graph-Polarized Fusion Network (GPF). The GPF consists of two branches: the Fusion Graph Neural Network (FGNN) classifier in the GNN branch conducts feature learning on large, irregular target regions using both Graph Convolutional Network (GCN) and Graph Attention Network (GAT) as feature extraction operators. The features are integrated using three aggregators, namely Min, Max, and Weighted Add, followed by updating the nodes through 2D convolutional layers. The Polarized Neural Network (PNN) classifier of the CNN branch primarily works on small, target regular regions using Polarized Self-Attention (PSA) to conduct high-resolution processing on the two dimensions of space and channel without increasing time loss. Additionally, GPF employs residual connections to extract features from long distances and multi-angles. It also uses weighted fusion to integrate the superpixel-level and pixel-level features obtained from the two branches. Rigorous experiments on five real datasets demonstrate that GPF can fully mine the latent features of HSIs, achieving competitive results compared with other state-of-the-art methods.

The land cover classification of single-modal remote sensing (RS) data has recently reached a bottleneck. The joint use of multi-modal RS data to improve classification performances has received much attention. Convolutional Neural Networks are powerful tools in feature extraction and contextual modeling. While they have attendant drawbacks to capture the sequence attributes of spectral signatures and struggle to acquire discriminative spectral-spatial features from a global perspective due to limitations inherent in their network backbones. The transformer backbone is a promising approach for addressing these challenges and generating novel insights in multi-modal RS image classification. In this article, we present a new model called Multi-modal Transformer Network (MTNet) that leverages transformer advantages to capture both the specific and shared characteristics of hyperspectral (HS) and light detection and ranging (LiDAR) data. HS images contain a wide range of bands with rich spectral information and LiDAR data provide accurate elevation information without affecting by environmental factors. The well-designed module Hyperspectral Spectral Transformer can learn spectrally local sequence information from neighbouring bands of HS images, yielding group-wise spectral embeddings comprising rich diagnostic information about land covers. Furthermore, the HS and LiDAR spatial transformers aim to mine the pixel-wise feature embedding relationships in a global manner, capturing spatial and elevation information of HS and LiDAR, respectively. Finally, the feature embedding tokens of two modalities are integrated jointly and a new transformer encoder is redesigned to explore the shared spatial characteristics between the two modalities. We evaluate the classification performances of the proposed MTNet on three public HS-LiDAR datasets by conducting extensive experiments, exhibiting superiority over conventional classifiers and state-of-the-art networks.

Pansharpening is a procedure that fuses high-resolution panchromatic (PAN) images and low-resolution multispectral (LMS) images to derive high-resolution multispectral (HMS) images. Despite its rapid development, most existing pansharpening techniques integrate the information of PAN and LMS invariantly in the spatial dimension, ignoring the uneven spatial dependence of restoring HMS with the aid of PAN information and resulting in ineffective fusion results. In this work, we propose an Uncertainty-aware Adaptive Pansharpening Network (UAPN) that integrates PAN information spatial-variantly to restore LMS information with an uncertainty mechanism. Specifically, we first estimate the epistemic and aleatoric uncertainties together, which model the spatial-variant distributions of restoring the LMS image to the HMS image. Then, we introduce Uncertainty-conditioned Adaptive Convolution (UAC) to adaptively integrate LMS and PAN information, where its parameters are spatially variable by conditioning on the uncertainty estimations. Furthermore, we propose a multi-stage uncertainty-driven loss function to explicitly force the network to concentrate on restoring challenging areas of the LMS image. Extensive experimental results demonstrate the superiority of our UAPN with fewer parameters and flops, outperforming other state-of-the-art methods both qualitatively and quantitatively on multiple satellite datasets. The code is available at https://github.com/keviner1/UAPN..

Pan-sharpening aims to super-solve low-spatial resolution multiple spectral (MS) images with the guidance of high-resolution (HR) texture-rich panchromatic (PAN) images. Recently, deep-learning-based pan-sharpening approaches have dominated this field and achieved remarkable advancement. However, most promising algorithms are devised in one-way mapping and have not fully explored the mutual dependencies between PAN and MS modalities, thus impacting the model performance. To address this issue, we propose a novel information compensation and integration network for pan-sharpening by effective cross-modality joint learning in this work. First, the cross-central difference convolution is employed to explicitly extract the texture details of the PAN images. Second, we implement the compensation process by imitating the classical back-projection (BP) technique where the extracted PAN textures are employed to guide the intrinsic information learning of MS images iteratively. Subsequently, we devise the hierarchical transformer to integrate the comprehensive relations of stage-iteration information from spatial and temporal contexts. Extensive experiments over multiple satellite datasets demonstrate the superiority of our method to the existing state-of-the-art methods. The source code is available athttps://github.com/manman1995/pansharpening.

Learning-based infrared small object detection methods currently rely heavily on the classification backbone network. This tends to result in tiny object loss and feature distinguishability limitations as the network depth increases. Furthermore, small objects in infrared images are frequently emerged bright and dark, posing severe demands for obtaining precise object contrast information. For this reason, we in this paper propose a simple and effective "U-Net in U-Net" framework, UIU-Net for short, and detect small objects in infrared images. As the name suggests, UIU-Net embeds a tiny U-Net into a larger U-Net backbone, enabling the multi-level and multi-scale representation learning of objects. Moreover, UIU-Net can be trained from scratch, and the learned features can enhance global and local contrast information effectively. More specifically, the UIU-Net model is divided into two modules: the resolution-maintenance deep supervision (RM-DS) module and the interactive-cross attention (IC-A) module. RM-DS integrates Residual U-blocks into a deep supervision network to generate deep multi-scale resolution-maintenance features while learning global context information. Further, IC-A encodes the local context information between the low-level details and high-level semantic features. Extensive experiments conducted on two infrared single-frame image datasets, i.e., SIRST and Synthetic datasets, show the effectiveness and superiority of the proposed UIU-Net in comparison with several state-of-the-art infrared small object detection methods. The proposed UIU-Net also produces powerful generalization performance for video sequence infrared small object datasets, e.g., ATR ground/air video sequence dataset. The codes of this work are available openly at https://github.com/danfenghong/IEEE.

Owing to the powerful feature extraction and representation capabilities, deep learning (DL) has been successfully applied in hyperspectral unmixing (HU). However, only relying on hyperspectral data for unmixing fails to distinguish objects with similar spectral information, resulting in the degradation of unmixing performance. To this end, this paper presents a novel multimodal unmixing network, MUNet for short, by considering the height information of light detection and ranging (LiDAR) data in a squeeze-and-excitation (SE) attention fashion to guide the unmixing process toward a more accurate performance. MUNet is capable of efficiently embedding the height information obtained from LiDAR data into the autoencoder unmixing architecture through the attention mechanism, thereby fusing more spatial information to obtain ideal unmixing results. Experimental results conducted on the real multimodal dataset demonstrate the effectiveness and superiority of the proposed MUNet compared to several state-of-the-art deep unmixing approaches.

Over the past few decades, a large collection of feature ex-traction and classification algorithms have been developed for land cover mapping using remote sensing data. Although these methods have shown the gradually-increasing performance, their potential inevitably meets the bottleneck due to the lack of high-quality and diversified remote sensing bench-mark datasets, particularly for the multimodal cases. Accordingly, this, to a larger extent, limits the development of the corresponding methodologies and the practical application of land cover classification. To this end, we aim in this pa-per to introduce and build several multimodal remote sensing benchmark datasets for land cover classification. Further-more, two new multimodal land cover classification bench-mark datasets, i.e., Berlin and Augsburg, are openly available. Experiments are conducted on the two datasets for evaluating the performance of several multimodal feature learning and classification methods.

Panchromatic (PAN) and multi-spectral (MS) image fusion, named Pan-sharpening, refers to super-resolve the low-resolution (LR) multi-spectral (MS) images in the spatial domain to generate the expected high-resolution (HR) MS images, conditioning on the corresponding high-resolution PAN images. In this paper, we present a simple yet effective alternating reverse filtering network for pan-sharpening. Inspired by the classical reverse filtering that reverses images to the status before filtering, we formulate pan-sharpening as an alternately iterative reverse filtering process, which fuses LR MS and HR MS in an interpretable manner. Different from existing model-driven methods that require well-designed priors and degradation assumptions, the reverse filtering process avoids the dependency on pre-defined exact priors. To guarantee the stability and convergence of the iterative process via contraction mapping on a metric space, we develop the learnable multi-scale Gaussian kernel module, instead of using specific filters. We demonstrate the theoretical feasibility of such formulations. Extensive experiments on diverse scenes to thoroughly verify the performance of our method, significantly outperforming the state of the arts.

Polarimetric synthetic aperture radar (PolSAR) image classification has been investigated vigorously in various remote sensing applications. However, it is still a challenging task nowadays. One significant barrier lies in the speckle effect embedded in the PolSAR imaging process, which greatly degrades the quality of the images and further complicates the classification. To this end, we present a novel PolSAR image classification method that removes speckle noise via low-rank (LR) feature extraction and enforces smoothness priors via the Markov random field (MRF). Especially, we employ the mixture of Gaussian-based robust LR matrix factorization to simultaneously extract discriminative features and remove complex noises. Then, a classification map is obtained by applying a convolutional neural network with data augmentation on the extracted features, where local consistency is implicitly involved, and the insufficient label issue is alleviated. Finally, we refine the classification map by MRF to enforce contextual smoothness. We conduct experiments on two benchmark PolSAR data sets. Experimental results indicate that the proposed method achieves promising classification performance and preferable spatial consistency.

In this letter, a novel neural architecture search (NAS) method based on reinforcement learning, called RLNAS, is devised to realize the automatic architecture design in the field of hyperspectral unmixing (HU). This method first train the search network in the constructed self-supervised datasets based on hyperspectral images. The block-based searching and weight-sharing strategies are then introduced to reduce the computational cost in the training phase. The final optimal architecture is obtained by optimizing the multi-objective reward function to balance the trade-off between accuracy and computational efficiency. Compared with the state-of-the-art unmixing algorithms, the proposed RLNAS method can yield better unmixing results on synthetic and real hyperspectral datasets, which verifies its effectiveness and superiority. In addition, the proposed method offers promising potential of the NAS for HU.

Deep learning (DL) has been garnering increasing attention in remote sensing (RS) due to its powerful data representation ability. In particular, deep models have been proven to be effective for RS data classification based on a single given modality. However, with one single modality, the ability in identifying the materials remains limited due to the lack of feature diversity. To overcome this limitation, we present a simple but effective multimodal DL baseline by following a deep encoder–decoder network architecture, EndNet for short, for the classification of hyperspectral and light detection and ranging (LiDAR) data. EndNet fuses the multimodal information by enforcing the fused features to reconstruct the multimodal input in turn. Such a reconstruction strategy is capable of better activating the neurons across modalities compared with some conventional and widely used fusion strategies, e.g., early fusion, middle fusion, and late fusion. Extensive experiments conducted on two popular hyperspectral and LiDAR data sets demonstrate the superiority and effectiveness of the proposed EndNet in comparison with several state-of-the-art baselines in the hyperspectral-LiDAR classification task. The codes will be available athttps://github.com/danfenghong/IEEE_GRSL_EndNet, contributing to the RS community.

Owing to easy acquisition and large coverage from the space, multispectral (MS) imaging has garnered growing interest in various applications of remote sensing. However, the limited spectral information of MS data, to a great extent, leads to difficulties in classifying the materials more accurately, particularly for those classes that have very similar visual appearances. To address this issue effectively, we attempt to enhance the spectral resolution of MS imagery, enabling the identification of materials at a more precise level by the means of richer spectral information. More specifically, we propose to learn locality-constrained sparse coding (LCSC) for short, on partially overlapped hyperspectral (HS)-MS pairs (i.e., dictionary). LCSC is capable of capturing neighboring relations well by enforcing the local constraint for each pixel. Such a strategy makes it possible to better reconstruct HS products from MS images and partially overlapped HS images. Reconstruction and unmixing are explored as potential applications to assess the performance of spectral enhancement. Extensive experiments are conducted on two HS-MS data sets in comparison with several state-of-the-art baselines, which demonstrate the effectiveness of the proposed LCSC algorithm in the task of spectral enhancement.

Hyperspectral image super-resolution (HISR) is to fuse a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI), aiming to obtain a high-resolution hyperspectral image (HR-HSI). Recently, various convolution neural network (CNN) based techniques have been successfully applied to address the HISR problem. However, these methods generally only consider the relation of a local neighborhood by convolution kernels with a limited receptive field, thus ignoring the global relationship in a feature map. In this paper, we design a transformer-based architecture (called Fusformer) for the HISR problem, which is the first attempt to apply the transformer architecture to this task to the best of our knowledge. Thanks to the excellent ability of feature representations, especially by the self-attention in the transformer, our approach can globally explore the intrinsic relationship within features. Considering the specific HISR problem, since the LR-HSI holds the primary spectral information, our method estimates the spatial residual between the upsampled LR-MSI and the desired HR-HSI, reducing the burden of training the whole data in a smaller mapping space. Various experiments show that our approach outperforms current state-of-the-art HISR methods. The code is available at https://github.com/J-FHu/Fusformer.

Hyperspectral images (HSIs) usually have finer spectral resolution but coarser spatial resolution than multispectral images (MSIs). To obtain a desired HSI with higher spatial resolution, great research attention has been paid to achieving hyperspectral super-resolution by fusing the observed HSI with an auxiliary MSI of the same scene. However, most of the existing HSI-MSI fusion methods rely either on prior knowledge of the degradation model or on sufficient training data, hindering their practicality and interpretability. In this letter, we propose a novel unsupervised HSI-MSI fusion network with the ability of degradation adaptive learning, namely, UDALN. Specifically, we propose three modules to straightly encode the spatial and spectral transformations across resolutions, i.e., SpaDnet, SpeUnet, and SpeDnet. Through an elaborately designed three-stage unsupervised training strategy, the estimated network parameters can exhibit clear physical meanings of degradation processes and therefore help guarantee a faithful reconstruction of the desired HSI. The experimental results on two widely used hyperspectral datasets demonstrate the effectiveness of our method in comparison to the state-of-the-art HSI-MSI fusion models. (Code available athttps://github.com/JiaxinLiCAS/UDALN_GRSL.)

Convolutional neural networks (CNNs) have exhibited commendable performance in the hyperspectral images (HSIs) classification task with manually annotated limited available training data for supervision. The accurate classification of pixel-wise land covers using traditional CNNs is often hampered by the presence of wrong (noisy) labels in the training data and can easily be overfitted to the label noises. However, training on noisy labeled data inevitably suffers from performance degradation since CNNs tend to overfit the label noises. To overcome this problem, we propose a lightweight heterogeneous kernel convolution (HetConv3D) for HSI classification with noisy labels, whereHetConv3Duses two different types of convolutional kernels, i.e., spectral and spatial domains, and fuses them to produce the final feature maps that are less prompted to the noises and also reduces the computation time. The experiments are conducted using three well-known HSI datasets, i.e., Kennedy Space Center (KSC), Salinas Scene (SA), and University of Pavia (UP), and results are compared with traditional supervised classification methods, including support vector machine (SVM), random forest (RF), CNN3D, ContextNet, MS3DNet, lightweight dual-channel residual network (DCRN), and HetConv3DNet. The superior performance exhibited by the proposed modelHetConv3D-HSIconfirms the importance of learning a fusion of spatial and spectral kernel features. The source code will be made available publicly athttps://github.com/purbayankar/HetConv3DNet.

Due to sensor malfunction and atmosphere disturbances, high-dimensional optical remote sensing (HORS) images often suffer from information missing, such as dead pixels and thick clouds. Tensor decomposition methods have been used to estimate the missing data of HORS images. However, most existing models hardly consider the inherent properties and effective structural information of HORS images. To this end, we propose a novel total variation (TV) regularized weighted tensor ring (TR) decomposition model to recover the missing content of HORS images. The TR decomposition has the powerful low-rank (LR) representation ability to recover the HORS data by employing three low-dimensional tensors, i.e., TR factors. An initialization step and proper weights are designed to enhance the flexibility for exploring different LR properties of TR factors. To further preserve the spatial smoothness, the local spatial TV from three directions is incorporated into the TR decomposition framework. Furthermore, an augmented Lagrange multiplier (ALM) algorithm is designed for solving the resulting optimization problem. Experiments on HORS images demonstrate the performances of the proposed method over the current state-of-the-art baselines.

Infrared objects acquired from a long-distance have small sizes and are easily submerged by a complex and variable background. The existing deep network detection framework suffers greatly from the feature spatial resolution loss caused by the networks’ depth and multiple downsampling operations, which is extremely detrimental for small object detection. So, a crucial and urgent goal is, how to trade-off network depth and feature spatial resolution, while learning feature context representation and interaction to distinguish from the background. To this end, we propose a deep interactive U-Net architecture (short for DI-U-Net) with high feature learning and feature interaction ability. First, feature learning is first achieved through a multi-level and high-resolution network structure. This structure ensures feature resolution as the network depth increase, and also focus on the object’s global context information. Then, the feature interactive is further achieved by the dense feature encoder (DFI) module to learn object local context information. The proposed method yields strong object context representation and well discriminability, as well as a good fit for infrared small object detection. Extensive experiments are conducted on the SISRT dataset and Synthetic dataset, demonstrating the superiority and effectiveness of the proposed deeper U-Net compared to previous state-of-the-art detection methods.

Hyperspectral imaging has attracted growing interest among researchers from the geoscience and remote sensing fields owing to its very rich spectral information. However, the high spectral dimensionality of hyperspectral images (HSI) tends to suffer from information redundancy. Manifold embedding is a mainstream strategy of nonlinear hyperspectral dimensionality reduction (DR). The sensitivity to the noise and the inflexibility to the out-of-sample problem (i.e., new samples) are the main drawbacks of the manifold embedding-based methods. To this end, we propose to learn a deep embedder in a self-supervised fashion for hyperspectral DR, called hyper-embedder. Hyper-embedder effectively reduces the computational complexity and storage-costing compared to conventional embedding models and improves the robustness against various noises, e.g., spectral variabilities. More significantly, hyper-embedder is capable of learning an explicit nonlinear mapping to make a one-to-one match between each original pixel (spectral signature) in the HSI and its dimension-reduced representation. These low-dimensional representations can be generated and given by existing and classic nonlinear manifold embedding methods. In this letter, we attempt to learn the correspondence or mapping by optimizing a deep regression network. The to-be-developed network cannot only capture the local topological knowledge graph of all spectral signatures of hyperspectral data but be applicable to fast prediction and inference of samples from other hyperspectral scenes. The proposed hyper-embedder outperforms existing state-of-the-art hyperspectral DR algorithms on two commonly used hyperspectral datasets, i.e., Indian pines and Augsburg scenes.

Graph convolutional networks (GCN) have begun to show their potential in hyperspectral image classification in recent years. However, most of the current GCN methods are designed to learn node features on fixed and homogeneous graphs, and it is difficult for them to learn effective node features on heterogeneous graphs. The limitation is particularly evident in hyperspectral classification because of the different types of nodes and edges. The Graph Transformer Network with the graph attention mechanism (GTN-A) is proposed to address this shortcoming in this paper. It can generate a new graph structure, which is represented by a more useful meta-path, so that node features can be better aggregated. The experiments conducted on two benchmark datasets illustrate the effectiveness of our method.

Convolutional neural networks (CNN) have been widely used in hyperspectral image classification (HSIC). Although the current CNN-based methods have achieved good performance, they still face a series of challenges. For example, the receptive field is limited, information is lost in down-sampling layer, and a lot of computing resources are consumed for deep networks. To overcome these problems, we proposed a multi-level spectral-spatial transformer network (MSTNet) for HSIC. The structure of MSTNet is an image-based classification framework, which is efficient and straightforward. Based on this framework, we designed a self-attentive encoder. Firstly, HSIs are processed into sequences. Meanwhile, a learned positional embedding is added to integrate spatial information. Then, a pure transformer encoder is employed to learn feature representations. Finally, the multi-level features are processed by decoders to generate the classification results in the original image size. The experimental results based on three real hyperspectral data sets demonstrate the efficiency of the proposed method in comparison with the other related CNN-based methods.

Owing to the powerful and automatic representation capabilities, deep learning (DL) techniques have made significant breakthroughs and progress in hyperspectral unmixing (HU). Among the DL approaches, autoencoders (AEs) have become a widely-used and promising network architecture. However, these AE-based methods heavily rely on manual design and may not be a good fit for specific datasets. To unmix hyperspectral images more intelligently, we propose an automatic neural architecture search model for HU, AutoNAS for short, to determine the optimal network architecture by considering channel configurations and convolution kernels simultaneously. In AutoNAS, the self-supervised training mechanism based on hyperspectral images is first designed for generating the training samples of the supernet. Then, the affine parameter sharing strategy is adopted by applying different affine transformations on the supernet weights in the training phase, which enables finding the optimal channel configuration. Furthermore, on the basis of the obtained channel configuration, the evolutionary algorithm with additional computational constraints is introduced into networks to achieve flexible convolution kernel search by evaluating unmixing results of different architectures in the supernet. Extensive experiments conducted on four hyperspectral datasets demonstrate the effectiveness and superiority of the proposed AutoNAS in comparison with several state-of-the-art unmixing algorithms.

Convolutional Neural Networks (CNNs) have been extensively studied for Hyperspectral Image Classification (HSIC). However, CNNs are critically attributed to a large number of labeled training samples, which outlays high costs in terms of time and resources. Moreover, CNNs are trained on some samples and have been tested on the entire HSI. Perhaps, the entire HSI is taken into account at test time to appropriately generate the ground truth maps. In order to obtain a higher accuracy while considering the limited availability of training samples and disjoint validation and test samples, this work proposes a fast and compact 3D CNN-based Active Learning (AL) for HSIC that integrates both deep transfer learning and AL into a unified framework. In the proposed methodology, a 3D CNN model is trained with very few training samples (i.e., 5%, only) and in the next phase, the most informative and heterogeneous samples are queried from the validation set (candidate set) based on the fuzziness, mutual information and breaking ties of the trained model. The 3D CNN model is later fine-tuned (rather retraining from scratch) with the new training samples (i.e., 200 samples are selected in each iteration) to reduce the computational cost. The proposed method has been compared with the state-of-the-art traditional and deep models proposed for HSIC. Experimental results proved the superiority of our proposed method on several benchmark HSI datasets with significantly fewer labeled samples. Matlab demo can be accessed on GitHub: github.com/mahmad00.

Recently, deep learning (DL) approaches have been widely applied to the pansharpening problem, which is defined as fusing a low-resolution multispectral (LRMS) image with a high-resolution panchromatic (PAN) image to obtain a high-resolution multispectral (HRMS) image. However, most DL-based methods handle this task by designing black-box network architectures to model the mapping relationship from LRMS and PAN to HRMS. These network architectures always lack sufficient interpretability, which limits their further performance improvements. To address this issue, we adopt the model-driven method to design an interpretable deep network structure for pansharpening. First, we present a new pansharpening model using the convolutional sparse coding (CSC), which is quite different from the current pansharpening frameworks. Second, an alternative algorithm is developed to optimize this model. This algorithm is further unfolded to a network, where each network module corresponds to a specific operation of the iterative algorithm. Therefore, the proposed network has clear physical interpretations, and all the learnable modules can be automatically learned in an end-to-end way from the given dataset. Experimental results on some benchmark datasets show that our network performs better than other advanced methods both quantitatively and qualitatively.

Hyperspectral image (HSI) has received considerable attention in the field of target detection due to its powerful ability to capture the spectral information of land covers, and plenty of detection algorithms have been explored. However, these methods generally leverage the difference between the spectrum of the target to be detected and the background spectrum to accomplish target detection, and so are susceptible to the problem of spectral variability. In this article, we propose a global-to-local hierarchical detection algorithm for HSI (G2LHTD). Firstly, extended morphological attribute profile (EMAP) is first used to model global spatial texture information from HSI. Subsequently, a diverse-direction constrained energy minimization (D2CEM) detector is developed to consider the spatial information within eight neighborhoods around each pixel in HSI, yielding comprehensive local spatial information. More substantially, to effectively discriminate the neighborhood information in diverse directions, we devise an adaptive neighborhood feature aggregation (ANFA) strategy, which will comprehensively evaluate the significance of neighborhood information in diverse directions. As a result, the spatial features of HSI can be comprehensively considered for hyperspectral target detection (HTD). Extensive experiments, conducted on four standard datasets, demonstrate the effectiveness of the proposed method. The codes of this work will be available at https://github.com/zhonghaocheng/G2LHTD_Master for the sake of reproducibility.

In recent years, deep learning (DL) has attracted increasing attention in hyperspectral unmixing (HU) applications due to its powerful learning and data fitting ability. The autoencoder (AE) framework, as an unmixing baseline network, achieves good performance in HU by automatically learning low-dimensional embeddings and reconstructing data. Nevertheless, the conventional AE-based architecture, which focuses more on the pixel-level reconstruction loss, tends to lose some significant detailed information of certain materials (e.g., material-related properties) in the reconstruction process. Therefore, inspired by the perception mechanism, we propose a cycle-consistency unmixing network, called CyCU-Net, by learning two cascaded AEs in an end-to-end fashion, to enhance the unmixing performance more effectively. CyCU-Net is capable of reducing the detailed and material-related information loss in the process of reconstruction by relaxing the original pixel-level reconstruction assumption to cycle consistency dominated by the cascaded AEs. More specifically, cycle consistency can be achieved by a newly proposed self-perception loss, which consists of two spectral reconstruction terms and one abundance reconstruction term. By taking advantage of the self-perception loss in the network, the high-level semantic information can be well preserved in the unmixing process. Moreover, we investigate the performance gain of CyCU-Net with extensive ablation studies. Experimental results on one synthetic and three real hyperspectral data sets demonstrate the effectiveness and competitiveness of the proposed CyCU-Net in comparison with several state-of-the-art unmixing algorithms.

Deep learning (DL) has aroused wide attention in hyperspectral unmixing (HU) owing to its powerful feature representation ability. As a representative of unsupervised DL approaches, autoencoder (AE) has been proven to be effective to better capture nonlinear components of hyperspectral images than the traditional model-driven linearized methods. However, only using hyperspectral images for unmixing fails to distinguish objects in complex scene, especially for different endmembers with similar materials. To overcome this limitation, we propose a novel multimodal unmixing network for hyperspectral images, called MUNet, by considering the height differences of light detection and ranging (LiDAR) data in a squeeze-and-excitation (SE)-driven attention fashion to guide the unmixing process, yielding performance improvement. MUNet is capable of fusing multimodal information and using the attention map derived by LiDAR to aid network that focuses on more discriminative and meaningful spatial information regarding scenes. Moreover, attribute profile (AP) is adopted to extract the geometrical structures of different objects to better model the spatial information of LiDAR. Experimental results on synthetic and real datasets demonstrate the effectiveness and superiority of the proposed method compared with several state-of-the-art unmixing algorithms. The codes will be available athttps://github.com/hanzhu97702/IEEE_TGRS_MUNet, contributing to the remote sensing community.

Hyperspectral (HS) images are characterized by approximately contiguous spectral information, enabling the fine identification of materials by capturing subtle spectral discrepancies. Owing to their excellent locally contextual modeling ability, convolutional neural networks (CNNs) have been proven to be a powerful feature extractor in HS image classification. However, CNNs fail to mine and represent the sequence attributes of spectral signatures well due to the limitations of their inherent network backbone. To solve this issue, we rethink HS image classification from a sequential perspective with transformers, and propose a novel backbone network called \ul{SpectralFormer}. Beyond band-wise representations in classic transformers, SpectralFormer is capable of learning spectrally local sequence information from neighboring bands of HS images, yielding group-wise spectral embeddings. More significantly, to reduce the possibility of losing valuable information in the layer-wise propagation process, we devise a cross-layer skip connection to convey memory-like components from shallow to deep layers by adaptively learning to fuse "soft" residuals across layers. It is worth noting that the proposed SpectralFormer is a highly flexible backbone network, which can be applicable to both pixel- and patch-wise inputs. We evaluate the classification performance of the proposed SpectralFormer on three HS datasets by conducting extensive experiments, showing the superiority over classic transformers and achieving a significant improvement in comparison with state-of-the-art backbone networks. The codes of this work will be available at https://github.com/danfenghong/IEEE_TGRS_SpectralFormer for the sake of reproducibility.

The existing cloud removal methods either need a cloud mask as prior knowledge or detect clouds before cloud removal processing, i.e., the detection and removal processes are separate. In this article, we propose a box-constrained (BC) smooth low-rank plus group sparse model to simultaneously detect and remove clouds, by formulating the degraded data as the summation of image and cloud components. For the cloud component, we propose a group sparse function along the spectral dimension. This is motivated by our observations that: 1) one tube is contaminated by clouds if any pixels of this tube are contaminated by clouds; 2) the positions of tubes, which are taken at different times, contaminated by clouds are different. For the image component, we propose to use a tensor rank based on the tensor singular value decomposition. The tensor rank characterizes the global property of the image component and could not keep the cloud-free information unchanged. To address the problem, we introduce a BC on the image component to force its cloud-free information to be equal to the corresponding values of observed data. Owing to the BC, the proposed model integrates the cloud detection and removal processes so that the two processes could promote each other and result in a promising result. To solve the proposed model, we develop an efficient algorithm that can generate the cloud mask, image component, and cloud component alternately. Extensive experiments on synthetic and real data show that the proposed method is competitive compared with the completion and other cloud removal methods.

Data fusion of hyperspectral and light detection and ranging (LiDAR) is conducive to obtain more comprehensive surface information and thereby achieve better classification result in Earth Monitoring Systems. However, lack of labeled samples usually limits the performance of supervised classifiers, and the heterogeneity of multi-source data also brings great challenges to data fusion. Aiming to address these issues, we propose a triplet semi-supervised deep convolutional neural network (TSDN) for fusion classification of hyperspectral and LiDAR. Specifically, we utilize three basic pathways to extract deep learning features: 1D-CNN for spectral features in hyperspectral, 2D-CNN for spatial features in hyperspectral and Cascade Net for elevation features in LiDAR data. Furthermore, a novel label calibration module (LCM) is proposed to generate effective pseudo labels with high confidence based on the superpixel segmentation by comparing the multi-view classification results for assisting semi-supervised model training. In addition, we design a novel 3D-Cross Attention Block to enhance the complementary spatial features of multi-source data. Experiments on three public HSI-LiDAR benchmarks: Houston, Trento, and MUUFL Gulfport have demonstrated the effectiveness and superiority of our proposed method.

Modality translation, which aims to translate images from a source modality to a target one, has attracted a growing interest in the field of remote sensing recently. Compared to translation problems in multimedia applications, modality translation in remote sensing often suffers from inherent ambiguities, i.e., a single input image could correspond to multiple possible outputs, and the results may not be valid in the following image interpretation tasks, such as classification and change detection. To address these issues, we make the attempt to utilizing time-series data to resolve the ambiguities. We propose a novel multimodality image translation framework, which exploits temporal information from two aspects: 1) by introducing a guidance image from given temporally neighboring images in the target modality, we employ a feature mask module and transfer semantic information from temporal images to the output without requiring the use of any semantic labels and 2) while incorporating multiple pairs of images in time series, a temporal constraint is formulated during the learning process in order to guarantee the uniqueness of the prediction result. We also build a multimodal and multitemporal dataset that contains synthetic aperture radar (SAR), visible, and short-wave length infrared band (SWIR) image time series of the same scene to encourage and promote research on modality translation in remote sensing. Experiments are conducted on the dataset for two cross-modality translation tasks (SAR to visible and visible to SWIR). Both qualitative and quantitative results demonstrate the effectiveness and superiority of the proposed model.

An important task of spectral unmixing is to recover the signatures of endmembers from a hyperspectral dataset in which no pure signature is exposed. Most algorithms are based on linear mixing model and require that data should be sufficiently scattered to guarantee model uniqueness. However, data may not be scattered sufficiently enough and become incomplete. In this case, can we still recover the endmembers accurately? Moreover, if we wrongly estimate an endmember, the error may be propagated to other endmembers. For these purposes, we propose a new model, namely, affine simplicial cone (ASC), to capture the local geometric feature of data. This model requires less geometric information and relaxes the conditions of model uniqueness. Then, we present analyses on the error propagation of turbulent endmembers and the condition of local uniqueness. Based on ASC, we present an endmember purification problem to recover only one endmember. In this way, error propagation can be alleviated, and local uniqueness can be easily satisfied. Finally, we develop an endmember purifying algorithm (EPA) to solve this problem. Our experiments demonstrate that the performance of EPA is competitive to the state-of-the-art unmixing algorithms not only for the synthetic datasets but also for the real hyperspectral remote sensing datasets. We can conclude that the ASC model and the EPA algorithm have the potential capability for hyperspectral data exploration.

Convolutional Neural Networks (CNNs) have been extensively utilized for Hyperspectral (HSI) as well as Light Detection and Ranging (LiDAR) data Classification. However, CNNs have not been much explored for joint HSI and LiDAR image classification. Therefore, this article proposes a joint feature learning (HSI and LiDAR) and fusion mechanism using CNN and Spatial Morphological blocks which generates highly accurate land-cover maps. The CNN model comprises three Conv3D layers and is directly applied to the HSIs for extracting discriminative spectral-spatial feature representation. On the contrary, the spatial morphological block is able to capture the information relevant to the height or shape of the different land-cover regions from LiDAR data. The LiDAR features are extracted using morphological dilation and erosion layers which increase the robustness of the proposed model by considering elevation information as an additional feature. Finally, both the obtained features from CNNs and spatial morphological blocks are combined using an additive operation prior to the classification. Extensive experiments are shown with widely used HSIs and LiDAR datasets, i.e., University of Houston (UH), Trento, and MUUFL Gulfport scene. The reported results show that the proposed model significantly outperforms traditional methods and other state-of-the-art deep learning models. The source code for the proposed model will be made available publicly at https://github.com/AnkurDeria/HSI+LiDAR.

The hyperspectral images are composed of a variety of textures across the different bands which increase the spectral similarity and make it difficult to predict the pixel-wise labels without inducing additional complexity at the feature level. To extract robust and discriminative features from the different regions of land cover, the hyperspectral research community is still seeking such type of convolutions which can efficiently deal with fine-grained texture information during the feature extraction phase, which often overlook this aspect by vanilla convolution. To overcome the above shortcoming, this article proposes a generalized gradient centralized 3D convolution (G2C-Conv3D) operation, which is a weighted combination between the vanilla and gradient centralized 3D convolutions (GC-Conv3D) to extract both theintensity-levelsemantic information andgradient-levelinformation. This can be easily plugged into the existing HSI feature extraction networks to boost the performance of accurate prediction for land-cover types. To validate the feasibility of the proposedG2C-Conv3D, we have considered the existing CNN3D, MS3DNet, ContextNet, and SSRN feature extraction models and as well as CAE3D, VAE3D, and SAE3D autoencoder (AE) networks, respectively. All these networks are embedded withG2C-Conv3Dconvolution to implement both generalized gradient centralized feature extraction networks (G2C-FE) and generalized gradient centralized AE networks (G2C-AE) for fine-grained spectral–spatial feature learning. In addition,G2C-Conv2Dis also considered with few networks. The extensive experiments are conducted on four most widely used hyperspectral datasets i.e., IP, KSC, UH, and UP, respectively, and compared with the nine methods. The results demonstrate that the proposedG2C-Conv3Dcan effectively enhance the feature learning ability of the existing networks and both the qualitative and quantitative results show the superiority and effectiveness of the proposedG2C-Conv3D. The source codes will be publicly available athttps://github.com/danfenghong/G2C-Conv3D-HSI.

Convolutional neural networks (CNNs) with excellent spatial feature extraction abilities have become popular in remote sensing (RS) image change detection (CD). However, CNNs often focus on the extraction of spatial information but ignore important spectral and temporal sequences for hyperspectral images (HSIs). In this paper, we propose a joint spectral, spatial, and temporal transformer for hyperspectral image change detection (HSI-CD), named SST-Former. First, the SST-Former position-encodes each pixel on the cube to remember the spectral and spatial sequences. Second, a spectral transformer encoder structure is used to extract spectral sequence information. Then, a class token for storing the class information of a single temporal HSI concatenates the output of the spectral transformer encoder. The spatial transformer encoder is used to extract spatial texture information in the next step. Finally, the features of different temporal HSIs are sent as the input of temporal transformer, which is used to extract useful CD features between the current HSI pairs and obtain the binary CD result through multilayer perception (MLP). We evaluate SST-Former on three HSI-CD datasets by numerous experiments, showing that it performs better than other excellent methods both visually and qualitatively.

In recent years, enormous research has been made to improve the classification performance of single-modal remote sensing (RS) data. However, with the ever-growing availability of RS data acquired from satellite or airborne platforms, simultaneous processing and analysis of multimodal RS data pose a new challenge to researchers in the RS community. To this end, we propose a deep-learning-based new framework for multimodal RS data classification, where convolutional neural networks (CNNs) are taken as a backbone with an advanced cross-channel reconstruction module, called CCR-Net. As the name suggests, CCR-Net learns more compact fusion representations of different RS data sources by the means of the reconstruction strategy across modalities that can mutually exchange information in a more effective way. Extensive experiments conducted on two multimodal RS datasets, including hyperspectral (HS) and light detection and ranging (LiDAR) data, i.e., the Houston2013 dataset, and HS and synthetic aperture radar (SAR) data, i.e., the Berlin dataset, demonstrate the effectiveness and superiority of the proposed CCR-Net in comparison with several state-of-the-art multimodal RS data classification methods. The codes will be openly and freely available athttps://github.com/danfenghong/IEEE_TGRS_CCR-Netfor the sake of reproducibility.

In this article, we propose a novel graph convolutional network (GCN) for pansharpening, defined as GCPNet, which consists of three main modules: the spatial GCN module (SGCN), the spectral band GCN module (BGCN), and the atrous spatial pyramid module (ASPM). Specifically, due to the nature of GCN, the proposed SGCN and BGCN are capable of exploring the long-range relationship between the object and the global state in the spatial and spectral aspects, which benefits pansharpened results and has not been fully investigated before. In addition, the designed ASPM is equipped with multiscale atrous convolutions and learns richer local feature information, so as to cover the objects of different sizes in satellite images. To further enhance the representation of our proposed GCPNet, asynchronous knowledge distillation is introduced to provide compact features by heterogeneous task imitation in a teacher–student paradigm. In the paradigm, the teacher network acts as a variational autoencoder to extract compact features of the ground-truth MS images. The student network, devised for pansharpening, is trained with the assistance of the teacher network to transfer the important information of the expected ground-truth MS images. Extensive experimental results on different satellite datasets demonstrate that our proposed network outperforms the state-of-the-art methods both visually and quantitatively. The source code is released athttps://github.com/Keyu-Yan/GCPNet.

Owing to the powerful data representation ability of deep learning (DL) techniques, tremendous progress has been recently made in hyperspectral image (HSI) classification. Convolutional neural network (CNN), as a main part of the DL family, has been proven to be considerably effective to extract spatial-spectral features for HSIs. Nevertheless, its classification performance, to a great extent, depends on the quality and quantity of samples in the network training process. To select those samples, either labeled or unlabeled, that can be used to enhance the generalization ability of CNNs and further improve the classification accuracy, we propose an iterative semi-supervised CNNs framework by means of active learning and superpixel segmentation techniques, dubbed as semi-active CNNs (SA-CNNs), for HSI classification. More specifically, we start to pre-train a CNNs-based model on a small-scale unbiased labeled set and infer unlabeled data using the trained model, i.e., generating pseudo-labels. Then, the reliable samples, which consist of two parts: high label-homogeneity and most informativeness, are actively selected from superpixel segments. These selected labeled and unlabeled samples with their labels and pseudo-labels are re-fed into the next-round network training. Moreover, three different schedules, i.e.,log-,exp-, andlinear-schedules, are progressively adopted to fully explore their potentials in sample selection, until a labeling budget is finally reached. Extensive experiments are conducted on three benchmark HSI datasets, demonstrating substantial performance improvements of the proposed SA-CNNs over other similar competitors.

Blind hyperspectral unmixing (HU) has long been recognized as a crucial component in analyzing the hyperspectral imagery (HSI) collected by airborne and spaceborne sensors. Due to the highly ill-posed problems of such a blind source separation scheme and the effects of spectral variability in hyperspectral imaging, the ability to accurately and effectively unmixing the complex HSI still remains limited. To this end, this article presents a novel blind HU model, called sparsity-enhanced convolutional decomposition (SeCoDe), by jointly capturing spatial–spectral information of HSI in a tensor-based fashion. SeCoDe benefits from two perspectives. On the one hand, the convolutional operation is employed in SeCoDe to locally model the spatial relation between the targeted pixel and its neighbors, which can be well explained by spectral bundles that are capable of addressing spectral variabilities effectively. It maintains, on the other hand, physically continuous spectral components by decomposing the HSI along with the spectral domain. With sparsity-enhanced regularization, an alternative optimization strategy with alternating direction method of multipliers (ADMM)-based optimization algorithm is devised for efficient model inference. Extensive experiments conducted on three different data sets demonstrate the superiority of the proposed SeCoDe compared to previous state-of-the-art methods. We will also release the code athttps://github.com/danfenghong/IEEE_TGRS_SeCoDeto encourage the reproduction of the given results.

Due to the limitations of imaging systems, satellite hyperspectral imagery (HSI), which yields rich spectral information in many channels, often suffers from poor spatial resolution. HSI super-resolution (SR) refers to the fusion of high spatial resolution multispectral imagery (MSI) and low spatial resolution HSI to generate HSI that has both a high spatial and high spectral resolution. However, most existing SR methods assume that the two original images used are perfectly registered: in reality, nonrigid deformation areas can exist locally in the two images even if prior registration of the control points has been carried out. To address this problem, we propose a novel unsupervised spectral unmixing and image deformation correction network—NonRegSRNet—with multimodal and multitask learning that can be used for the joint registration of HSI and MSI and to produce SR imagery. More specifically, NonRegSRNet integrates the dense registration and SR tasks into a unified model that includes a triplet convolutional neural network. This allows these two tasks to complement each other so that better registration and SR results can be achieved. Furthermore, because the point spread function (PSF) and spectral response function (SRF) are often unavailable, two special convolutional layers are designed to adaptively learn the parameters of the PSF and SRF, which makes the proposed model more adaptable. Experimental results demonstrate that the proposed method has the ability to produce highly accurate and stable reconstructed images under complex nonrigid deformation conditions. (Code available athttps://github.com/saber-zero/NonRegSRNet)

As recognized, the ground truth multi-spectral (MS) images possess the complementary information (e.g., high-frequency component) of low-resolution (LR) MS images, which can be considered as privileged information to alleviate the spectral distortion and insufficient spatial texture enhancement. Since existing supervised pan-sharpening methods only utilize the ground truth MS image to supervise the network training, its potential value has not been fully explored. To accomplish this, we propose a heterogeneous knowledge-distilling pan-sharpening framework that distills pan-sharpening by imitating the ground truth reconstruction task in both the feature space and network output. In our work, the teacher network performs as a variational auto-encoder to extract effective features of the ground truth MS. The student network, acting as pan-sharpening, is trained by the assistance of the teacher network with the process-oriented feature imitation learning. Moreover, we design a customized information-lossless multi-scale invertible neural module to effectively fuse LR-MS and panchromatic (PAN) images, producing expected pan-sharpened results. To reduce the artifacts generated by the knowledge distillation process, a knowledge-driven refinement sub-network is further devised according to the pan-sharpening imaging model. Extensive experimental results on different satellite datasets validate that the proposed network outperforms the state-of-the-art methods both visually and quantitatively. The source code will be released at https://github.com/manman1995/pansharpening.

Recently, embedding and metric-based few-shot learning (FSL) has been introduced into hyperspectral image classification (HSIC) and achieved impressive progress. To further enhance the performance with few labeled samples, we in this paper propose a novel FSL framework for HSIC with a class-covariance metric (CMFSL). Overall, the CMFSL learns global class representations for each training episode by interactively using training samples from the base and novel classes, and a synthesis strategy is employed on the novel classes to avoid overfitting. During the meta-training and meta-testing, the class labels are determined directly using the Mahalanobis distance measurement rather than an extra classifier. Benefiting from the task-adapted class-covariance estimations, the CMFSL can construct more flexible decision boundaries than the commonly used Euclidean metric. Additionally, a lightweight cross-scale convolutional network (LXConvNet) consisting of 3D and 2D convolutions is designed to thoroughly exploit the spectral-spatial information in the high-frequency and low-frequency scales with low computational complexity. Furthermore, we devise a spectral-prior-based refinement module (SPRM) in the initial stage of feature extraction, which cannot only force the network to emphasize the most informative bands while suppressing the useless ones, but also alleviate the effects of the domain shift between the base and novel categories to learn a collaborative embedding mapping. Extensive experiment results on four benchmark data sets demonstrate that the proposed CMFSL can outperform the state-of-the-art methods with few-shot annotated samples.

Over the past decades, enormous efforts have been made to improve the performance of linear or nonlinear mixing models for hyperspectral unmixing (HU), yet their ability to simultaneously generalize various spectral variabilities (SVs) and extract physically meaningful endmembers still remains limited due to the poor ability in data fitting and reconstruction and the sensitivity to various SVs. Inspired by the powerful learning ability of deep learning (DL), we attempt to develop a general DL approach for HU, by fully considering the properties of endmembers extracted from the hyperspectral imagery, called endmember-guided unmixing network (EGU-Net). Beyond the alone autoencoder-like architecture, EGU-Net is a two-stream Siamese deep network, which learns an additional network from the pure or nearly pure endmembers to correct the weights of another unmixing network by sharing network parameters and adding spectrally meaningful constraints (e.g., nonnegativity and sum-to-one) toward a more accurate and interpretable unmixing solution. Furthermore, the resulting general framework is not only limited to pixelwise spectral unmixing but also applicable to spatial information modeling with convolutional operators for spatial-spectral unmixing. Experimental results conducted on three different datasets with the ground truth of abundance maps corresponding to each material demonstrate the effectiveness and superiority of the EGU-Net over state-of-the-art unmixing algorithms. The codes will be available from the website: https://github.com/danfenghong/IEEE_TNNLS_EGU-Net.

Recently, the image segmentation has been a significant research direction in the field of optical remote sensing data processing. However, due to the limitation of the optical imaging mechanism, traditional image segmentation methods are not efficient for processing the optical remote sensing images, especially influencing by the complex weather conditions. In order to ensure the classification performance, synthetic aperture radar (SAR) data are employed as complementary to the data procedure for enhancing the capability of land cover interpretation. Then a dual-stream high-resolution network (HRNet) is proposed to combine two types of heterogeneous data (SAR and optical image), and a multi-modal squeeze-and-excitation (SE) module is exploited to make feature maps fused. Experiments show that the proposed method has excellent performance on the remote sensing data acquired by GF2 and GF3 satellites.

Owing to the powerful ability in learning low-dimensional representations and reconstruction, autoencoders (AEs) have been successfully applied in hyperspectral unmixing (HU). However, AE-based unmixing architectures, to a great extent, need to be carefully designed in a manual fashion, leading to the bulk of costs in manpower and time. To unmix hyperspectral images more intelligently, we propose an AI-powered evolvable neural architecture search method for HU, EvoNAS for short, to optimally determine the network architecture by the means of the evolutionary algorithm instead of gradient-based or reinforcement learning-based rewards. In EvoNAS, a supernet with all candidate architectures is first trained to learn the unmixing mapping in a self-supervised manner. The optimal network is then constructed by evaluating unmixing results of different architectures in the supernet. EvoNAS is capable of saving tremendous computational cost, since it inherits the weights of the pre-trained supernet and avoids training from scratch during the search phase. Experimental results conducted on two real hyperspectral datasets verify the effectiveness and superiority of the EvoNAS and show the huge potential of the NAS for HU.

With the ever-growing availability of different remote sensing (RS) products from both satellite and airborne platforms, simultaneous processing and interpretation of multimodal RS data have shown increasing significance in the RS field. Different resolutions, contexts, and sensors of multimodal RS data enable the identification and recognition of the materials lying on the earth's surface at a more accurate level by describing the same object from different points of the view. As a result, the topic on multimodal RS data fusion has gradually emerged as a hotspot research direction in recent years. This paper aims at presenting an overview of multimodal RS data fusion in several mainstream applications, which can be roughly categorized by 1) image pansharpening, 2) hyperspectral and multispectral image fusion, 3) multimodal feature learning, and (4) crossmodal feature learning. For each topic, we will briefly describe what is the to-be-addressed research problem related to multimodal RS data fusion and give the representative and state-of-the-art models from shallow to deep perspectives.

With the ever-growing availability of remote sensing (RS) data from either satellite or airborne sensors, simultaneous processing and analysis of multimodal data have been paid more and more attention by researchers in various RS-related applications. In this paper, we propose a multimodal convolutional neural network with an advanced cross-channel reconstruction module, called CCR-Net. As the name suggests, CCR-Net enables a more compact fusion of different RS data sources by the means of the reconstruction strategy across modalities that can mutually exchange information in a more effective way. Experiment are conducted on a widely-used dataset, including hyperspectral and Light Detection and Ranging (LiDAR) data, i.e., Houston2013, to verify the effectiveness and superiority of the proposed CCR - N et in comparison with several state-of-the-art baseline methods.

In the classification of hyperspectral image (HSI), there exists a common issue that the collected HSI data set is always contaminated by various noise (e.g., Gaussian, stripe, and deadline), degrading the classification results. To tackle this issue, we modify the 3-dimensional discrete wavelet transform (3DDWT) method by considering the noise effect on feature quality and propose an enhanced 3DDWT (E-3DDWT) approach to extract the feature and meanwhile alleviate the noise. Specifically, the proposed E-3DDWT method first applies classical 3DDWT method to the HSI data cube and thus can generate eight subcubes in each level. Then, the stripe noise is concentrated into several subcubes due to its spatial vertical property. Finally, we abandon these subcubes and obtain the feature cube by stacking the remaining ones. After acquiring the feature, we then adopt the convolutional neural network (CNN) model with an active learning strategy for classification since CNN has been verified to be a state-of-the-art feature extraction method for HSI classification, and active learning strategy can alleviate the insufficient labeled sample issue to some extent. In addition, we apply the Markov random field to enhance the final categorized results. Experiments on two synthetically striped data sets show that our proposed approach achieves better categorized results than other advanced methods.

Over the past decades, numerous methods have been proposed to solve the linear or nonlinear mixing problems in hyperspectral unmixing (HU). The existence of spectral variabilities and nonlinearity limits, to a great extent, the unmixing ability of most traditional approaches, particularly in complex scenes. In recent years, deep learning (DL) has been garnering increasing attention in nonlinear HU owing to its powerful learning and fitting ability. However, the DL-based methods tend to generate trivial unmixing results due to the lack of considering physically meaningful endmember information. To this end, we propose a novel siamese network, called the deep half-siamese network (Deep HSNet), for HU by fully considering diverse endmember properties extracted using different endmember extraction algorithms. Moreover, the proposed Deep HSNet, beyond the previous autoencoder-like architecture, adopts another subnetwork to learn the endmember information effectively to guide the unmixing process in a reasonable and accurate way. The experimental results conducted on the synthetic and real hyperspectral data sets validate the effectiveness and superiority of the Deep HSNet over several state-of-the-art unmixing algorithms.

Conventional nonlinear subspace learning techniques (e.g., manifold learning) usually introduce some drawbacks in explainability (explicit mapping) and cost effectiveness (linearization), generalization capability (out-of-sample), and representability (spatial-spectral discrimination). To overcome these shortcomings, a novel linearized subspace analysis technique with spatial-spectral manifold alignment is developed for a semisupervised hyperspectral dimensionality reduction (HDR), called joint and progressive subspace analysis (JPSA). The JPSA learns a high-level, semantically meaningful, joint spatial-spectral feature representation from hyperspectral (HS) data by: 1) jointly learning latent subspaces and a linear classifier to find an effective projection direction favorable for classification; 2) progressively searching several intermediate states of subspaces to approach an optimal mapping from the original space to a potential more discriminative subspace; and 3) spatially and spectrally aligning a manifold structure in each learned latent subspace in order to preserve the same or similar topological property between the compressed data and the original data. A simple but effective classifier, that is, nearest neighbor (NN), is explored as a potential application for validating the algorithm performance of different HDR approaches. Extensive experiments are conducted to demonstrate the superiority and effectiveness of the proposed JPSA on two widely used HS datasets: 1) Indian Pines (92.98%) and 2) the University of Houston (86.09%) in comparison with previous state-of-the-art HDR methods. The demo of this basic work (i.e., ECCV2018) is openly available at https://github.com/danfenghong/ECCV2018_J-Play.

Extensive attention has been widely paid to enhance the spatial resolution of hyperspectral (HS) images with the aid of multispectral (MS) images in remote sensing. However, the ability in the fusion of HS and MS images remains to be improved, particularly in large-scale scenes, due to the limited acquisition of HS images. Alternatively, we super-resolve MS images in the spectral domain by the means of partially overlapped HS images, yielding a novel and promising topic: spectral superresolution (SSR) of MS imagery. This is challenging and less investigated task due to its high ill-posedness in inverse imaging. To this end, we develop a simple but effective method, called joint sparse and low-rank learning (J-SLoL), to spectrally enhance MS images by jointly learning low-rank HS-MS dictionary pairs from overlapped regions. J-SLoL infers and recovers the unknown HS signals over a larger coverage by sparse coding on the learned dictionary pair. Furthermore, we validate the SSR performance on three HS-MS data sets (two for classification and one for unmixing) in terms of reconstruction, classification, and unmixing by comparing with several existing state-of-the-art baselines, showing the effectiveness and superiority of the proposed J-SLoL algorithm. Furthermore, the codes and data sets will be available at https://github.com/danfenghong/IEEE_TGRS_J-SLoL, contributing to the remote sensing (RS) community.

This article addresses the general problem of single-look multi-master SAR tomography. For this purpose, we establish the single-look multi-master data model, analyze its implications for the single and double scatterers, and propose a generic inversion framework. The core of this framework is the nonconvex sparse recovery, for which we develop two algorithms: one extends the conventional nonlinear least squares (NLS) to the single-look multi-master data model and the other is based on bi-convex relaxation and alternating minimization (BiCRAM). We provide two theorems for the objective function of the NLS subproblem, which lead to its analytic solution up to a constant phase angle in the 1-D case. We also report our findings from the experiments on different acceleration techniques for BiCRAM. The proposed algorithms are applied to a real TerraSAR-X data set and validated with the height ground truth made available by an SAR imaging geodesy and simulation framework. This shows empirically that the single-master approach, if applied to a single-look multi-master stack, can be insufficient for layover separation, and the multi-master approach can indeed perform slightly better (despite being computationally more expensive) even in the case of single scatterers. In addition, this article also sheds light on the special case of single-look bistatic SAR tomography, which is relevant for the current and future SAR missions such as TanDEM-X and Tandem-L.

Classification and identification of the materials lying over or beneath the earth's surface have long been a fundamental but challenging research topic in geoscience and remote sensing (RS), and have garnered a growing concern owing to the recent advancements of deep learning techniques. Although deep networks have been successfully applied in single-modality-dominated classification tasks, yet their performance inevitably meets the bottleneck in complex scenes that need to be finely classified, due to the limitation of information diversity. In this work, we provide a baseline solution to the aforementioned difficulty by developing a general multimodal deep learning (MDL) framework. In particular, we also investigate a special case of multi-modality learning (MML)-cross-modality learning (CML) that exists widely in RS image classification applications. By focusing on “what,” “where,” and “how” to fuse, we show different fusion strategies as well as how to train deep networks and build the network architecture. Specifically, five fusion architectures are introduced and developed, further being unified in our MDL framework. More significantly, our framework is not only limited to pixel-wise classification tasks but also applicable to spatial information modeling with convolutional neural networks (CNNs). To validate the effectiveness and superiority of the MDL framework, extensive experiments related to the settings of MML and CML are conducted on two different multimodal RS data sets. Furthermore, the codes and data sets will be available at https://github.com/danfenghong/IEEE_TGRS_MDL-RS, contributing to the RS community.

Convolutional neural networks (CNNs) have been attracting increasing attention in hyperspectral (HS) image classification due to their ability to capture spatial-spectral feature representations. Nevertheless, their ability in modeling relations between the samples remains limited. Beyond the limitations of grid sampling, graph convolutional networks (GCNs) have been recently proposed and successfully applied in irregular (or nongrid) data representation and analysis. In this article, we thoroughly investigate CNNs and GCNs (qualitatively and quantitatively) in terms of HS image classification. Due to the construction of the adjacency matrix on all the data, traditional GCNs usually suffer from a huge computational cost, particularly in large-scale remote sensing (RS) problems. To this end, we develop a new minibatch GCN (called miniGCN hereinafter), which allows to train large-scale GCNs in a minibatch fashion. More significantly, our miniGCN is capable of inferring out-of-sample data without retraining networks and improving classification performance. Furthermore, as CNNs and GCNs can extract different types of HS features, an intuitive solution to break the performance bottleneck of a single model is to fuse them. Since miniGCNs can perform batchwise network training (enabling the combination of CNNs and GCNs), we explore three fusion strategies: additive fusion, elementwise multiplicative fusion, and concatenation fusion to measure the obtained performance gain. Extensive experiments, conducted on three HS data sets, demonstrate the advantages of miniGCNs over GCNs and the superiority of the tested fusion strategies with regard to the single CNN or GCN models. The codes of this work will be available at https://github.com/danfenghong/IEEE_TGRS_GCN for the sake of reproducibility.

In this article, we have great interest in investigating a common but practical issue in remote sensing (RS)-can a limited amount of one information-rich (or high-quality) data, e.g., hyperspectral (HS) image, improve the performance of a classification task using a large amount of another information-poor (low-quality) data, e.g., multispectral (MS) image? This question leads to a typical cross-modality feature learning. However, classic cross-modality representation learning approaches, e.g., manifold alignment, remain limited in effectively and efficiently handling such problems that the data from high-quality modality are largely absent. For this reason, we propose a novel graph-induced aligned learning (GiAL) framework by 1) adaptively learning a unified graph (further yielding a Laplacian matrix) from the data in order to align multimodality data (MS-HS data) into a latent shared subspace; 2) simultaneously modeling two regression behaviors with respect to labels and pseudo-labels under a multitask learning paradigm; and 3) dramatically updating the pseudo-labels according to the learned graph and refeeding the latest pseudo-labels into model learning of the next round. In addition, an optimization framework based on the alternating direction method of multipliers (ADMMs) is devised to solve the proposed GiAL model. Extensive experiments are conducted on two MS-HS RS data sets, demonstrating the superiority of the proposed GiAL compared with several state-of-the-art methods.

This article addresses the problem of semantic segmentation with limited cross-modality data in large-scale urban scenes. Most prior works have attempted to address this issue by using multimodal deep neural networks (DNNs). However, their ability to effectively blending different properties across multimodalities and robustly learning representations from complex scenes remains limited, particularly in the absence of sufficient and well-annotated training images. This leads to a challenge related to cross-modality learning with multimodal DNNs. To this end, we introduce two novel plug-and-play units in the network: self-generative adversarial networks (GANs) module and mutual-GANs module, to learn perturbation-insensitive feature representations and to eliminate the gap between multimodalities, respectively, yielding more effective and robust information transfer. Furthermore, a patchwise progressive training strategy is devised to enable effective network learning with limited samples. We evaluate the proposed network on two multimodal (hyperspectral and multispectral) overhead image data sets and achieve a significant improvement in comparison with several state-of-the-art methods.

Due to the proliferation of large-scale remote-sensing (RS) archives with multiple annotations, multilabel RS scene classification and retrieval are becoming increasingly popular. Although some recent deep learning-based methods are able to achieve promising results in this context, the lack of research on how to learn embedding spaces under the multilabel assumption often makes these models unable to preserve complex semantic relations pervading aerial scenes, which is an important limitation in RS applications. To fill this gap, we propose a new graph relation network (GRN) for multilabel RS scene categorization. Our GRN is able to model the relations between samples (or scenes) by making use of a graph structure which is fed into network learning. For this purpose, we define a new loss function called scalable neighbor discriminative loss with binary cross entropy (SNDL-BCE) that is able to embed the graph structures through the networks more effectively. The proposed approach can guide deep learning techniques (such as convolutional neural networks) to a more discriminative metric space, where semantically similar RS scenes are closely embedded and dissimilar images are separated from a novel multilabel viewpoint. To achieve this goal, our GRN jointly maximizes a weighted leave-one-out K-nearest neighbors ( KNN) score in the training set, where the weight matrix describes the contributions of the nearest neighbors associated with each RS image on its class decision, and the likelihood of the class discrimination in the multilabel scenario. An extensive experimental comparison, conducted on three multilabel RS scene data archives, validates the effectiveness of the proposed GRN in terms of KNN classification and image retrieval. The codes of this article will be made publicly available for reproducible research in the community.

The total variation (TV) regularization has been widely used in various applications related to hyperspectral (HS) signal and image processing due to its potential in modeling the underlying smoothness of HS data. However, most existing TV norms usually tend to generate spatial oversmoothing or artifacts. To this end, we propose a novel l0- l1hybrid TV ( l0- l1HTV) regularization with the applications to HS mixed noise removal and compressed sensing (CS). More specifically, l0- l1HTV can be regarded as a globally and locally integrated TV regularizer, where the l0gradient constraint is incorporate into the l1spatial-spectral TV ( l1-SSTV). l1-SSTV is capable of exploiting the local structure information across both spatial and spectral domains, while the l0gradient can promote a globally spectral-spatial smoothness by directly controlling the number of nonzero gradients of HS images. This efficient combination considers more comprehensive prior knowledge of HS images, yielding sharper edge preservation and resolving the above drawbacks of existing pure TV norms. More significantly, l0- l1HTV can be easily injected into HS-related processing models, and an effective algorithm based on the alternating direction method of multipliers (ADMM) is developed to solve the optimization problems. Extensive experiments conducted on several HS data sets substantiate the superiority and effectiveness of the proposed method in comparison with many state-of-the-art methods.

Due to the limitations of hyperspectral imaging systems, hyperspectral imagery (HSI) often suffers from poor spatial resolution, thus hampering many applications of the imagery. Hyperspectral super resolution refers to fusing HSI and MSI to generate an image with both high spatial and high spectral resolutions. Recently, several new methods have been proposed to solve this fusion problem, and most of these methods assume that the prior information of the point spread function (PSF) and spectral response function (SRF) are known. However, in practice, this information is often limited or unavailable. In this work, an unsupervised deep learning-based fusion method-HyCoNet-that can solve the problems in HSI-MSI fusion without the prior PSF and SRF information is proposed. HyCoNet consists of three coupled autoencoder nets in which the HSI and MSI are unmixed into endmembers and abundances based on the linear unmixing model. Two special convolutional layers are designed to act as a bridge that coordinates with the three autoencoder nets, and the PSF and SRF parameters are learned adaptively in the two convolution layers during the training process. Furthermore, driven by the joint loss function, the proposed method is straightforward and easily implemented in an end-to-end training manner. The experiments performed in the study demonstrate that the proposed method performs well and produces robust results for different data sets and arbitrary PSFs and SRFs.

Interferometric phase restoration has been investigated for decades and most of the state-of-the-art methods have achieved promising performances for InSAR phase restoration. These methods generally follow the nonlocal filtering processing chain, aiming at circumventing the staircase effect and preserving the details of phase variations. In this article, we propose an alternative approach for InSAR phase restoration, that is, Complex Convolutional Sparse Coding (ComCSC) and its gradient regularized version. To the best of the authors' knowledge, this is the first time that we solve the InSAR phase restoration problem in a deconvolutional fashion. The proposed methods can not only suppress interferometric phase noise, but also avoid the staircase effect and preserve the details. Furthermore, they provide an insight into the elementary phase components for the interferometric phases. The experimental results on synthetic and realistic high- and medium-resolution data sets from TerraSAR-X StripMap and Sentinel-1 interferometric wide swath mode, respectively, show that our method outperforms those previous state-of-the-art methods based on nonlocal InSAR filters, particularly the state-of-the-art method: InSAR-BM3D. The source code of this article will be made publicly available for reproducible research inside the community.

This work presents a novel hyperspectral embedding technique by learning a deep regression network in an unsupervised fashion, which aims at reducing the computational complexity and storage-costing of traditional manifold embedding methods as well as improving the representation ability of spectral signatures effectively. The proposed method attempts to learn an explicit and unified nonlinear mapping from all patch-wise correspondences of original hyperspectral data and dimension-reduced products generated by some existing manifold learning approaches. This process can be well performed by means of a deep regression model. The learned model is not only capable of locally capturing the manifold structure of the whole hyperspectral image from densely patch-based random sampling but also better applicable to high-efficient out-of-sample inference. Experimental results conducted on the real hyperspectral data demonstrate the effectiveness and superiority of the proposed hyperspectral embedding technique.

Recently, spectral enhancement of multispectral imagery has attracted a growing interest in the remote sensing (RS) community. Without any prior knowledge, this task is highly ill-conditioned in inverse problems. To this end, we develop a simple but effective method, called locally linear reconstruction (LLR), to spectrally enhance the multispectral imagery (MSI) using partially overlapped hyperspectral data. LLR learns reconstruction coefficients of each pixel from the MSI and shares the same weights to recover the unknown hyperspectral signals over a larger coverage. We validate the performance of the proposed LLR on the real hyperspectral data in comparison with several state-of-the-art baselines, demonstrating its effectiveness and superiority.

Due to the ever-growing diversity of the data source, multimodality feature learning has attracted more and more attention. However, most of these methods are designed by jointly learning feature representation from multimodalities that exist in both training and test sets, yet they are less investigated in the absence of certain modality in the test phase. To this end, in this letter, we propose to learn a shared feature space across multimodalities in the training process. By this way, the out-of-sample from any of multimodalities can be directly projected onto the learned space for a more effective cross-modality representation. More significantly, the shared space is regarded as a latent subspace in our proposed method, which connects the original multimodal samples with label information to further improve the feature discrimination. Experiments are conducted on the multispectral-Light Detection and Ranging (LIDAR) and hyperspectral data set provided by the 2018 IEEE GRSS Data Fusion Contest to demonstrate the effectiveness and superiority of the proposed method in comparison with several popular baselines.

The semantic interpretation using point clouds, especially regarding light detection and ranging (LiDAR) point cloud classification, has attracted a growing interest in the fields of photogrammetry, remote sensing, and computer vision. In this letter, we aim at tackling a general and typical feature learning problem in 3-D point cloud classification- how to represent geometric features by structurally considering a point and its surroundings in a more effective and discriminative fashion? Recently, enormous efforts have been made to design the geometric features, yet it is less investigated to fully explore the potentials of the features. For that, there have been many filter-based studies proposed by selecting a subset of the whole feature space for better representing the local geometry structure. However, such a hard-threshold selection strategy inevitably suffers from information loss. In addition, the construction of the geometric features is relatively sensitive to the size of the neighborhood. To this end, we propose to extract multi-scaled feature representations and locally embed them into a low-dimensional and robust subspace where a more compact representation with the intrinsic structure preservation of the data is expected to be obtained, thereby further yielding a better classification performance. In our case, we apply a popular manifold learning approach, that is, locality-preserving projections, for the task of learning low-dimensional embedding. Experimental results conducted on one LiDAR point cloud data set provided by the 2018 IEEE Data Fusion Contest demonstrate the effectiveness of the proposed method in comparison with several commonly used state-of-the-art baselines.

Geospatial object detection (GOD) of remote sensing imagery has been attracting increasing interest in recent years, due to the rapid development in spaceborne imaging. Most of the previously proposed object detectors are very sensitive to object deformations, such as scaling and rotation. To this end, we propose a novel and efficient framework for GOD in this letter, called Fourier-based rotation-invariant feature boosting (FRIFB). A Fourier-based rotation-invariant feature is first generated in polar coordinate. Then, the extracted features can be further structurally refined using aggregate channel features. This leads to a faster feature computation and more robust feature representation, which is good fitting for the coming boosting learning. Finally, in the test phase, we achieve a fast pyramid feature extraction by estimating a scale factor instead of directly collecting all features from the image pyramid. Extensive experiments are conducted on two subsets of NWPU VHR-10 data set, demonstrating the superiority and effectiveness of the FRIFB compared to the previous state-of-the-art methods.

In this article, we propose an efficient and effective framework to fuse hyperspectral and light detection and ranging (LiDAR) data using two coupled convolutional neural networks (CNNs). One CNN is designed to learn spectral-spatial features from hyperspectral data, and the other one is used to capture the elevation information from LiDAR data. Both of them consist of three convolutional layers, and the last two convolutional layers are coupled together via a parameter-sharing strategy. In the fusion phase, feature-level and decision-level fusion methods are simultaneously used to integrate these heterogeneous features sufficiently. For the feature-level fusion, three different fusion strategies are evaluated, including the concatenation strategy, the maximization strategy, and the summation strategy. For the decision-level fusion, a weighted summation strategy is adopted, where the weights are determined by the classification accuracy of each output. The proposed model is evaluated on an urban data set acquired over Houston, USA, and a rural one captured over Trento, Italy. On the Houston data, our model can achieve a new record overall accuracy (OA) of 96.03%. On the Trento data, it achieves an OA of 99.12%. These results sufficiently certify the effectiveness of our proposed model.

So far, a large number of advanced techniques have been developed to enhance and extract the spatially semantic information in hyperspectral image processing and analysis. However, locally semantic change, such as scene composition, relative position between objects, spectral variability caused by illumination, atmospheric effects, and material mixture, has been less frequently investigated in modeling spatial information. Consequently, identifying the same materials from spatially different scenes or positions can be difficult. In this article, we propose a solution to address this issue by locally extracting invariant features from hyperspectral imagery (HSI) in both spatial and frequency domains, using a method called invariant attribute profiles (IAPs). IAPs extract the spatial invariant features by exploiting isotropic filter banks or convolutional kernels on HSI and spatial aggregation techniques (e.g., superpixel segmentation) in the Cartesian coordinate system. Furthermore, they model invariant behaviors (e.g., shift, rotation) by the means of a continuous histogram of oriented gradients constructed in a Fourier polar coordinate. This yields a combinatorial representation of spatial-frequency invariant features with application to HSI classification. Extensive experiments conducted on three promising hyperspectral data sets (Houston2013 and Houston2018) to demonstrate the superiority and effectiveness of the proposed IAP method in comparison with several state-of-the-art profile-related techniques. The codes will be available from the website: https://sites.google.com/view/danfeng-hong/data-code.

Recently, enormous efforts have been made to improve the performance of the linear or nonlinear mixing model for hyperspectral unmixing, yet their ability to handle spectral variability and extract physically meaningful endmembers remains limited. Based on the powerful learning ability of deep learning, we propose a weakly-supervised unmixing network, called WU-Net, to break the bottleneck. Beyond the autoencoder-like architecture, WU-Net learns an additional network from the pure or nearly-pure endmembers to correct the weights of another unmixing network towards a more accurate and interpretable unmixing solution, thus yielding a two-stream deep network. Experimental results conducted on two different datasets, one fully artificial simulation dataset and one simulated EnMap dataset generated from a real HyMap dataset, demonstrate the effectiveness and superiority of WU-Net over several state-of-the-art algorithms.

Hyperspectral images and polarimetric synthetic aperture radar (PolSAR) data are two important data sources, yet they barely appear under the same scope, even though multi-modal data fusion is attracting more and more attention. To our best knowledge, this paper investigates for the first time semi-supervised manifold alignment (SSMA) for the fusion of the hyperspectral image and PolSAR data. The SSMA searches a latent space where different data sources are aligned, which is accomplished by using the label information and the topological structure of the data. This paper is the first attempt to apply topological data analysis (TDA), a recent mathematic sub-field of data analysis, in remote sensing. It aims to reveal relevant information from the shape of a data in its feature space, and has been proven powerful in medicine. The paper also proposes a novel algorithm, MAPPER-regularized manifold alignment, which embeds the TDA into a semi-supervised manifold alignment for the fusion of the hyper-spectral image and PolSAR data. The proposed algorithm exhibits superior performance in fusing a simulated EnMAP data set and a Sentinel-1 data set for an image of Berlin.

In this paper, we propose to extract the multi-scaled and rotation-insensitive deep features to address the issues of object multi-solutions and rotations in geospatial object detection. To this end, we develop a novel object detection framework where a rotation-insensitive convolution neural network is applied for extracting multi-scaled and direction-insensitive feature representation and then the learned features can be fed into the ensemble classifier learning with fast feature pyramid. Such a non-end-to-end learning strategy intuitively reduces the computational cost without the additional performance loss, yielding an effective and efficient light-weight object detection framework. Experimental results conducted on the NWPU VHR-10 dataset demonstrate that the proposed framework outperforms several state-of-the-art baselines.

With the breakthrough of the spatial resolution of optical remote sensing images at the sub-meter level and the explosive development of deep learning, geospatial object detection has achieved a growing interest in remote sensing community. However, labeling large training datasets in object level is still an expensive and tedious procedure. This might lead to the poor model generalization and degraded network learning ability. To this end, a weakly-supervised deep network (WSDN) is developed for geospatial object detection by applying a digital surface model (DSM)-aided auto-labeling and a pre-trained network learned from the task-independent dataset. Experimental results conducted on the stereo aerial imagery of a large camping site are performed to demonstrate that the proposed WSDN yields better detection results, with 62.78% precision and 55.13% recall.

By considering the spectral signature as a sequence, recurrent neural networks (RNNs) have been successfully used to learn discriminative features from hyperspectral images (HSIs) recently. However, most of these models only input the whole spectral bands into RNNs directly, which may not fully explore the specific properties of HSIs. In this paper, we propose a cascaded RNN model using gated recurrent units to explore the redundant and complementary information of HSIs. It mainly consists of two RNN layers. The first RNN layer is used to eliminate redundant information between adjacent spectral bands, while the second RNN layer aims to learn the complementary information from nonadjacent spectral bands. To improve the discriminative ability of the learned features, we design two strategies for the proposed model. Besides, considering the rich spatial information contained in HSIs, we further extend the proposed model to its spectral-spatial counterpart by incorporating some convolutional layers. To test the effectiveness of our proposed models, we conduct experiments on two widely used HSIs. The experimental results show that our proposed models can achieve better results than the compared models.

With a large amount of open satellite multispectral (MS) imagery (e.g., Sentinel-2 and Landsat-8), considerable attention has been paid to global MS land cover classification. However, its limited spectral information hinders further improving the classification performance. Hyperspectral imaging enables discrimination between spectrally similar classes but its swath width from space is narrow compared to MS ones. To achieve accurate land cover classification over a large coverage, we propose a cross-modality feature learning framework, called common subspace learning (CoSpace), by jointly considering subspace learning and supervised classification. By locally aligning the manifold structure of the two modalities, CoSpace linearly learns a shared latent subspace from hyperspectral-MS (HS-MS) correspondences. The MS out-of-samples can be then projected into the subspace, which are expected to take advantages of rich spectral information of the corresponding hyperspectral data used for learning, and thus leads to a better classification. Extensive experiments on two simulated HS-MS data sets (University of Houston and Chikusei), where HS-MS data sets have tradeoffs between coverage and spectral resolution, are performed to demonstrate the superiority and effectiveness of the proposed method in comparison with previous state-of-the-art methods.

Multi-modal data fusion has recently been shown promise in classification tasks in remote sensing. Optical data and radar data, two important yet intrinsically different data sources, are attracting more and more attention for potential data fusion. It is already widely known that a machine learning-based methodology often yields excellent performance. However, the methodology relies on a large training set, which is very expensive to achieve in remote sensing. The semi-supervised manifold alignment (SSMA), a multi-modal data fusion algorithm, has been designed to amplify the impact of an existing training set by linking labeled data to unlabeled data via unsupervised techniques. In this paper, we explore the potential of SSMA in fusing optical data and polarimetric synthetic aperture radar (SAR) data, which are multi-sensory data sources. Furthermore, we propose a MAPPER-induced manifold alignment (MIMA) for the semi-supervised fusion of multi-sensory data sources. Our proposed method unites SSMA with MAPPER, which is developed from the emerging topological data analysis (TDA) field. To the best of our knowledge, this is the first time that SSMA has been applied on fusing optical data and SAR data, and also the first time that TDA has been applied in remote sensing. The conventional SSMA derives a topological structure using k-nearest neighbor (kNN), while MIMA employs MAPPER, which considers the field knowledge and derives a novel topological structure through the spectral clustering in a data-driven fashion. The experimental results on data fusion with respect to land cover land use classification and local climate zone classification suggest superior performance of MIMA.

Spatiotemporal image fusion is considered as a promising way to provide Earth observations with both high spatial resolution and frequent coverage, and recently, learning-based solutions have been receiving broad attention. However, these algorithms treating spatiotemporal fusion as a single image super-resolution problem, generally suffers from the significant spatial information loss in coarse images, due to the large upscaling factors in real applications. To address this issue, in this paper, we exploit temporal information in fine image sequences and solve the spatiotemporal fusion problem with a two-stream convolutional neural network called StfNet. The novelty of this paper is twofold. First, considering the temporal dependence among image sequences, we incorporate the fine image acquired at the neighboring date to super-resolve the coarse image at the prediction date. In this way, our network predicts a fine image not only from the structural similarity between coarse and fine image pairs but also by exploiting abundant texture information in the available neighboring fine images. Second, instead of estimating each output fine image independently, we consider the temporal relations among time-series images and formulate a temporal constraint. This temporal constraint aiming to guarantee the uniqueness of the fusion result and encourages temporal consistent predictions in learning and thus leads to more realistic final results. We evaluate the performance of the StfNet using two actual data sets of Landsat-Moderate Resolution Imaging Spectroradiometer (MODIS) acquisitions, and both visual and quantitative evaluations demonstrate that our algorithm achieves state-of-the-art performance.

With the rapid development of spaceborne imaging techniques, object detection in optical remote sensing imagery has drawn much attention in recent decades. While many advanced works have been developed with powerful learning algorithms, the incomplete feature representation still cannot meet the demand for effectively and efficiently handling image deformations, particularly objective scaling and rotation. To this end, we propose a novel object detection framework, called Optical Remote Sensing Imagery detector (ORSIm detector), integrating diverse channel features extraction, feature learning, fast image pyramid matching, and boosting strategy. An ORSIm detector adopts a novel spatial-frequency channel feature (SFCF) by jointly considering the rotation-invariant channel features constructed in the frequency domain and the original spatial channel features (e.g., color channel and gradient magnitude). Subsequently, we refine SFCF using learning-based strategy in order to obtain the high-level or semantically meaningful features. In the test phase, we achieve a fast and coarsely scaled channel computation by mathematically estimating a scaling factor in the image domain. Extensive experimental results conducted on the two different airborne data sets are performed to demonstrate the superiority and effectiveness in comparison with the previous state-of-the-art methods.

Hyperspectral imagery collected from airborne or satellite sources inevitably suffers from spectral variability, making it difficult for spectral unmixing to accurately estimate abundance maps. The classical unmixing model, the linear mixing model (LMM), generally fails to handle this sticky issue effectively. To this end, we propose a novel spectral mixture model, called the augmented linear mixing model (ALMM), to address spectral variability by applying a data-driven learning strategy in inverse problems of hyperspectral unmixing. The proposed approach models the main spectral variability (i.e., scaling factors) generated by variations in illumination or typography separately by means of the endmember dictionary. It then models other spectral variabilities caused by environmental conditions (e.g., local temperature and humidity, atmospheric effects) and instrumental configurations (e.g., sensor noise), as well as material nonlinear mixing effects, by introducing a spectral variability dictionary. To effectively run the data-driven learning strategy, we also propose a reasonable prior knowledge for the spectral variability dictionary, whose atoms are assumed to be low-coherent with spectral signatures of endmembers, which leads to a well-known low-coherence dictionary learning problem. Thus, a dictionary learning technique is embedded in the framework of spectral unmixing so that the algorithm can learn the spectral variability dictionary and estimate the abundance maps simultaneously. Extensive experiments on synthetic and real datasets are performed to demonstrate the superiority and effectiveness of the proposed method in comparison with previous state-of-the-art methods.

This paper presents a novel spectral mixture model to address spectral variability in inverse problems of hyperspectral unmixing. Based on the linear mixture model (LMM), our model introduces a spectral variability dictionary to account for any residuals that cannot be explained by the LMM. Atoms in the dictionary are assumed to be low-coherent with spectral signatures of endmembers. A dictionary learning technique is proposed to learn the spectral variability dictionary while solving unmixing problems simultaneously. Experimental results on synthetic and real datasets demonstrate that the performance of the proposed method is superior to state-of-the-art methods.

In this paper, we propose an efficient identification algorithm for RFID systems based on EPC C1 Gen2 RFID standard1. Specifically, the proposed anti-collision algorithm is based on the observation of sub-frame during an identification process, and makes effective use of idle and collision statistics to accurately estimate the tag backlog and determine the proper frame size for the next inventory round. Simulation results are supplemented to demonstrate the advantages of the proposed algorithm in achieving time and computation efficiency.

Manifold learning has been successfully applied to hyperspectral dimensionality reduction to embed nonlinear and nonconvex manifolds in the data. However, dimensionality reduction by manifold learning is sensitive to non-uniform data distribution and the selection of neighbors. To address the two issues to some extents, in this work a new manifold framework based on locality linear embedding (LLE), namely local normalization and local feature selection (LNLFS), is proposed. Classification is explored as a potential application to validate the proposed algorithm. Classification accuracy using data obtained using different dimensionality reduction methods is evaluated and compared, while applying two kinds of strategies for selecting the training and test samples: random sampling and region-based sampling. Experimental results show the classification accuracy obtained with LNLFS is superior to state-of-the-art dimensionality reduction methods.