In recent years, Large Vision-Language Models (LVLMs) have significantly advanced multimodal tasks. However, their inference requires intensive processing of numerous visual tokens and incurs substantial computational overhead. Existing methods typically compress visual tokens either at the input stage or in early model layers, ignoring variations across tasks and depths. To address these limitations, we introduce TOP-RL, a Task-Optimized Progressive token pruning framework based on Reinforcement Learning. TOP-RL formulates visual token pruning as a multi-stage Markov Decision Process (MDP). It employs an agent trained with dense and fine-grained reward signals to progressively generate differentiable binary masks. This enables TOP-RL to adaptively select crucial visual tokens tailored to each task, effectively balancing accuracy and computational efficiency. Extensive experiments on leading multimodal datasets and advanced LVLMs validate that TOP-RL effectively learns task-optimized pruning policies, significantly boosting inference efficiency while preserving robust performance. For instance, LLaVA-NeXT equipped with TOP-RL achieves a 1.9x speedup in inference time and a 9.3x reduction in FLOPs, with 96% performance preserved.

Change detection is essential in Earth observation, yet current models heavily rely on large-scale annotated datasets. Generative models offer a promising alternative by synthesizing training data, but generating temporally coherent image pairs with realistic, semantically meaningful changes remains a significant challenge. Existing approaches typically simulate changes by generating pre- and post-change label maps using either heuristic rules (e.g., copy-pasting) or text prompts. However, the former offers limited change diversity, while the latter often fails to maintain spatial consistency between image pairs. We observe that the noise space of diffusion models encodes strong generative capacity and spatial controllability: localized perturbations in the noise can yield meaningful, interpretable changes in corresponding image regions. Motivated by this, we propose Noise2Change, a framework for simulating change directly in the noise domain. The key idea is to manipulate the semantic composition of the initial noise sampled from the noise domain, such that the diffusion process generates structurally consistent pre- and post-change images reflecting realistic transformations. Since the unperturbed noise is shared between both images, the resulting pairs exhibit strong temporal alignment and semantic coherence, effectively addressing the trade-off between realism and consistency. Concretely, we employ a discrete diffusion model to extract high-level semantics from the initial noise. Guided by these semantics, we introduce a change simulation strategy that optimizes the noise to encode intended changes. The modified noise is then used to drive the diffusion process, yielding pre- and post-change label maps with natural structural transitions. These maps are passed through a unified framework for image generation and label refinement, producing highly aligned image-label pairs. Our framework supports diverse change types across a wide range of scenarios. Extensive experiments on multiple change detection tasks demonstrate that our method achieves superior performance compared to existing generative approaches.

Large-scale images play a crucial role in geospatial surveying, as they cover an extensively broad view and diverse objects. Due to computational limitations, existing methods rely on generating large-scale images in patches. However, the lack of global guidance in these methods often leads to significant logical errors among different patches. To address this issue, we propose a Global Consistency Diffusion model (Glob-Diffusion) for large-scale image generation. The core idea is to utilize the global consistency of small-scale images to guide the generation of large-scale images. Specifically, we introduce a Hierarchical Distributed Guidance (HDG) module that extracts patch prompts with different semantic hierarchies from small-scale images, distributedly embedding them into the generation of large-scale images to maintain global consistency across various regions. In addition, we further design a Region Guided Adapter (RGA) that dynamically optimizes the guidance strength of patch prompts by comparing differences across generated regions, effectively improving the realism of large-scale images. Our method demonstrates remarkable visual synthesis results across various natural scenes, effectively preserving global consistency in large-scale images, and also significantly enhancing the generation quality of large-scale remote sensing images. Code will be available at https://github.com/kyh433/Glob-Diffusion.

Heterogeneous remote sensing image change detection (HRSICD) seeks to identify surface changes by comparing images captured at different times. However, CD faces significant challenges due to heterogeneity arising from varying sensor types and imaging conditions. Recently, powerful vision-language models like CLIP have emerged, with strong semantic decoding abilities. Opening new possibilities for using linguistic information as an auxiliary in visual tasks, potentially driving breakthroughs in HCD. Capitalizing on this prospect, we investigate graph learning with vision-language features and introduce LEVPC, the first language-enhanced visual perception consistency framework for HCD. First, we create a mutual information-guided graph aggregation module. Specifically, it builds modality-invariant structured relationships among visual nodes by using language features as connecting bridges, providing a consistent foundation for comparing changes. To reduce modeling bias from heterogeneity, language is used as an anchor to aggregate features, ensuring a unified expression of visual representations. In summary, language guides the generation and aggregation of multiple subgraphs from visual inputs, ultimately building robust representations of structural relationships within a shared semantic space. Moreover, a change semantic compensation module is introduced, which analyses the change intensity between bi-temporal data from a vision-language perspective. And then adds change-related semantic descriptions for salient change regions, enhancing the expressiveness of visual change features. Experiments on multiple datasets validate the superior performance of LEVPC in HCD, achieving an average increase of 2.6% in Kappa. The code will be publicly available at https://github.com/sylXIDIAN/LEVPC.

Gradient sparsification (GS) is an effective method for reducing communication overhead in distributed training. For the first time, we introduce the concept of Multi-dimensional information into GS and propose a new gradient sparsification method named Multi-dimensional information-aware Gradient sparsification (MultiGS), which achieves high compression ratio with negligible accuracy loss and is applicable to mainstream network architectures. MultiGS reconstructs the layer-wise gradient by combining the high-frequency components of the local gradient and the low-frequency components of the sparsified global gradient that effectively addresses the issue of stale gradients and alleviates model bifurcation. Through the convergence proof of MultiGS for smooth non-convex problems and comparison with momentum SGD in convergence speed, we show that such new perspective approach is theoretically reasonable and practically effective. As validated with several mainstream model families (i.e., ResNets, VGGNet, LSTM, Vision Transformer, and Large Language Models), our MultiGS shows better accuracy over previous GS methods. Moreover, empirical results show that when a sufficient number of training nodes are available, MultiGS accelerates the distributed training by more than 3×, which is better than existing sparsification method.

Hyperspectral anomaly detection (HAD) is widely used in Earth observation and deep space exploration. A major challenge for HAD is the complex background of the input hyperspectral images (HSIs), resulting in anomalies confused in the background. On the other hand, most existing HAD methods require training a separate model for each HSI, resulting in poor generalization in practical applications. This paper starts the first attempt to study a new and generalizable background learning problem without labeled samples. We present a novel solution BSDM (background suppression diffusion model) for HAD, which can simultaneously learn latent background distributions and generalize to different datasets for suppressing complex background. It is featured in three aspects: (1) For the complex background of HSIs, we design pseudo-background noise and learn the potential background distribution in it with a diffusion model (DM). (2) For the generalizability problem, we apply a statistical offset module so that the BSDM adapts to datasets of different domains without labeling samples. (3) For achieving background suppression, we innovatively improve the inference process of DM by feeding the original HSIs into the denoising network, which removes the background as noise. Our work paves a new background suppression way for HAD that can improve HAD performance without the prerequisite of manually labeled data. Assessments and generalization experiments of four HAD methods on several real HSI datasets demonstrate the above three unique properties of the proposed method. Our project is available at https://github.com/majitao-xd/BSDM-HAD.

RGB-X multimodal vision tasks present a highly promising approach to enhancing model performance in complex visual conditions. Existing multimodal frameworks are based on either the symmetric parallel network of feature fusion or the shared network of input fusion. However, parallel networks suffer from uncontrollable parameters and imbalanced optimization across modal branches, while shared networks often lead to a lack of diversity in gradient optimization. To address these challenges, we propose the LoRA-driven Multimodal Extractor (LoME), following a comprehensive analysis of existing multimodal frameworks. The low-rank properties of modal adapters for LoME ensure controllable growth in model parameters as the number of modalities increases. The dynamic parameter fusion between adapters and the shared feature extractor decouples gradient optimization directions, effectively mitigating imbalances caused by multimodal data biases while preserving complementary features. Moreover, we employ a training strategy based on dynamic rank allocation to reduce computational overhead and enhance modal diversity expression. We validate the effectiveness and generalizability of LoME across three multimodal vision tasks. LoME achieves superior performance compared to previous state-of-the-art methods on multiple datasets. For example, on the DroneVehicle dataset, our method achieves a 10.4% improvement in accuracy compared to the SOTA method, while the parameter overhead is reduced to 23% of the previous network (44.63M). The code has been open-sourced at https://github.com/zyszxhy/LoME.

While change detection (CD) is crucial for tracking dynamic changes on the Earth's surface, it faces substantial challenges in real-world settings caused by seasonal variations and sensor-related interference. Current CD models often suffer performance degradation under such conditions, mainly due to two key challenges. First, most existing CD datasets lack sufficient temporal and environmental diversity, as they are typically collected over constrained time spans. This limits the models' ability to generalize across varying conditions. Second, many CD methods are heavily data-driven and rely on simplified assumptions, leading to models that are not adequately designed to handle the complex, heterogeneous nature of real-world scenarios. Together, these challenges restrict the robustness and practical applicability of current CD approaches. To overcome these challenges, we make the following contributions in this paper: 1) Regarding data diversity, we construct a comprehensive benchmark by introducing five typical perturbations (fog, snow, motion blur, Gaussian noise, and impulse noise) into three classical CD datasets and supplementing them with a real-world seasonal dataset, resulting in 75 interference-rich scenarios. This enables a systematic evaluation under diverse real-world conditions, revealing that such perturbations induce severe distribution shifts across both temporal phases and hierarchical network layers, leading to substantial performance degradation in existing models. 2) Algorithmically, we propose Real-CD, a novel method specifically designed to address distribution shifts in real-world CD. The core of Real-CD is to leverage bi-temporal correlations to perform adaptive distribution alignment across hierarchical layers and temporal phases. Specifically, we propose the Distribution Shifts Alleviation Module (DSAM) to correct distribution shifts. The DSAM captures bi-temporal differences and similarities to formulate temporal-specific adjustment strategies for each LayerNorm (LN) layer. To stabilize the optimization of DSAM, we propose the Distribution Consistency Optimization Strategy (DCOS), which introduces a flip-based auxiliary task that encourages the model to maintain distributional consistency under complex bi-temporal disturbances. Consequently, our method outperforms other state-of-the-art approaches and achieves the best performance on the proposed dataset. Our datasets and code implementation will be available at https://github.com/fangyee-ISALAB/Real-CD.

Defect detection in multimedia data plays a pivotal role in industrial manufacturing. However, existing methods are primarily designed for closed-world scenarios and can only identify defect classes in the training data, limiting their ability to effectively detect unknown class defects that arise during production. To address this critical limitation, we propose a novel approach by introducing industrial defect open set recognition (IDOSR), which overcomes the challenge of recognizing unknown defect classes. Furthermore, to tackle the issues of limited training samples and subtle inter-class differences in IDOSR, we present a high-frequency feature enhancement open set recognition (HFFE-OSR) method. Specifically, HFFE-OSR employs a high-frequency structural feature fusion enhancement strategy to meticulously extract and fuse defect-related high-frequency structural features. This enables the network to comprehensively learn defect target representations even under limited training samples, resulting in robust feature extraction for known classes, thereby improving the discriminability between known and unknown classes and addressing the difficulty of distinguishing between them. Additionally, a class mutual information constraint strategy is introduced to measure and reduce the mutual information among defect features from different classes. This ensures the independence of defect features across known classes, further enhancing their discriminability and significantly improving recognition performance for known classes. Extensive experiments demonstrate that the proposed method significantly outperforms state-of-the-art OSR methods on ID-OSD and MVTec datasets, achieving improvements of at least 7% in accuracy (ACC), 22% in F1 score, and 10% in AUROC, highlighting the effectiveness of our approach in industrial defect detection.

Zero-shot generalized anomaly detection (ZGAD) plays a critical role in industrial automation and health screening. Recent studies have shown that ZGAD methods built on visual-language models (VLMs) like CLIP have excellent cross-domain detection performance. Different from other computer vision tasks, ZGAD needs to jointly optimize both image-level anomaly classification and pixel-level anomaly segmentation tasks for determining whether an image contains anomalies and detecting anomalous parts of an image, respectively, this leads to different granularity of the tasks. However, existing methods ignore this problem, processing these two tasks with one set of broad text prompts used to describe the whole image. This limits CLIP to align textual features with pixel-level visual features and impairs anomaly segmentation performance. Therefore, for precise visual-text alignment, in this paper we propose a novel fine-grained text prompts generation strategy. We then apply the broad text prompts and the generated fine-grained text prompts for visual-textual alignment in classification and segmentation tasks, respectively, accurately capturing normal and anomalous instances in images. We also introduce the Text Prompt Shunt (TPS) model, which performs joint learning by reconstruction the complementary and dependency relationships between the two tasks to enhance anomaly detection performance. This enables our method to focus on fine-grained segmentation of anomalous targets while ensuring accurate anomaly classification, and achieve pixel-level comprehensible CLIP for the first time in the ZGAD task. Extensive experiments on 13 real-world anomaly detection datasets demonstrate that TPS achieves superior ZGAD performance across highly diverse datasets from industrial and medical domains.

Distributed machine learning (DML) is promising for training large models on large datasets. In DML, multiple workers collaborate on the training of neural networks, significantly reducing the time required for neural network training. The efficiency of DML is heavily influenced by communication, making it crucial to balance the trade-off between communication cost and model performance in current research. Local methods are excellent at reducing communication costs, yet face degradation in accuracy and generalizability. Indeed, global knowledge is valuable for improving performance in local methods. However, the theoretical analysis of global knowledge validity is lacking, and global knowledge can currently only be used in the global aggregation of local methods due to communication limitations and staleness. To this end, in this paper, we establish the mechanism of global knowledge guidance and propose Adaptive Global Knowledge Guided Distributed Stochastic Gradient Descent (AdaGK-SGD) to extend the guidance of global knowledge to the whole distributed training process without any additional communication. Specifically, we define the maximum lifetime of global knowledge based on the mechanism, and establish a correlation between the maximum lifetime and the validity of global knowledge to circumvent the adverse effects of global knowledge staleness. The Maximum Lifetime of Global Knowledge module of our algorithm can be applied separately to other algorithms. In addition, considering the application, we provide a straightforward and efficient strategy for achieving the maximum lifetime adaptive setting. We establish the convergence rate of AdaGK-SGD for convex and non-convex scenarios. Numerically, we find that AdaGK-SGD can significantly improve the accuracy and generalizability of distributed algorithms compared with existing methods.

Federated Learning (FL) is an emerging direction in distributed machine learning that enables jointly training a model without sharing the data. However, as the size of datasets grows exponentially, computational costs of FL increase. In this paper, we propose the first Coreset Selection criterion for Federated Learning (FedCS) by exploring the Distance Contrast (DC) in feature space. Our FedCS is inspired by the discovery that DC can indicate the intrinsic properties inherent to samples regardless of the networks. Based on the observation, we develop a method that is mathematically formulated to prune samples with high DC. The principle behind our pruning is that high DC samples either contain less information or represent rare extreme cases, thus removal of them can enhance the aggregation performance. Besides, we experimentally show that samples with low DC usually contain substantial information and reflect the common features of samples within their classes, such that they are suitable for constructing coreset. With only two time of linear-logarithmic complexity operation, FedCS leads to significant improvements over the methods using whole dataset in terms of computational costs, with similar accuracies. For example, on the CIFAR-10 dataset with Dirichlet coefficient α = 0.1, FedCS achieves 58.88% accuracy using only 44% of the entire dataset, whereas other methods require twice the data volume as FedCS for same performance.

Multimodal image fusion and semantic segmentation are critical for autonomous driving. Despite advancements, current models often struggle with segmenting densely packed elements due to a lack of comprehensive fusion features for guidance during training. While the Segment Anything Model (SAM) allows precise control during fine-tuning through its flexible prompting encoder, its potential remains largely unexplored in the context of multimodal segmentation for natural images. In this paper, we introduce SAM into multimodal image segmentation for the first time, proposing a novel framework that combines Latent Space Token Generation (LSTG) and Fusion Mask Prompting (FMP) modules. This approach transforms the training methodology for multimodal segmentation from a traditional black-box approach to a controllable, prompt-based mechanism. Specifically, we obtain latent space features for both modalities through vector quantization and embed them into a cross-attention-based inter-domain fusion module to establish long-range dependencies between modalities. We then use these comprehensive fusion features as prompts to guide precise pixel-level segmentation. Extensive experiments on multiple public datasets demonstrate that our method significantly outperforms SAM and SAM2 in multimodal autonomous driving scenarios, achieving an average improvement of 4.1% over the state-of-the-art method in segmentation mIoU, and the performance is also optimized in other multi-modal visual scenes.

Weakly-Supervised Semantic Segmentation (WSSS) aims to train segmentation models by weak labels, which is receiving significant attention due to its low annotation cost. Existing approaches focus on generating pseudo labels for supervision while largely ignoring to leverage the inherent semantic correlation among different pseudo labels. We observe that pseudo-labeled pixels that are close to each other in the feature space are more likely to share the same class, and those closer to the distribution centers tend to have higher confidence. Motivated by this, we propose to model the underlying label distributions and employ cross-label constraints to generate more accurate pseudo labels. In this paper, we develop a unified WSSS framework named Adaptive Gaussian Mixtures Model, which leverages a GMM to model the label distributions. Specifically, we calculate the feature distribution centers of pseudo-labeled pixels and build the GMM by measuring the distance between the centers and each pseudo-labeled pixel. Then, we introduce an Online Expectation-Maximization (OEM) algorithm and a novel maximization loss to optimize the GMM adaptively, aiming to learn more discriminative decision boundaries between different class-wise Gaussian mixtures. Based on the label distributions, we leverage the GMM to generate high-quality pseudo labels for more reliable supervision. Our framework is capable of solving different forms of weak labels: image-level labels, points, scribbles, blocks, and bounding-boxes. Extensive experiments on PASCAL, COCO, Cityscapes, and ADE20 K datasets demonstrate that our framework can effectively provide more reliable supervision and outperform the state-of-the-art methods under all settings.

Recently, unsupervised domain adaptation (UDA) techniques have been introduced for cross-scene hyperspectral image (HSI) classification tasks. These techniques aim to transfer knowledge from labeled source scenes to unlabeled target scenes, addressing the issue of limited supervisory information. However, most UDA methods fail to analyze the variability of domain shifts from different source samples to target ones, thus limiting the domain adaptation effect. To this end, this paper develops a consistency-aware customized learning (CACL) approach for cross-scene HSI classification. Overall, domain-level and class-level distribution alignment are designed separately. The former is implemented by adversarial training between the feature extractor and the domain discriminator. For the latter, the spectral-spatial prototypes of the source and target domains are first dynamically extracted, respectively. Then the prototype-based labels are assigned to the target domain samples, according to the cosine similarity-based cross-domain category prototype matching strategy. Considering that the consistency of the prototype-based labels with the predicted pseudo-labels reflects the degree of domain shifts of the target samples, a customized learning strategy is developed via inter-/intra-domain contrastive learning. With the joint domain-level and fine-grained class-level distribution alignment, the supervised information from the source domain is better migrated to the target domain, improving classification performance. Comprehensive experiments on two single-modal and one multi-modal cross-scene datasets demonstrate the effectiveness of the proposed algorithm.

Mamba architecture achieves the same performance as attention mechanisms with linear complexity, leading to significant progress in remote sensing land cover classification. However, existing Mamba methods rarely leverage the representational complementarity and consistency between different modalities, resulting in challenges such as incomplete fusion. To address these issues, we propose Semi-Mamba, a novel semi-supervised framework specifically designed for high-dimensional multi-modal data fusion. We introduce the Mamba Cross-Modality Fusion Module, which enables cross-modal learning of temporal features through state-space model interactions and smooth integration of input matrices, enhancing the fusion of richer feature representations. Additionally, to tackle the inherent difficulty of acquiring pixel-level annotations in remote sensing datasets, we introduce a multi-modal semi-supervised mechanism. This mechanism utilizes cross-modal supervision between different modalities to maximize data utilization and improve learning efficiency. It effectively enables joint training on both labeled and unlabeled data without relying on pseudo-labels. We integrate these innovations into a unified end-to-end framework. Compared to state-of-the-art CNN and Transformerbased architectures, our framework shows a significant improvement of over 3.12%, setting a new benchmark for semi-supervised multi-modal data fusion. The code has open sourced at https://github.com/LDXDU/Semi_Mamba_RS.

Model compression methods such as pruning and quantization have been proposed to facilitate the deployment of convolutional neural networks (CNNs) on resource-constrained devices. Existing methods aim to combine the two for simultaneous improvement in compression ratio and runtime efficiency. However, most of the joint methods adopt linear tandem structures. Due to the lack of a unified framework, different optimization directions result in suboptimal solutions, especially when the compression ratio is extremely high. In this paper, we propose a novel adaptive curvature-based compression (ACC) method, which achieves a dual-depth unified joint optimization of pruning and quantization. In the first depth, we unify the pruning and quantization criteria using mean curvature, which leverages the discrete nature of image data and the continuum theory of differential geometry. In the second depth, we replace the traditional training process in the joint pruning-quantization method with curvature-aware knowledge distillation (CKD), unifying the two-stage approach into a simple but powerful parallel step. Our method is effective and interpretable by utilizing inherent properties to promote the understanding of information distribution and the importance of feature maps. Extensive experiments on multiple advanced benchmarks and diverse downstream task datasets have validated the superiority and generalizability of our ACC. Notably, we can achieve a 1.05% Top-1 accuracy improvement over the baseline under an extreme compression ratio of 454.55×, outperforming existing state-of-the-art (SOTA) methods.

Inspired by the human visual system (HVS), no-reference image quality assessment (NR-IQA) has made significant progress without relying on perfect reference images. The HVS is primarily influenced by the combined effects of representational information with different receptive fields and attribute categories when capturing subjective perceived quality. However, existing methods only roughly or partially utilize representations of multi-dimensional information. Furthermore, current NR-IQA methods either rely on convolutional neural networks (CNNs) with limited local perception or depend on the computational complexity of vision transformers (ViTs). To make up for the shortcomings of these two architectures, an emerging visual state space model (VMamba) is introduced. Motivated by this, this paper presents a NR-IQA method via VIsual State space model with Graph-based feature Aggregation (VISGA). Specifically, we utilize a plain, pre-training-free, and feature-enhanced VMamba as the backbone. To align with the perceptual mechanisms of the HVS by effectively using features with different dimensional information, a graph convolutional network-based multi-receptive field and multi-level aggregation module is designed to deeply explore the correlations and interactions of multi-dimensional representations. Additionally, we propose a gated local enhancement module with patch-wise perception to enhance the local perception of VMamba. Extensive experiments conducted on seven databases demonstrate that VISGA achieves outstanding performance. Notably, our model remains state-of-the-art when training with very few parameters. The code is released athttps://github.com/xirihao/VISGA.

Change Detection (CD) is a crucial and challenging task in remote sensing observations. Despite the remarkable progress driven by deep learning in remote sensing change detection, several challenges remain regarding global information representation and efficient interaction. The traditional Siamese network structure, which extracts features from bitemporal images using a weight-sharing network and generates a change map, but often neglects phase interaction information between images. Additionally, multi-scale feature fusion methods frequently use FPN-like structures, leading to lossy cross-layer information transmission and hindering the effective utilization of features. To address these issues, we propose a multi-scale interaction fusion network (MIFNet) that fuses bitemporal features at an early stage, using deep supervision techniques to guide early fusion features in obtaining abundant semantic representation of changes, also we construct a dual complementary attention module (DCA) to capture temporal information. Furthermore, we introduce a collection-allocation fusion mechanism, which is different from previous layer-by-layer fusion methods since it collects global information and embeds features at different levels to achieve effective cross-layer information transmission and promote global semantic feature representation. Extensive experiments demonstrate that our method achieves competitive results on the LEVIR-CD+ dataset, outperforming other advanced methods on both the LEVIR-CD and SYSU-CD datasets, with F1 improved by 0.96% and 0.61%, respectively, compared to the most advanced models.

Cross-scene hyperspectral image (HSI) classification presents significant challenges due to domain shifts, which amplify epistemic uncertainty and lead to substantial performance drops in unseen scenes. While evidence deep learning (EDL) has shown promise in modeling uncertainty, existing methods fall short, as they do not explicitly account for the epistemic uncertainty arising from spatial-spectral feature interactions. To address these challenges, we propose the spectral-spatial evidence deep learning for cross-scene hyperspectral image classification (HyperEDL) framework, which introduces the spatial-spectral multiorder aggregation module (SS-Moga). This module effectively captures and adaptively encodes multiorder contextual interactions from both spatial and spectral perspectives. By combining multiorder contextual encoding with spatial-spectral confidence, our approach fully aggregates multiorder evidence to mitigate epistemic uncertainty arising from knowledge gaps between seen and unseen scenes. Specifically, it uses Dirichlet distribution to capture correlation between spatial-spectral knowledge about different scenes, which can be generalized to unseen scenes. Extensive experiments on three benchmark datasets demonstrate that HyperEDL outperforms state-of-the-art methods, showcasing its effectiveness and strong generalization ability.

Hyperspectral image (HSI) classification suffers from severe catastrophic forgetting in exemplar-free lifelong learning, where models must continuously learn new land cover categories without accessing historical training samples. This challenge persists due to high-dimensional spectral-volumetric complexity and cross-task spectral drift, which current methods inadequately address. We propose HyperSC, a novel framework that synergizes spectral-consistent auxiliary samples synthesis with stability-plasticity fused learning. The framework consists of three key components: a Spectral Consistency Model Inversion (SCMI) module, a Spectral Progressive Enhancement (SPE) module, and a Fusion Distillation Learning (FDL) module. The SCMI module synthesizes class-conditional auxiliary samples through spectral moment matching, in which the mean and variance of each spectral band are constrained to match class-specific real historical data distributions, thereby achieving spectral consistency. The SPE module injects class-specific Gaussian noise and applies momentum-based updating to enhance sample diversity while preserving spectral fidelity. The FDL module jointly trains on fused real and auxiliary samples by coordinating cross-entropy classification, output-layer knowledge distillation, and intermediate-layer feature alignment, thereby enabling plasticity for new class learning while maintaining stability against catastrophic forgetting for previous tasks. Extensive experiments on three HSI datasets (Indian Pines, Houston, Salinas) demonstrate HyperSC’s superiority compared to previous exemplar-free lifelong learning methods. The code is available at https://github.com/lzlsxs/hypersc.

Hyperspectral image (HSI) classification models suffer from a phenomenon known as catastrophic forgetting, which refers to the sharp decline in performance on previously learned tasks after learning a new one when continuously acquiring new knowledge from a sequence of tasks. In recent years, some lifelong learning approaches have been proposed for HSI classification. Despite some progress, the challenge of catastrophic forgetting in lifelong learning remains significant and unresolved. In this article, we propose a novel lifelong learning framework for HSI classification, which is based on exemplar replay and cross-spectral–spatial feature knowledge distillation (KD), termed HyperKD. Specifically, the proposed framework incorporates a min-max cross-selection (MMCS) module tailored to HSI characteristics with a cross-spectral-spatial knowledge distillation (CSSKD) module. The MMCS module selects the most representative or diverse samples as exemplars from previous tasks for replay. Additionally, the CSSKD module not only transfers the prediction logit distribution from the previous network to the current network but also transfers the spectral-spatial feature distribution via cross-network KD, without directly assessing the similarity of feature distributions, thereby retaining more knowledge and mitigating forgetting. Through experiments conducted on a series of tasks, including the Pavia, Indian Pines, Salinas, and Houston datasets, our approach demonstrates superior performance compared to previous lifelong learning methods for HSI classification, effectively mitigating catastrophic forgetting. The code implementation of our approach will be publicly available athttps://github.com/lzlsxs/hyperkd.

Object detection algorithms are pivotal components of UAV imaging systems, extensively employed in complex fields. However, images captured by high-mobility UAVs often suffer from motion blur cases, which significantly impedes the performance of advanced object detection algorithms. To address these challenges, we propose an innovative object detection algorithm specifically designed for blurry images, named dual-stream restoration embedding blur-feature fusion network (DREB-Net). First, DREB-Net addresses the particularities of blurry image object detection problem by incorporating a blurry image restoration auxiliary branch (BRAB) during the training phase. Second, it fuses the extracted shallow features via multilevel attention-guided feature fusion (MAGFF) module, to extract richer features. Here, the MAGFF module comprises local attention modules and global attention modules, which assign different weights to the branches. Then, during the inference phase, the deep feature extraction of the BRAB can be removed to reduce computational complexity and improve detection speed. In loss function, a combined loss of mean squared error (MSE) and SSIM is added to the BRAB to restore blurry images. Finally, DREB-Net introduces fast Fourier transform in the early stages of feature extraction, via a learnable frequency domain amplitude modulation module (LFAMM), to adjust feature amplitude and enhance feature processing capability. Compared to the baseline, DREB-Net achieved an approximate 7% increase in both mAP50 and mAR50 across two experimental datasets. Experimental results indicate that DREB-Net can still effectively perform object detection tasks under motion blur in captured images, showcasing excellent performance and broad application prospects. Our source code will be available athttps://github.com/EEIC-Lab/DREB-Net.git.

Interpreting remote sensing (RS) images plays a crucial role in numerous applications such as environmental monitoring, urban planning, agricultural management and disaster assessment. Nevertheless, remote sensing data is frequently dispersed among various organizations. Privacy concerns and data-sharing limitations make it difficult to utilize large-scale datasets within a centralized training framework. Federated learning (FL) provides a promising approach by facilitating collaborative model training across decentralized data sources, eliminating the need for data centralization. However, the application of FL in RS scenarios is challenging due to the resource-limited edge nodes cannot meet the high demand for computation and memory resources to train deep learning models. Neural network lightweighting techniques have the potential to enhance model efficiency, but existing methods still present significant challenges, such as reliance on initial training of dense models during lightweighting and potential performance degradation after lightweighting. To address these challenges, we propose RS-IML, a novel FL framework for RS Image Interpretation based on Intrinsic Mask Learning without training dense models. RS-IML comprises three key components. First, we introduce the intrinsic dimension of objective landscape that the neural network is projected onto a low-dimensional subnetwork to lightweight neural networks without training dense models. Second, we propose intrinsic dimension parameter averaging to aggregate inconsistent local intrinsic models while suppressing the adverse effects of non-intrinsic parameters between local models. Third, we fine-tune the specific parameters of local intrinsic models to mitigate global intrinsic noise for better performance. Extensive experiments demonstrate the effectiveness of our proposed RS-IML. It achieves a significant improvement in model efficiency during lightweighting compared to existing methods while obtaining superior accuracy.

Hyperspectral image change detection (HSI-CD) benefits from HSIs with continuous spectral bands, which uniquely enables the analysis of more subtle changes. Existing methods have achieved desirable performance relying on multi-temporal homogenous HSIs over the same region, which is generally difficult to obtain in real scenes. HSI-RGB multimodal CD overcomes the constraint of limited HSI availability by incorporating another temporal RGB data, and the combination of advantages within different modalities enhances the robustness of detection results. Nevertheless, due to the different imaging mechanisms between two modalities, existing HSI CD methods cannot be directly applied. In this paper, we propose a cycle translation-based collaborative training (co-training) for HSI-RGB multimodal CD, which achieves cross-modal mutual guidance to collaboratively learn complementary difference information from diverse modalities for identifying changes. Specifically, a cross-modal guided CycleGAN-based image translation module is designed to implement bi-directional image translation, which mitigates modal difference and enables the extraction of information related to land cover changes. Then, a spatial-spectral interactive co-training CD module is proposed to achieve iterative interaction between cross-modal information, which jointly extracts the multimodal difference features to generate the final results. The proposed method outperforms several leading CD methods in extensive experiments carried out on both real and synthetic datasets. In addition, a new public HSI-RGB multimodal dataset along with our code are available at https://github.com/Jiahuiqu/CT2Net.

Dense pixel-wise labeling of large-scale remote sensing images (RSI) is very time-consuming, while sparse labels (i.e., points, scribbles, or blocks) can be an efficient way to reduce labeling costs. Most existing sparse label-based methods adopt only one type of label for image segmentation, which cannot reflect the complex land covers in the RSI for training the model, thus leading to inferior segmentation performance. We observe that land covers with different shapes and complexity can be optimally represented by different sparse labels. Inspired by this observation, we propose a novel sparse labeling framework, termed Hybrid Sparse Labeling (HSLabeling), for large-scale RSI segmentation. Our HSLabeling can adaptively select the optimal hybrid sparse labels for different land covers, according to labeling cost and segmentation contribution of different sparse labels. Specifically, we first propose a label segmentation contribution information estimation module that estimates the information of different sparse labels according to the diversity and shape of land covers. After that, we propose an Optimal Hybrid Labeling Strategy (OHLS) to assign optimal types of labels for different land covers. In the OHLS, label assignment is formulated as an optimization problem that trades off label segmentation contribution information and labeling cost. We employ the greedy algorithm to efficiently solve the optimization problem and adaptively assign labels for varied land covers. Extensive experiments on three large-scale RSI datasets have demonstrated that our HSLabeling achieves almost fully supervised performance with extremely low labeling costs. In addition, compared with the single type sparse label, HSLabeling can also utilize much lower labeling costs to obtain the same performance. The source code is available at https://github.com/linjiaxing99/HSLabeling.

Fast and reliable interpretation of high-dimensional hyperspectral images (HSIs) can provide great support to remote sensing-based Earth observations. Targets of interest in HSI can be detected using deep neural networks (DNNs) for background learning on an acquired image where the occurrence probability of background samples is much greater than that of targets, accounting for more than 95% of the whole scene. However, there is an increasing gap between theory and feasible application, because of the contradiction between massive hyperspectral data and resource-limited Internet of Things (IoT)/edge device hardware like satellite. To facilitate the deployment of hyperspectral target detection (HTD) in an edge computing environment, we introduce distributed background learning-a decentralized deep learning approach to meet the computing requirements of exploding high-dimensional data and larger DNNs. To address the communication bottleneck caused by gradient exchange during distributed learning, the proposed gradient compression solution, named gradient compression via centroid (GCC), uniquely compresses the most replaceable gradients with redundant information, thereby reducing communication overhead while maintaining accuracy. To illustrate the feasibility of the proposed method, we test it over two very large hyperspectral datasets with a total size of about 3.2 gigabytes (GBs) on a distributed system based on Ring All-reduce. We show that HTD based on distributed background learning outperforms those developed on a single node in terms of speed. Besides, the GCC compresses 50% gradients with only 0.01% loss of target detection accuracy to greatly reduce the communication overhead, surpassing existing gradient compression methods. It is expected that this framework will accelerate the introduction of distributed training on IoT/edge devices.

Gradient sparsification and quantization offer a promising prospect to alleviate the communication overhead problem in distributed learning. However, direct combination of the two results in suboptimal solutions, due to the fact that sparsification and quantization haven't been learned together. In this paper, we propose Joint Sparsification-Quantization (JointSQ) inspired by the discovery that sparsification can be treated as 0-bit quantization, regardless of architectures. Specifically, we mathematically formu-late JointSQ as a mixed-precision quantization problem, expanding the solution space. It can be solved by the designed MCKP-Greedy algorithm. Theoretical analysis demon-strates the minimal compression noise of JointSQ, and ex-tensive experiments on various network architectures, including CNN, RNN, and Transformer, also validate this point. Under the introduction of computation overhead consistent with or even lower than previous methods, JointSQ achieves a compression ratio of 1000× on different models while maintaining near-lossless accuracy and brings 1.4× to 2.9× speedup over existing methods.

Recently, the Vision Transformer (ViT) becomes a promising model for building extraction from remote sensing images. However, most of these methods ignore the misalignment issue caused by the semantic inconsistency between different layers. To settle this issue, we propose a semantic alignment network (SANet) for building extraction. First, a dense prediction network with the encoder-decoder architecture is developed. The encoder is constituted by stacking multiple ViT blocks to extract multi-level features. Then, the semantic alignment module (SAM) is designed as the decoder to align and fuse low-level and high-level features, by use of semantic flow field estimation. The semantic flow assesses discrepancy between features of adjacent layers to find a flow field that guide the alignment between two-layer feature maps effectively. By this manner, the semantic gap can be narrowed. Experimental results on the two well-known datasets demonstrate the effectiveness of the proposed method.

The fusion of visible and infrared images aims to produce high-quality fusion images with rich textures and salient target information. Existing methods lack interactivity and flexibility in the execution of fusion. It is unfeasible to express the requirements to modify the fusion effect, and the different regions in the source images are treated equally across the identical fusion model, which causes fusion homogenization and low distinction. Besides, their pre-defined fusion strategies invariably lead to monotonous effects, which are insufficiently comprehensive. They fail to adequately consider data credibility, scene illumination, and noise degradation inherent in the source information. To address these issues, we propose the Te xt-driven and Region-aware Flexible visible and infrared image fusion, termed as TeRF. On the one hand, we propose a flexible image fusion framework with multiple large language and vision models, which facilitates the visual-text interaction. On the other hand, we aggregate comprehensive fine-tuning paradigms for the different fusion requirements to build a unified fine-tuning pipeline. It allows the linguistic selection of the regions and effects, yielding visually appealing fusion outcomes. Extensive experiments demonstrate the competitiveness of our method both qualitatively and quantitatively compared to existing state-of-the-art methods. Our code is publicly available at https://github.com/Baixuzx7/TeRF.

With the rapid development of imaging sensor technology in the field of remote sensing, multi-modal remote sensing data fusion has emerged as a crucial research direction for land cover classification tasks. While diffusion models have made great progress in generative models and image classification tasks, existing models primarily focus on single-modality and single-client control, that is, the diffusion process is driven by a single modal in a single computing node. To facilitate the secure fusion of heterogeneous data from clients, it is necessary to enable distributed multi-modal control, such as merging the hyperspectral data of organization A and the LiDAR data of organization B privately on each base station client. In this study, we propose a multi-modal collaborative diffusion federated learning framework called FedDiff. Our framework establishes a dual-branch diffusion model feature extraction setup, where the two modal data are inputted into separate branches of the encoder. Our key insight is that diffusion models driven by different modalities are inherently complementary in terms of potential denoising steps on which bilateral connections can be built. Considering the challenge of private and efficient communication between multiple clients, we embed the diffusion model into the federated learning communication structure, and introduce a lightweight communication module. Qualitative and quantitative experiments validate the superiority of our framework in terms of image quality and conditional consistency. To the best of our knowledge, this is the first instance of deploying a diffusion model into a federated learning framework, achieving optimal both privacy protection and performance for heterogeneous data. Our FedDiff surpasses existing methods in terms of performance on three multi-modal datasets, achieving a classification average accuracy of 96.77% while reducing the communication cost.

Various network compression methods, such as pruning and quantization, have been proposed to synergistically reduce resource requirements. However, existing joint compression works are based on black-box optimization and do not interpret the interaction mechanism between these two compression techniques, leading to a slow and unstable convergence of compression strategy. To address this issue, we present Markov-PQ, the first interpretable pruning-quantization co-compression framework using a Markov Chain. In Markov-PQ, the joint strategy search is modeled as a Markov Chain and decoupled with Bayes Rule into pruning and quantization strategy searching. Specifically, the quantization state accounts for the co-compression state from the last time and is updated by a learnable transition probability matrix. To ensure differentiability, we design a forward-hard and backward-soft quantization. The pruning state is influenced not only by the last co-compression state but also by the concurrent quantization state. In addition, to perceive the current layer-wise bit sensitivity and alleviate the long-tail problem, a complexity-aware regularizer is devised to re-evaluate the filter importance. Extensive experiments demonstrate the superiority of Markov-PQ. For example, with an accuracy loss of only 0.33%, we can achieve a$56.12\times $acceleration for ResNet-18 on ImageNet2012.

Remote sensing (RS) scene classification based on deep neural networks (DNNs) has recently drawn remarkable attention. However, the DNNs contain a great number of parameters and require a huge amount of computational costs, which are hard to deploy on edge devices such as onboard embedded systems. To address this issue, in this paper, we propose a target-aware knowledge distillation (TAKD) method for RS scene classification. By considering the characteristics among the target and background regions of the RS images, the TAKD can adaptively distill the knowledge from the teacher model to create a lightweight student model. Specifically, we first introduce a target extraction module that utilizes heatmaps to highlight target regions on the teacher’s feature maps. Next, we propose an adaptive fusion module that aggregates these heatmaps to capture objects with varying scales. Finally, we design a target-aware loss that enables the transfer of knowledge in the target regions from the teacher model to the student model, greatly reducing background disturbance. Our distillation scheme that does not require extra learning parameters is both simple and effective, significantly improving the accuracy of the student model without any additional computational or resource costs. Our experiments on three benchmark datasets demonstrate that our proposed TAKD outperforms the existing state-of-the-art distillation methods.

Recently, there have been significant advancements in Hyperspectral Image (HSI) classification methods employing Transformer architectures. However, these methods, while extracting spectral-spatial features, may introduce irrelevant spatial information that interferes with HSI classification. To address this issue, this paper proposes a Spectral Query Spatial Transformer (SQSFormer) framework. The proposed framework utilizes the center pixel (i.e., pixel to be classified) to adaptively query relevant spatial information from neighboring pixels, thereby preserving spectral features while reducing the introduction of irrelevant spatial information. Specifically, this paper introduces a Rotation-Invariant Position Embedding module to integrate random central rotation and center relative position embedding, mitigating the interference of absolute position and orientation information on spatial feature extraction. Moreover, a Spectral-Spatial Center Attention module is designed to enable the network to focus on the center pixel by adaptively extracting spatial features from neighboring pixels at multiple scales. The pivotal characteristic of the proposed framework achieves adaptive spectral-spatial information fusion using the Spectral Query Spatial paradigm, reducing the introduction of irrelevant information and effectively improving classification performance. Experimental results on multiple public datasets demonstrate that our framework outperforms previous state-of-the-art methods. For the sake of reproducibility, the source code of SQSFormer will be publicly available at https://github.com/chenning0115/SQSFormer.

Recently, hyperspectral image (HSI) classification methods based on Transformers have developed rapidly. However, these methods still face challenges in handling the widely varying scales and diverse spatial distribution patterns commonly found in HSIs. To address these issues, this article proposes a simple, yet novel HSI classification framework named the spectral view attention Transformer (SVAFormer). Built on the Transformer mechanism, this framework enhances the integration of spectral and spatial features by allowing the spectral token, corresponding to the pixel to be classified, to access spatial neighborhood information from multiple perspectives and levels. Specifically, the framework employs random masking techniques to provide spectral tokens with spatial neighborhood information from different viewpoints, enabling the model to handle diverse land-cover distribution patterns. Additionally, the framework introduces a spectral token-aware pooling layer between adjacent Transformer blocks, which preserves the central role of spectral tokens while progressively expanding the spatial scale represented by each token. This reduces the Transformer’s focus on spatially fragmented information and enables spectral tokens to concentrate on spatial neighborhood information at various levels and scales. The key characteristic of this framework is its ability to effectively handle land-cover features of different scales and shapes by strengthening the fusion of spectral and spatial characteristics. Experimental results on multiple public datasets demonstrate that our framework outperforms previous state-of-the-art methods. For the sake of reproducibility, the source code of SVAFormer will be publicly available athttps://github.com/chenning0115/SVAFormer.

The acquisition of a substantial volume of precisely dense pixel-annotated samples plays a crucial role in the effective training of deep learning-based change detection models. Nevertheless, in real-world scenarios, pairwise labeling of massive bitemporal remote sensing images is often laborious and time-consuming, resulting in the lack of labeled samples. In this article, we propose a novel point-based weakly supervised learning approach, called as the consistency-aligned regional growth network (CARGNet), for remote sensing change detection. Unlike pixel-level labels, point labels are easy to label and usually sparse, which leads to a lack of boundary information, making it difficult for the model to accurately capture the details of the changed objects. Therefore, learning directly from them may mislead the training of the network. To address these problems, we introduce a point-based changed regional growth (PCRG) module and consistency alignment (CA) constraint into CARGNet, which breaks the limitation of point labels in losing important target details. Specifically, our CARGNet contains two branches: a base decoder branch and an expanded decoder branch. First, we utilize the PCRG module to generate the expanded annotations from the point annotations. Then, the base decoder is supervised by the original point annotations, while the expanded decoder is supervised by the expanded annotations. Finally, the CA constraint is thereby achieved by minimizing the discrepancy between the predictions from both the base and the expanded decoders, which greatly improves the performance of the model. Experimental results on LEVIR-CD-Point and DSIFN-CD-Point datasets demonstrate that our proposed CARGNet can achieve highly competitive results compared with state-of-the-art fully-supervised methods. Code and datasets are available athttps://github.com/Wanderlust717/CARGNet.

Building extraction is a challenging research direction in remote sensing image (RSI) interpretation. Due to the fact that a building has not only its own local structures but also similar architectural styles with other buildings located in a global area (e.g., street or community), fusing local and global features becomes a promising way to improve performance of building extraction. Focused on this, we propose a new complementarity-aware local-global feature fusion network (CLGFF-Net) by integrating a convolutional branch and a Transformer branch. The two branches respectively capture local patterns and global long-range dependencies of RSIs, thereby leading to highly complementary features. To dig out the implicit complementary information for fusion, we develop a complementarity-aware fusion module (CFM) which separates shared features (SFs) and distinct features (DFs) between two branches, by building a commonalities analysis path and two difference analysis paths. Meanwhile, to make sure the similarity of SFs and dissimilarity of DFs, a triplet loss function is designed to enforce the distances between SFs to be near and DFs to be far. By this way, complementary information can be explicitly included in DFs and is adaptively exchanged between two branches for fusion. Besides, since multilayer features in each branch generally convey different-level semantic information, a multi-layer fusion scheme (MLFS) is designed to fuse them by introducing cross-layer connections and gate mechanism. By coupling CFMs with MLFS, the abilities in characterizing local and global context information, as well as different-level semantic information, can be fully exploited for better mapping of complicated building objects. Experimental results demonstrate the effectiveness of our proposed method.

Current methods for salient object detection in optical remote sensing images (RSI-SOD) adhere strictly to the conventional supervised train-test paradigm, where models remain fixed after training and are directly applied to test samples. However, this paradigm faces significant challenges in adapting to test-time images due to the inherent variability in remote sensing scenes. Salient objects exhibit considerable differences in size, type, and topology across RSIs, complicating accurate localization in unseen test images. Moreover, the acquisition of RSIs is highly susceptible to atmospheric conditions, often leading to degraded image quality and a notable domain shift between training and testing phases. In this work, we explore test-time model adaptation for RSI-SOD and introduce a novel multitask collaboration approach to tackle these challenges. Our approach integrates a self-supervised auxiliary task, specifically image reconstruction, with the primary supervised task of saliency prediction to achieve collaborative learning. This is accomplished through an architecture that comprises a shared feature encoder and two distinct task-specific decoders. Most importantly, the self-supervised image reconstruction task optimizes model parameters using unlabeled test-time images, allowing adaptation to test distributions and enabling flexibly scene-dependent representation learning. In addition, we design a cross-task modulation module (CMM) positioned between the task-specific decoders, which fully exploits intertask correlations to enhance the adjustment of saliency representations. Extensive experimental evaluations confirm the superiority of our method across three widely used RSI-SOD benchmarks and validate the robustness of our proposed test-time adaptation strategy against diverse types of RSI corruptions.

Deep learning (DL) has demonstrated remarkable performance in the classification of hyperspectral images (HSIs) by leveraging its powerful ability to automatically learn deep spectral–spatial features over the years. Nevertheless, the limited supervisory signals along with a vast number of parameters in deep models still pose critical challenges when utilizing a restricted number of samples for training deep networks. To better handle this issue, this article proposes an end-to-end framework called guided group contrastive learning (GGCL) that adaptively integrates unsupervised information into a supervised contrastive learning framework. The proposed method employs a similarity-guided module that measures the spectral–spatial similarity of unsupervised samples based on supervised signals and effectively groups them. Then, the similarity signals of both supervised and unsupervised data are combined with contrastive learning to achieve intragroup feature aggregation and intergroup feature separation with guided group contrastive loss (GGCLoss). The pivotal characteristic of the proposed method lies in the end-to-end incorporation of unsupervised information with supervised signals for contrastive learning. Experiments on three public HSI datasets demonstrate that the proposed method can achieve better performance than existing state-of-the-art (SOTA) methods. For ease of reproducibility, the code of the proposed GGCL will be publicly available athttps://github.com/fanerlight/GGCL_HSI.

Multi-Satellite, multi-modality in-orbit fusion is a challenging task as it explores the fusion representation of complex high-dimensional data under limited computational resources. Deep neural networks can reveal the underlying distribution of multimodal remote sensing data, but the in-orbit fusion of multimodal data is more difficult because of the limitations of different sensor imaging characteristics, especially when the multimodal data follow nonindependent identically distribution (Non-IID) distributions. To address this problem while maintaining classification performance, this article proposes a manifold-driven multi-modality fusion framework, FedFusion, which randomly samples local data on each client to jointly estimate the prominent manifold structure of shallow features of each client and explicitly compresses the feature matrices into a low-rank subspace through cascading and additive approaches, which is used as the feature input of the subsequent classifier. Considering the physical space limitations of the satellite constellation, we developed a multimodal federated learning (FL) module designed specifically for manifold data in a deep latent space. This module achieves iterative updating of the subnetwork parameters of each client through global weighted averaging, constructing a framework that can represent compact representations of each client. The proposed framework surpasses existing methods in terms of performance on three multimodal datasets, achieving a classification average accuracy of 94.35% while compressing communication costs by a factor of 4. Furthermore, extensive numerical evaluations of real-world satellite images were conducted on the orbiting edge computing architecture based on Jetson TX2 industrial modules, which demonstrated that FedFusion significantly reduced training time by 48.4 min (15.18%) while optimizing accuracy. The codes will be available at:https://github.com/LDXDU/FedFusion.

Transformers have significantly advanced hyperspectral image (HSI) classification through their proficiency in modeling long sequences. However, the high dimensionality of HSIs poses a particular challenge for Transformers due to their quadratic computational complexity. In natural language processing, state-space models (SSMs) such as Mamba hold great promise for handling long sequence tasks with significantly reduced computational overhead. However, the original Mamba lacks consideration for the spectral and spatial information inherent in HSIs. Inspired by this, we propose the HyperMamba, a novel spectral-spatial adaptive Mamba for HSI classification. The core idea of HyperMamba involves adaptively scanning spatial neighborhood pixels and dynamically enhancing spectral bands for spectral scanning based on acquired spatial neighborhood information. Specifically, HyperMamba consists of two core modules: the spatial neighborhood adaptive scanning (SNAS) module and the spectral adaptive enhancement scanning (SAES) module. Initially, the SNAS module analyzes the spectral characteristics of classified pixels, adaptively selecting the optimal neighborhood for spatial scanning by balancing spatial neighborhood information and local spatial structure. Subsequently, the SAES module dynamically enhances the spectral features of classified pixels using neighborhood spectral information and conducts spectral scanning. Finally, the spectral features of the target pixels are fed into a single fully connected layer classifier, achieving high-precision HSI classification. Extensive experiments demonstrate the effectiveness of HyperMamba, surpassing state-of-the-art methods across three widely used HSI datasets. The code will be available athttps://github.com/chiangliu/HyperMamba.

As a newly emerging advance in deep generative models, diffusion models have achieved state-of-the-art results in many fields, including computer vision, natural language processing, and molecule design. The remote sensing (RS) community has also noticed the powerful ability of diffusion models and quickly applied them to a variety of tasks for image processing. Given the rapid increase in research on diffusion models in the field of RS, it is necessary to conduct a comprehensive review of existing diffusion model-based RS papers, to help researchers recognize the potential of diffusion models and provide some directions for further exploration. Specifically, this article first introduces the theoretical background of diffusion models, and then systematically reviews the applications of diffusion models in RS, including image generation, enhancement, and interpretation. Finally, the limitations of existing RS diffusion models and worthy research directions for further exploration are discussed and summarized.

The spectral super-resolution (SpeSR) from multispectral images (MSIs) to hyperspectral images (HSIs) can bring rich spectral information. The deep learning-based methods have demonstrated their powerful ability for the SpeSR task, which requires the paired HSI/MSI to train the model. However, HSIs and MSIs are always obtained at different times and under different imaging conditions, covering different areas. To address this issue, in this paper, a framework named BliEstGAN based on the generative adversarial network (GAN) is proposed to estimate the spectral resolution degradation between unpaired HSIs and MSIs that can be used for the blind SpeSR. Specifically, each MSI imaging sensor has its own unique spectral sampling process, which can be modeled as a spectral degradation from its paired HSI. Different spectral degradations can be discriminated by the deep model. Therefore, the generator of the GAN is used to estimate the spectral degradation from HSIs to MSIs, and the discriminator of the GAN is adopted to distinguish whether the estimated and real spectral degradation are similar. The large difference in spatial resolution between MSIs and HSIs makes them easy to discriminate against. Therefore, smooth hyperspectral and multispectral patches are extracted from HSIs and MSIs to eliminate this difference in spatial resolution. Furthermore, according to the imaging sensor mechanism, some special regularization terms are designed for the generator to guarantee its correct convergence. Finally, the estimated spectral resolution degradation can be adopted to generate HSI/MSI pairs for the supervised learning-based SpeSR methods. Experimental results demonstrate the effectiveness of the proposed method.

In long-term Earth observation, change detection (CD) is a crucial and intricate task with applications spanning diverse fields, including land resource planning and natural disaster monitoring. Most existing CD approaches typically output segmentation results in raster format. However, raster format results suffer from higher memory usage, poorer shape accuracy, magnified distortions, and challenges in topological editing. To address the issues of raster format, we propose a novel end-to-end change vectorization network (CVNet), which is the first attempt to extract changes using vector format. The CVNet directly learns the vector components of changed objects and uses them to construct vectors. Specifically, since the vectorization of CD faces the inherent imbalance between changed and unchanged samples, we first introduce the Change-Collector to collect the changed regions and combine them into more compact samples. Next, the vector components learning model (VCLM) is introduced to capture the fundamental components for constructing the vectors, including change maps, junction positions, and segmentation masks. Finally, the changed instances obtained from the masks are used to divide and connect junctions to generate the vector output. To verify the effectiveness of the proposed framework, we construct two building change vectorization datasets by modifying the WHU-CD and LEVIR-CD benchmarks. Experimental results demonstrate that the CVNet outperforms the existing postprocess vectorization methods in terms of the visual effect and all evaluation metrics. The dataset and source code will be made publicly available athttps://github.com/yyyyll0ss/CVNet.

Efficient extraction of spectral sequences and geospatial information is crucial in hyperspectral image (HSI) classification. Recurrent neural networks (RNNs) and Transformers excel in capturing long-range spectral features, while convolutional neural networks (CNNs) excel in aggregating spatial information through convolutional kernels. However, RNNs and Transformers suffer from low-computational efficiency, and CNNs have limitations in perceiving global contextual information. To address these issues, this article proposes GraphMamba—an efficient graph structure learning vision Mamba for HSI classification. Specifically, GraphMamba is a novel hyperspectral information processing paradigm that preserves spatial-spectral features by constructing spatial-spectral cubes and employs a linear spectral encoder to enhance the operability of subsequent tasks. The core components of GraphMamba include the HyperMamba module, which enhances computational efficiency, and the SpatialGCN module, designed for adaptive spatial context awareness. The HyperMamba mitigates clutter interference by employing a global mask (GM) and introduces a parallel training and inference architecture to alleviate computational bottlenecks. Meanwhile, the SpatialGCN utilizes weighted multihop aggregation (WMA) for spatial encoding, emphasizing highly correlated spatial structural features. This approach enables flexible aggregation of contextual information while minimizing spatial noise interference. Notably, the encoding modules of the proposed GraphMamba architecture are both flexible and scalable, providing a novel approach for the joint mining of spatial-spectral information in hyperspectral images. Extensive experiments were conducted on three different scales of real HSI datasets. When compared with state-of-the-art classification methods, GraphMamba demonstrated superior performance. The core code will be released athttps://github.com/ahappyyang/GraphMamba.

With the rapid development of remote sensing imaging technology, change detection (CD) with heterogeneous images has become a hot topic in the community. Given the distinct physical properties of heterogeneous images, it is difficult for direct extraction of change information. Some models that transform heterogeneous images into a mutual feature domain can be beneficial. However, the transformation may be influenced by the changed areas that are not the discrepancy of the domains, which further decreases the accuracy of CD. To solve the problem, we propose a reciprocal transformation-based joint deep and broad learning (RTDBL) model for CD with heterogeneous images. In the RTDBL model, in order to rapidly extract features, a deep feature extraction (DFE) module is designed without the need for training. In addition, for directly highlighting change information and eliminating the influence of changed areas, a reciprocal heterogeneous nodes transformation (RHNT) module is designed to construct regression functions for achieving reciprocal transformation. Subsequently, to achieve cross-spatial information interaction, a structural nodes extraction (SNE) module is proposed for obtaining structural nodes. For effectively utilizing aforementioned information and exploring the connections of heterogeneous nodes, a heterogeneous dual broad learning (HDBL) is developed to predict the change map. According to the best of our knowledge, this is the first attempt that joints deep learning and broad learning for CD with heterogeneous images. The efficacy of the proposed RTDBL is demonstrated through experimental analysis on four widely used datasets, in comparison with ten state-of-the-art models.

Semi-supervised learning (SSL), which aims to learn with limited labeled data and massive amounts of unlabeled data, offers a promising approach to exploit the massive amounts of satellite Earth observation images. The fundamental concept underlying most state-of-the-art SSL methods involves generating pseudo-labels for unlabeled data based on image-level predictions. However, complex remote sensing (RS) scene images frequently encounter challenges, such as interference from multiple background objects and significant intra-class differences, resulting in unreliable pseudo-labels. In this paper, we propose the SemiRS-COC, a novel semi-supervised classification method for complex RS scenes. Inspired by the idea that neighboring objects in feature space should share consistent semantic labels, SemiRS-COC utilizes the similarity between foreground objects in RS images to generate reliable object-level pseudo-labels, effectively addressing the issues of multiple background objects and significant intra-class differences in complex RS images. Specifically, we first design a Local Self-Learning Object Perception (LSLOP) mechanism, which transforms multiple background objects interference of RS images into usable annotation information, enhancing the model's object perception capability. Furthermore, we present a Cross-Object Consistency Pseudo-Labeling (COCPL) strategy, which generates reliable object-level pseudo-labels by comparing the similarity of foreground objects across different RS images, effectively handling significant intra-class differences. Extensive experiments demonstrate that our proposed method achieves excellent performance compared to state-of-the-art methods on three widely-adopted RS datasets.

Recent advancements in deep learning techniques have pushed forward the frontiers of real photograph denoising. However, due to the inherent pooling operations in the spatial domain, current CNN-based denoisers are biased towards focusing on low-frequency representations, while discarding the high-frequency components. This will induce a problem for suboptimal visual quality as the image denoising tasks target completely eliminating the complex noises and recovering all fine-scale and salient information. In this work, we tackle this challenge from the frequency perspective and present a new solution pipeline, coined as frequency attention denoising network (FADNet). Our key idea is to build a learning-based frequency attention framework, where the feature correlations on a broader frequency spectrum can be fully characterized, thus enhancing the representational power of the network across multiple frequency channels. Based on this, we design a cascade of adaptive instance residual modules (AIRMs). In each AIRM, we first transform the spatial-domain features into the frequency space. Then, a learning-based frequency attention framework is devised to explore the feature inter-dependencies converted in the frequency domain. Besides this, we introduce an adaptive layer by leveraging the guidance of the estimated noise map and intermediate features to meet the challenges of model generalization in the noise discrepancy. The effectiveness of our method is demonstrated on several real camera benchmark datasets, with superior denoising performance, generalization capability, and efficiency versus the state-of-the-art.

Deep learning-based hyperspectral image (HSI) change detection (CD) approaches have a strong ability to leverage spectral-spatial-temporal information through automatic feature extraction, and currently dominate in the research field. However, their efficiency and universality are limited by the dependency on labeled data. Although the newly applied untrained networks can avoid the need for labeled data, their feature volatility from the simple difference space easily leads to inaccurate CD results. Inspired by the interesting finding that salient changes appear as bright "stripes" in a new feature space, we propose a novel unsupervised CD method that represents and models changes in stripes for HSIs (named as StripeCD), which integrates optimization modeling into an untrained network. The StripeCD method constructs a new feature space that represents change features in stripes and models them in a novel optimization manner. It consists of three main parts: 1) dual-branch untrained convolutional network, which is utilized to extract deep difference features from bitemporal HSIs and combined with a two-stage channel selection strategy to emphasize the important channels that contribute to CD. 2) multiscale forward-backward segmentation framework, which is proposed for salient change representation. It transforms deep difference features into a new feature space by exploiting the structure information of ground objects and associates salient changes with the stripe-shaped change component. 3) stripe-shaped change extraction model, which characterizes the global sparsity and local discontinuity of salient changes. It explores the intrinsic properties of deep difference features and constructs model-based constraints to better identify changed regions in a controllable manner. The proposed StripeCD method outperformed the state-of-the-art unsupervised CD approaches on three widely used datasets. In addition, the proposed StripeCD method indicates the potential for further investigation of untrained networks in facilitating reliable CD.

As the foundation of image interpretation, semantic segmentation is an active topic in the field of remote sensing. Facing the complex combination of multiscale objects existing in remote sensing images (RSIs), the exploration and modeling of contextual information have become the key to accurately identifying the objects at different scales. Although several methods have been proposed in the past decade, insufficient context modeling of global or local information, which easily results in the fragmentation of large-scale objects, the ignorance of small-scale objects, and blurred boundaries. To address the above issues, we propose a contextual representation enhancement network (CRENet) to strengthen the global context (GC) and local context (LC) modeling in high-level features. The core components of the CRENet are the local feature alignment enhancement module (LFAEM) and the superpixel affinity loss (SAL). The LFAEM aligns and enhances the LC in low-level features by constructing contextual contrast through multilayer cascaded deformable convolution and is then supplemented with high-level features to refine the segmentation map. The SAL assists the network to accurately capture the GC by supervising semantic information and relationship learned from superpixels. The proposed method is plug-and-play and can be embedded in any FCN-based network. Experiments on two popular RSI datasets demonstrate the effectiveness of our proposed network with competitive performance in qualitative and quantitative aspects.

Despite the great potential of convolutional neural networks (CNNs) in various tasks, the resource-hungry nature greatly hinders their wide deployment in cost-sensitive and low-powered scenarios, especially applications in remote sensing. Existing model pruning approaches, implemented by a "subtraction" operation, impose a performance ceiling on the slimmed model. Self-knowledge distillation (Self-KD) resorts to auxiliary networks that are only active in the training phase for performance improvement. However, the knowledge is holistic and crude, and the learning-based knowledge transfer is mediate and lossy. Here, we propose a novel model-compression method, termed block-wise partner learning (BPL), which comprises "extension" and "fusion" operations and liberates the compressed model from the bondage of baseline. Different from the Self-KD, the proposed BPL creates a partner for each block for performance enhancement in training. For the model to absorb more diverse information, a diversity loss (DL) is designed to evaluate the difference between the original block and the partner. Besides, the partner is fused equivalently instead of being discarded directly. After training, we can simply adopt the fused compressed model that contains the enhancement information of partners but with fewer parameters and less inference cost. As validated using the UC Merced land-use, NWPU-RESISC45, and RSD46-WHU datasets, the BPL demonstrates superiority over other compared model-compression approaches. For example, it attains a substantial floating-point operations (FLOPs) reduction of 73.97% with only 0.24 accuracy (ACC.) loss for ResNet-50 on the UC Merced land-use dataset. The code is available at https://github.com/zhangxin-xd/BPL.

Visual Relational Reasoning is the basis of many vision-and-language based tasks (e.g., visual question answering and referring expression comprehension). In this article, we regard the complex referring expression comprehension (c-REF) task as the reasoning basis, in which c-REF seeks to localise a target object in an image guided by a complex query. Such queries often contain complex logic and thus impose two critical challenges for reasoning: (i) Comprehending the complex queries is difficult since these queries usually refer to multiple objects and their relationships; (ii) Reasoning among multiple objects guided by the queries and then localising the target correctly are non-trivial. To address the above challenges, we propose a Transformer-based Relational Inference Network (Trans-RINet). Specifically, to comprehend the queries, we mimic the language-comprehending mechanism of humans, and devise a language decomposition module to decompose the queries into four types, i.e., basic attributes, absolute location, visual relationship and relative location. We further devise four modules to address the corresponding information. In each module, we consider the intra-(i.e., between the objects) and inter-modality relationships(i.e., between the queries and objects) to improve the reasoning ability. Moreover, we construct a relational graph to represent the objects and their relationships, and devise a multi-step reasoning method to progressively understand the complex logic. Since each type of the queries is closely related, we let each module interact with each other before making a decision. Extensive experiments on the CLEVR-Ref+, Ref-Reasoning, and CLEVR-CoGenT datasets demonstrate the superior reasoning performance of our Trans-RINet.

For real applications, existing HSI-SR methods are not only limited to unstable performance under unknown scenarios but also suffer from high computation consumption. In this paper, we develop a new coordination optimization framework for stable, interpretable, and lightweight HSI-SR. Specifically, we create a positive cycle between fusion and degradation estimation under a new probabilistic framework. The estimated degradation is applied to fusion as guidance for a degradation-aware HSI-SR. Under the framework, we establish an explicit degradation estimation method to tackle the indeterminacy and unstable performance caused by the black-box simulation in previous methods. Considering the interpretability in fusion, we integrate spectral mixing prior into the fusion process, which can be easily realized by a tiny autoencoder, leading to a dramatic release of the computation burden. Based on the spectral mixing prior, we then develop a partial fine-tune strategy to reduce the computation cost further. Comprehensive experiments demonstrate the superiority of our method against the state-of-the-arts under synthetic and real datasets. For instance, we achieve a 2.3 dB promotion on PSNR with$120\times$model size reduction and$4300 \times$FLOPs reduction under the CAVE dataset. Code is available in https://github.com/WenjinGuo/DAEM.

This paper introduces a robust and scalable Gaussian process regression (GPR) model via variational learning. This enables the application of Gaussian processes to a wide range of real data, which are often large-scale and contaminated by outliers. Towards this end, we employ a mixture likelihood model where outliers are assumed to be sampled from a uniform distribution. We next derive a variational formulation that jointly infers the mode of data, i.e., inlier or outlier, as well as hyperparameters by maximizing a lower bound of the true log marginal likelihood. Compared to previous robust GPR, our formulation approximates the exact posterior distribution. The inducing variable approximation and stochastic variational inference are further introduced to our variational framework, extending our model to large-scale data. We apply our model to two challenging real-world applications, namely feature matching and dense gene expression imputation. Extensive experiments demonstrate the superiority of our model in terms of robustness and speed. Notably, when matching 4k feature points, its inference is completed in milliseconds with almost no false matches. The code is at github.com/YifanLu2000/Robust-Scalable-GPR.

Sparsely annotated semantic segmentation (SASS) aims to learn a segmentation model by images with sparse labels (i.e., points or scribbles). Existing methods mainly focus on introducing low-level affinity or generating pseudo labels to strengthen supervision, while largely ignoring the inherent relation between labeled and unlabeled pixels. In this paper, we observe that pixels that are close to each other in the feature space are more likely to share the same class. Inspired by this, we propose a novel SASS framework, which is equipped with an Adaptive Gaussian Mixture Model (AGMM). Our AGMM can effectively endow reliable supervision for unlabeled pixels based on the distributions of labeled and unlabeled pixels. Specifically, we first build Gaussian mixtures using labeled pixels and their relatively similar unlabeled pixels, where the labeled pixels act as centroids, for modeling the feature distribution of each class. Then, we leverage the reliable information from labeled pixels and adaptively generated GMM predictions to supervise the training of unlabeled pixels, achieving online, dynamic, and robust selfsupervision. In addition, by capturing category-wise Gaussian mixtures, AGMM encourages the model to learn discriminative class decision boundaries in an end-to-end contrastive learning manner. Experimental results conducted on the PASCAL VOC 2012 and Cityscapes datasets demonstrate that our AGMM can establish new state-of-the-art SASS performance. Code is available at https://github.com/Luffy03/AGMM-SASS

Joint classification of multisource data for better Earth observation becomes an interesting but challenging problem. However, existing methods usually fail to be optimal due to the limitations in the heterogeneous feature representation and complementary information fusion. In this paper, we propose a new multi-level attention-based feature fusion method for the joint classification of HSI and LiDAR data. First, a two-stream deep network is built to extract the spectral-spatial feature of HSI and the elevation feature of LiDAR, respectively. To fully use the complementary and correlated information of HSI and LiDAR data, we adopt attention-based feature extraction and fusion module to deliver a high-discrimination feature representation both for cross-source and single-source data. Then, the extracted features are fed into fully connected layers to generate class probabilities. Finally, a decision-level fusion strategy is adopted to further improve the classification results. Extensive experiments on the Houston dataset demonstrate the effectiveness of the proposed method over some state-of-the-art approaches.

Semi-supervised semantic segmentation aims to learn a semantic segmentation model via limited labeled images and adequate unlabeled images. The key to this task is generating reliable pseudo labels for unlabeled images. Existing methods mainly focus on producing reliable pseudo labels based on the confidence scores of unlabeled images while largely ignoring the use of labeled images with accurate annotations. In this paper, we propose a Cross-Image Semantic Consistency guided Rectifying (CISC-R) approach for semi-supervised semantic segmentation, which explicitly leverages the labeled images to rectify the generated pseudo labels. Our CISC-R is inspired by the fact that images belonging to the same class have a high pixel-level correspondence. Specifically, given an unlabeled image and its initial pseudo labels, we first query a guiding labeled image that shares the same semantic information with the unlabeled image. Then, we estimate the pixel-level similarity between the unlabeled image and the queried labeled image to form a CISC map, which guides us to achieve a reliable pixel-level rectification for the pseudo labels. Extensive experiments on the PASCAL VOC 2012, Cityscapes, and COCO datasets demonstrate that the proposed CISC-R can significantly improve the quality of the pseudo labels and outperform the state-of-the-art methods. Code is available at https://github.com/Luffy03/CISC-R.

Supervised deep learning techniques have been widely explored in real photograph denoising and achieved noticeable performances. However, being subject to specific training data, most current image denoising algorithms can easily be restricted to certain noisy types and exhibit poor generalizability across testing sets. To address this issue, we propose a novel flexible and well-generalized approach, coined as dual meta attention network (DMANet). The DMANet is mainly composed of a cascade of the self-meta attention blocks (SMABs) and collaborative-meta attention blocks (CMABs). These two blocks have two forms of advantages. First, they simultaneously take both spatial and channel attention into account, allowing our model to better exploit more informative feature interdependencies. Second, the attention blocks are embedded with the meta-subnetwork, which is based on metalearning and supports dynamic weight generation. Such a scheme can provide a beneficial means for self and collaborative updating of the attention maps on-the-fly. Instead of directly stacking the SMABs and CMABs to form a deep network architecture, we further devise a three-stage learning framework, where different blocks are utilized for each feature extraction stage according to the individual characteristics of SMAB and CMAB. On five real datasets, we demonstrate the superiority of our approach against the state of the art. Unlike most existing image denoising algorithms, our DMANet not only possesses a good generalization capability but can also be flexibly used to cope with the unknown and complex real noises, making it highly competitive for practical applications.

Hyperspectral Image (HSI) classification is an important issue in remote sensing field with extensive applications in earth science. In recent years, a large number of deep learning-based HSI classification methods have been proposed. However, existing methods have limited ability to handle high-dimensional, highly redundant, and complex data, making it challenging to capture the spectral-spatial distributions of data and relationships between samples. To address this issue, we propose a generative framework for HSI classification with diffusion models (SpectralDiff) that effectively mines the distribution information of high-dimensional and highly redundant data by iteratively denoising and explicitly constructing the data generation process, thus better reflecting the relationships between samples. The framework consists of a spectral-spatial diffusion module, and an attention-based classification module. The spectral-spatial diffusion module adopts forward and reverse spectral-spatial diffusion processes to achieve adaptive construction of sample relationships without requiring prior knowledge of graphical structure or neighborhood information. It captures spectral-spatial distribution and contextual information of objects in HSI and mines unsupervised spectral-spatial diffusion features within the reverse diffusion process. Finally, these features are fed into the attention-based classification module for per-pixel classification. The diffusion features can facilitate cross-sample perception via reconstruction distribution, leading to improved classification performance. Experiments on three public HSI datasets demonstrate that the proposed method can achieve better performance than state-of-the-art methods. For the sake of reproducibility, the source code of SpectralDiff will be publicly available at https://github.com/chenning0115/SpectralDiff.

In recent years, hyperspectral image (HSI) classification and detection techniques based on deep learning have been widely applied to various aspects, such as environmental monitoring, urban planning, and energy surveys. As an important image content analysis method, instance segmentation can provide important support for the extraction of ground object information and monomeric application of HSI. This article introduces instance segmentation into HSI interpretation for the first time. In this article, we create the hyperspectral instance segmentation dataset (HS-ISD), which contains a total of 56 images, each with a size of$298\times301$and a number of channels of 48. More than 1000 architectural examples are annotated to apply to the research of HSI instance segmentation. In addition, considering that HSI contains rich spectral and spatial information, and the traditional instance segmentation network model cannot well utilize both types of information effectively, we propose the spectral–spatial feature pyramid network (Spectral–Spatial FPN). The Spectral–Spatial FPN can integrate multiscale spectral information and multiscale spatial information in the feature extraction stage through attention mechanism and bidirectional feature pyramid structure, so as to better improve the performance of the network model by spectral information and spatial information and realize the end-to-end instance segmentation of HSI. The experimental results conducted on the HS-ISD show that the proposed Spectral–Spatial FPN can achieve state-of-the-art results.

Large-scale pre-trained models, such as vision transformers, have made significant progress in remote sensing (RS) scene classification tasks. For a new scene classification task, it is popular to fully fine-tune the pre-trained model parameters to avoid training from scratch. Although such an approach achieves satisfactory results, it will lead to heavy computation and storage burden, which limits the transferability of large pre-trained models to different RS scene classification tasks. To address this challenge, we propose a parameter-efficient tuning approach called as the Instance-Aware Visual Prompting (IVP), which is the first work to explore the prompting in the field of RS scene classification. The proposed IVP adaptively generates prompts based on the complex background and highly variable characteristics of RS images, and updates only a few parameters to transfer the pre-trained RS Transformer model to different scene classification tasks. Specifically, instead of adapting the entire model parameters, we introduce some instance-specific prompt vectors into the input space. Then, considering the significant variability in RS images, we introduce an instance-level prompt generation module to generate specific prompts for each RS image by aggregating contextual information from the input. Finally, these prompt vectors will calibrate the pre-trained features to encode instance-specific information. Extensive experiments on three RS scene classification datasets demonstrate the superiority of IVP over other fine-tuning methods. For example, when updating just 1.1% parameters, the Swin Transformer model achieves about 1.83% and 1.42% improvement compared to the full fine-tuning method on NWPU-19 and NWPU-28, respectively.

Hyperspectral images (HSIs) can provide rich spectral-spatial information that has been widely utilized in many fields, such as national defense, mineralogy and agriculture. Most of the recent HSI interpretation methods are conducted in the raster pattern, which results in high memory costs, amplification distortion, and difficulties in topological editing. To address this issue, a novel end-to-end vectorization framework is proposed, called as the HSI Vectorization Network (HSI-VecNet), which learns a vector representation from spectral-spatial information through cross-level interactions. Specifically, this framework integrates low-level geometry information and high-level semantic instance information, which consists of two branches: the HSI Semantic Instance Segmentation (HSIS) and the Spectral-Spatial Junction Prediction (SSJP). The HSIS conducts the raster-based classification and extracts the semantic information of each object in the HSI. In addition, the SSJP exploits spectral-spatial information to predict the positions of junctions in the HSI. The instance information of each object and the relations of junctions are then fused to vectorize the HSI. To verify the effectiveness of the proposed method, four hyperspectral datasets are vectorially labeled. Experimental results on these datasets demonstrate that the proposed end-to-end HSI-VecNet outperforms existing post-process vectorization methods. Our model and datasets will be made publicly available at https://github.com/yyyyll0ss/HSI-VecNet.

Semi-supervised learning (SSL) is a promising approach to reduce the labeling burden in remote sensing scene classification tasks. However, most semi-supervised methods typically exploit the single-level semantic information of unlabeled data, ignoring the multi-level semantic structure prevalent in remote sensing data. The multi-level semantic structure, which contains the correlation of different categories and the multi-granularity semantic information, can help the scene classification model to more accurately measure the feature distance between different categories and more effectively utilize unlabeled data. Therefore, this paper proposes a multi-level label-aware semi-supervised scene classification framework, MLLA, which extends the semantic information captured in unlabeled data from single-level to multi-level to improve the scene classification performance. Specifically, we first propose a multi-level prototype awareness module to capture the multi-level semantic structure underlying remote sensing data. Then, based on this structure, a multi-level pseudo-label generation module is designed to assign multi-level pseudo-labels to the unlabeled data. Finally, by combining the labeled samples and the multi-level pseudo-labeled samples, the scene classification model is progressively trained. The experimental results on three benchmark datasets show that the proposed MLLA achieves excellent performance compared to other semi-supervised classification methods.

The capacity of satellites to supply high-resolution imaging has promoted the fine-grained object detection task in remote sensing images. However, this type of object detection is challenging due to low interclass feature differences in objects. To address this issue, we propose a prototypical contrastive learning-based detector (PCLDet) for fine-grained object detection in remote sensing images. The PCLDet first introduces the prototype to learn the fine-grained objects’ features, and then adopts contrastive learning to compare the target and the learned features, thus improving the differentiability of the fine-grained object. Specifically, we first introduce the prototype, which represents the feature centers of each class, and then construct a prototype bank to store the feature prototypes of each class. Then, we introduce contrastive learning to extract the discriminative features by maximizing the interclass distance and minimizing the intraclass distance. Furthermore, we propose the ProtoCL loss as a part of the model optimization, which enables more representative prototypes to be learned. Finally, to address the long-tail problem in the remote sensing fine-grained object detection dataset, we propose a new proposal sampler, the class-balanced sampler (CBS) that can sample each class equally. Extensive experiments demonstrate that our method can achieve state-of-the-art performance on a commonly used aerial fine-grained object dataset (Fair1M) and aerial fine-grained ship dataset (OFSD) while maintaining high efficiency. The code will be available at https://github.com/G-Naughty/PCLDet.

Multisource remote sensing data provide abundant and complementary information for land cover classification. Existing classification methods mainly focus on designing a multi-stream deep network to extract separate features of each single-source data, then adopting a fusing strategy to combine these extracted features for final classification. However, this kind of method neglects the sample correlation of single-source and cross-source data, which may deliver an unsatisfactory classification result when dealing with high intraclass-variability and low interclass-variability samples. To this end, a novel hashing-based deep metric learning (HDML) method is proposed for hyperspectral images (HSIs) and light detection and ranging (LiDAR) data classification in this paper. First, a two-stream deep network is built to extract the spectral-spatial features of HSI and the elevation features of LiDAR, respectively. To fully use the complementary and correlated information of HSI and LiDAR data, we adopt attention-based feature fusion (AFF) modules to deliver a high-discrimination fused feature both for cross-source and single-source feature fusion. Then, the extracted features are fed into fully connected layers to generate class probabilities, respectively. Different from most existing methods that only utilize semantic information of samples, we elaborately designed a loss function to simultaneously consider the label-based semantic loss and hashing-based metric loss. Finally, a decision-level fusion strategy is adopted to further improve the classification results. Extensive experiments on three public HSI and LiDAR data sets demonstrate the effectiveness of the proposed method over some state-of-the-art approaches.

High-ratio image compression is difficult because remote sensing images have complex background and rich information, and the correlation between features is weak. An accurate entropy model is an important way to solve the problem by enhancing the representation ability of the compression models. The entropy model is more suited to estimate the probability distributions with the sparse latent representations. This study proposes a novel entropy model (DWTGMM) based on discrete wavelet transform (DWT) and Gaussian mixture model (GMM) for remote sensing image compression. The method uses DWT to transform the latent representations into wavelet domain and obtains four sparse representations, and then uses the proposed DWTGMM to model them separately to estimate the probability distribution of each element. It is noteworthy that the DWT used in our approach does not require learning parameters and can be combined with other entropy models to acquire the distribution of latent representations. To evaluate our method, we construct three remote sensing image datasets, i.e., GoogleMap, GF1, and GF7. We compare our method with several popular learned compression models and traditional codecs. Experimental results show that the proposed method can achieve excellent performance with low complexity. Especially with the same model architecture, the DWTGMM achieves the best compression performance.

Convolutional neural networks (CNNs) have been successfully employed in remote sensing image classification because of their robust feature representation for different visual tasks and powerful graphics processing units (GPUs). The attendant problem is that high computational cost and high memory footprint hindering the application of CNNs for remote sensing applications in resource- and time-sensitive situations. Based on practical deployment requirements, we pioneer a pruning-quantization joint learning model compression method for remote sensing image classification, called co-compression via superior gene (CC-SG). An enhanced evolution algorithm (EEA) is adopted as the agent to search a “superior gene,” and immediately following, a director receives the “superior gene” and gives a compression mask and a resource constraint feedback to the agent. The network is eventually compressed and fine-tuned according to the optimal compression mask. Specifically, we introduce gene age and progressive shrinkage mutation rate to EEA and design a fitness function that balances accuracy and resource constraints. As validated using the UC Merced land-use and NWPU-RESISC45 datasets, the proposed CC-SG demonstrated superiority over other compared model compression approaches. For example, CC-SG attained substantial bit operations (BOPs) compression ratio of 40.04 with 0.956% accuracy increase for VGG-16 on UC Merced land-use dataset and 40.00 with 0.203% accuracy increase for ResNet-56 on NWPU-RESISC45 dataset. The code is available athttps://github.com/fanxxxxyi/CC-SG.

Accurate retinal fluid segmentation on Optical Coherence Tomography (OCT) images plays an important role in diagnosing and treating various eye diseases. The art deep models have shown promising performance on OCT image segmentation given pixel-wise annotated training data. However, the learned model will achieve poor performance on OCT images that are obtained from different devices (domains) due to the domain shift issue. This problem largely limits the real-world application of OCT image segmentation since the types of devices usually are different in each hospital. In this paper, we study the task of cross-domain OCT fluid segmentation, where we are given a labeled dataset of the source device (domain) and an unlabeled dataset of the target device (domain). The goal is to learn a model that can perform well on the target domain. To solve this problem, in this paper, we propose a novel Structure-guided Cross-Attention Network (SCAN), which leverages the retinal layer structure to facilitate domain alignment. Our SCAN is inspired by the fact that the retinal layer structure is robust to domains and can reflect regions that are important to fluid segmentation. In light of this, we build our SCAN in a multi-task manner by jointly learning the retinal structure prediction and fluid segmentation. To exploit the mutual benefit between layer structure and fluid segmentation, we further introduce a cross-attention module to measure the correlation between the layer-specific feature and the fluid-specific feature encouraging the model to concentrate on highly relative regions during domain alignment. Moreover, an adaptation difficulty map is evaluated based on the retinal structure predictions from different domains, which enforces the model focus on hard regions during structure-aware adversarial learning. Extensive experiments on the three domains of the RETOUCH dataset demonstrate the effectiveness of the proposed method and show that our approach produces state-of-the-art performance on cross-domain OCT fluid segmentation.

Network pruning is one of the chief means for improving the computational efficiency of Deep Neural Networks (DNNs). Pruning-based methods generally discard network kernels, channels, or layers, which however inevitably will disrupt original well-learned network correlation and thus lead to performance degeneration. In this work, we propose an Efficient Layer Compression (ELC) approach to efficiently compress serial layers by decoupling and merging rather than pruning. Specifically, we first propose a novel decoupling module to decouple the layers, enabling us readily merge serial layers that include both nonlinear and convolutional layers. Then, the decoupled network is losslessly merged based on the equivalent conversion of the parameters. In this way, our ELC can effectively reduce the depth of the network without destroying the correlation of the convolutional layers. To our best knowledge, we are the first to exploit the mergeability of serial convolutional layers for lossless network layer compression. Experimental results conducted on two datasets demonstrate that our method retains superior performance with a FLOPs reduction of 74.1% for VGG-16 and 54.6% for ResNet-56, respectively. In addition, our ELC improves the inference speed by 2× on Jetson AGX Xavier edge device.

Color plays an important role in human visual perception, reflecting the spectrum of objects. However, the existing infrared and visible image fusion methods rarely explore how to handle multi-spectral/channel data directly and achieve high color fidelity. This paper addresses the above issue by proposing a novel method with diffusion models, termed as Dif-Fusion, to generate the distribution of the multi-channel input data, which increases the ability of multi-source information aggregation and the fidelity of colors. In specific, instead of converting multi-channel images into single-channel data in existing fusion methods, we create the multi-channel data distribution with a denoising network in a latent space with forward and reverse diffusion process. Then, we use the the denoising network to extract the multi-channel diffusion features with both visible and infrared information. Finally, we feed the multi-channel diffusion features to the multi-channel fusion module to directly generate the three-channel fused image. To retain the texture and intensity information, we propose multi-channel gradient loss and intensity loss. Along with the current evaluation metrics for measuring texture and intensity fidelity, we introduce Delta E as a new evaluation metric to quantify color fidelity. Extensive experiments indicate that our method is more effective than other state-of-the-art image fusion methods, especially in color fidelity. The source code is available at https://github.com/GeoVectorMatrix/Dif-Fusion.

Neurologically, filter pruning is a procedure of forgetting and remembering recovering. Prevailing methods directly forget less important information from an unrobust baseline at first and expect to minimize the performance sacrifice. However, unsaturated base remembering imposes a ceiling on the slimmed model leading to suboptimal performance. And significantly forgetting at first would cause unrecoverable information loss. Here, we design a novel filter pruning paradigm termed Remembering Enhancement and Entropy-based Asymptotic Forgetting (REAF). Inspired by robustness theory, we first enhance remembering by over-parameterizing baseline with fusible compensatory convolutions which liberates pruned model from the bondage of baseline at no inference cost. Then the collateral implication between original and compensatory filters necessitates a bilateral-collaborated pruning criterion. Specifically, only when the filter has the largest intra-branch distance and its compensatory counterpart has the strongest remembering enhancement power, they are preserved. Further, Ebbinghaus curve-based asymptotic forgetting is proposed to protect the pruned model from unstable learning. The number of pruned filters is increasing asymptotically in the training procedure, which enables the remembering of pretrained weights gradually to be concentrated in the remaining filters. Extensive experiments demonstrate the superiority of REAF over many state-of-the-art (SOTA) methods. For example, REAF removes 47.55% FLOPs and 42.98% parameters of ResNet-50 only with 0.98% TOP-1 accuracy loss on ImageNet. The code is available at https://github.com/zhangxin-xd/REAF.

In recent years, deep learning methods have achieved great success for vehicle detection tasks in aerial imagery. However, most existing methods focus only on extracting latent vehicle target features, and rarely consider the scene context as vital prior knowledge. In this letter, we propose a scene context attention-based fusion network (SCAF-Net), to fuse the scene context of vehicles into an end-to-end vehicle detection network. First, we propose a novel strategy, patch cover, to keep the original target and scene context information in raw aerial images of a large scale as much as possible. Next, we use an improved YOLO-v3 network as one branch of SCAF-Net, to generate vehicle candidates on each patch. Here, a novel branch for the scene context is utilized to extract the latent scene context of vehicles on each patch without any extra annotations. Then, these two branches above are concatenated together as a fusion network, and we apply an attention-based model to further extract vehicle candidates of each local scene. Finally, all vehicle candidates of different patches, are merged by global nonmax suppress (g-NMS) to output the detection result of the whole original image. Experimental results demonstrate that our proposed method outperforms the comparison methods with both high detection accuracy and speed. Our code is released athttps://github.com/minghuicode/SCAF-Net.

Salient object detection (SOD) for optical remote sensing images (RSIs) aims at locating and extracting visually distinctive objects/regions from the optical RSIs. Despite some saliency models were proposed to solve the intrinsic problem of optical RSIs (such as complex background and scale-variant objects), the accuracy and completeness are still unsatisfactory. To this end, we propose a relational reasoning network with parallel multi-scale attention for SOD in optical RSIs in this paper. The relational reasoning module that integrates the spatial and the channel dimensions is designed to infer the semantic relationship by utilizing high-level encoder features, thereby promoting the generation of more complete detection results. The parallel multi-scale attention module is proposed to effectively restore the detail information and address the scale variation of salient objects by using the low-level features refined by multi-scale attention. Extensive experiments on two datasets demonstrate that our proposed RRNet outperforms the existing state-of-the-art SOD competitors both qualitatively and quantitatively.

The efficient and accurate registration of multitemporal images is essential for many remote sensing applications. With the increase in the imaging resolution and field in satellites, the acquired large-scale remote sensing images have brought serious challenges, since there exist parallax shifts and large background variations among different local regions of the acquired images. To address these issues, this article proposes an adaptive regional multiple features (ARMF) matching method for the registration of multitemporal large-scale high-resolution remote sensing images. Specifically, since large background variations in fixed-size regions of multitemporal images will cause insufficient features and the failure of features matching, the ARMF introduces an adaptive regions searching strategy, which utilizes the pyramid amplification technique to adaptively select the regions that can find the sufficient matched features. Then, the ARMF extracts multiple types of features (i.e., gradient feature, phase feature, and line feature) from the adaptive searched region that can more effectively represent the characteristics of the large regions. Finally, we utilize the feature matching error as the rule to adaptively select the suitable features as the descriptors of the region. The experimental results on large-scale multitemporal image data obtained from Google Earth demonstrated the proposed method can outperform several state-of-the-arts remote sensing registration approaches.

The use of deep learning for water extraction requires precise pixel-level labels. However, it is very difficult to label high-resolution remote-sensing images at the pixel level. Therefore, we study how to utilize point labels to extract water bodies and propose a novel method called the neighbor feature aggregation network (NFANet). Compared with pixel-level labels, point labels are much easier to obtain, but they will lose much information. In this article, we take advantage of the similarity between the adjacent pixels of a local water body, and propose a neighbor sampler to resample remote-sensing images. Then, the sampled images are sent to the network for feature aggregation. In addition, we use an improved recursive training algorithm to further improve the extraction accuracy, making the water boundary more natural. Furthermore, our method utilizes neighboring features instead of global or local features to learn more representative features. The experimental results show that the proposed NFANet method not only outperforms other studied weakly supervised approaches, but also obtains similar results as the state-of-the-art ones.

In recent years, deep learning presents a promising performance in hyperspectral image (HSI) classification, due to the powerful capability of automatically learning deep semantic characteristics of images. However, it is still difficult to learn highly discriminative features when limited samples are available for training a deep network. Focused on this issue, a novel end-to-end supervised contrastive learning network (SCL-Net) for spectral-spatial classification is proposed, in this paper. Instead of learning features of the individual sample, the supervised contrastive learning is introduced to capture the similarity and dissimilarity distribution properties of sample pairs in feature representation space. In this way, the need for plenty of training samples will be alleviated while an effective network training mechanism is provided for learning highly separative features. Here, the SCL-Net mainly consists of one pair-wise contrastive learning (PCL) sub-network and one multi-level spectral-spatial information fusion (MLSIF) sub-network. For the PCL sub-network, spectral vectors are projected into deep spectral features based on convolutional operators, which are then followed by distance evaluation between “positive” pairs of similar samples and “negative” pairs of dissimilar ones. Then, a spectral distance matrix is constructed to push the network to gradually learn better features of higher intra-class compactness and inter-class dispersion. For the MLSIF sub-network, a hybrid feature-decision fusion strategy is designed, where spatial and spectral features are jointly exploited to further boost classification performance. In specific, the feature fusion is conducted by connecting low/mid/high-level spectral and spatial features via weighting, while multiple class estimations based on multi-level fusion features are adaptively integrated via probabilistic decision fusion. Overall, these two sub-networks are collaboratively trained in one framework, by optimizing a defined joint loss function consisting of a contrastive loss and a cross-entropy loss. Compared with several state-of-the-art methods, the proposed method yields a superior classification performance in terms of both objective metrics and visual performance.

Unsupervised domain adaptive (UDA) image segmentation has recently gained increasing attention, aiming to improve the generalization capability for transferring knowledge from the source domain to the target domain. However, in high spatial resolution remote sensing image (RSI), the same category from different domains (e.g., urban and rural) can appear to be totally different with extremely inconsistent distributions, which heavily limits the UDA accuracy. To address this problem, in this article, we propose a novel deep covariance alignment (DCA) model for UDA RSI segmentation. The DCA can explicitly align category features to learn shared domain-invariant discriminative feature representations, which enhance the ability of model generalization. Specifically, a category feature pooling (CFP) module is first used to extract category features by combining coarse outputs and deep features. Then, we leverage a novel covariance regularization (CR) to enforce the intracategory features to be closer and the intercategory features to be further separate. Compared with the existing category alignment methods, our CR aims to regularize the correlation between different dimensions of the features, and thus performs more robustly when dealing with divergent category features of imbalanced and inconsistent distributions. Finally, we propose a stagewise procedure to train the DCA to alleviate error accumulation. Experiments on both rural-to-urban and urban-to-rural scenarios of the LoveDA dataset demonstrate the superiority of our proposed DCA over other state-of-the-art UDA segmentation methods. Code is available athttps://github.com/Luffy03/DCA.

The low resolution of remote sensing images often limits the land cover classification (LCC) performance. Super resolution (SR) can improve the image resolution, while greatly increasing the computational burden for the LCC due to the larger size of the input image. In this article, the SR-guided deep network (SRGDN) framework is proposed, which can generate meaningful structures from higher resolution images to improve the LCC performance without consuming more computational costs. In general, the SRGDN consists of two branches (i.e., SR branch and LCC branch) and a guidance module. The SR branch aims to increase the resolution of remote sensing images. Since high- and low-resolution image pairs cannot be directly provided by imaging sensors to train the SR branch, we introduce a self-supervised generative adversarial network (GAN) to estimate the downsampling kernel that can produce these image pairs. The LCC branch adopts the high-resolution network (HRNet) to retain as much resolution information with a few downsampling operations as possible. The guidance module teaches the LCC branch to learn the high-resolution information from the SR branch without the utilization of the higher-resolution images as the inputs. Furthermore, the guidance module introduces spatial pyramid pooling (SPP) to match the feature maps of different sizes in the two branches. In the testing stage, the guidance module and SR branch can be removed, and therefore do not create additional computational costs. Experimental results on three real datasets demonstrate the superiority of the proposed method over several well-known LCC approaches.

Hyperspectral image (HSI) classification is an important topic in the community of remote sensing, which has a wide range of applications in geoscience. Recently, deep learning-based methods have been widely used in HSI classification. However, due to the scarcity of labeled samples in HSI, the potential of deep learning-based methods has not been fully exploited. To solve this problem, a self-supervised learning (SSL) method with adaptive distillation is proposed to train the deep neural network with extensive unlabeled samples. The proposed method consists of two modules: adaptive knowledge distillation with spatial–spectral similarity and 3-D transformation on HSI cubes. The SSL with adaptive knowledge distillation uses the self-supervised information to train the network by knowledge distillation, where self-supervised knowledge is the adaptive soft label generated by spatial–spectral similarity measurement. The SSL with adaptive knowledge distillation mainly includes the following three steps. First, the similarity between unlabeled samples and object classes in HSI is generated based on the spatial–spectral joint distance (SSJD) between unlabeled samples and labeled samples. Second, the adaptive soft label of each unlabeled sample is generated to measure the probability that the unlabeled sample belongs to each object class. Third, a progressive convolutional network (PCN) is trained by minimizing the cross-entropy between the adaptive soft labels and the probabilities generated by the forward propagation of the PCN. The SSL with 3-D transformation rotates the HSI cube in both the spectral domain and the spatial domain to fully exploit the labeled samples. Experiments on three public HSI data sets have demonstrated that the proposed method can achieve better performance than existing state-of-the-art methods.

Deep learning-based methods have made significant progress in hyperspectral image (HSI) classification in recent years. However, deep learning-based methods usually rely on a large number of samples, and in many cases, it is difficult to label HSI and only limited training samples are available. To solve this problem, an HSI classification method based on adaptive spatial pyramid constraint (ASPC) is proposed to make full use of the global spatial neighborhood information of the labeled samples, which can improve the generalization ability of the classification model. The main steps of the proposed method are as follows. First, an HSI complexity evaluation method based on edge detection is proposed to assess the homogeneity of the objects in the HSI. Second, an HSI pyramid segmentation method based on spatial pyramid is proposed to generate multiscale subregions, where HSI complexity is used to adaptively determine the scale of the segmentation. Third, a spatial supervised constraint is proposed to generate the loss function of labeled subregions. Fourth, a spatial unsupervised constraint is proposed to generate the loss function of unlabeled subregions. The proposed method fully explores the spatial-spectral correlation between unlabeled samples and labeled samples, and add corresponding constraints to the training objective according to the correlation. By adding the ASPC, the trained model becomes more robust and can make full use of the limited training samples. To verify the effectiveness of the proposed method, three benchmark hyperspectral datasets are used to verify the performance of the proposed method. Experimental results show that the performance of this method is better than the existing state-of-the-art methods.

Feature extraction provides an effective tool to classify hyperspectral images (HSIs). However, most hyperspectral feature extraction methods tend to yield an over-smoothed phenomenon, which leads to inconsistency between the homogeneous regions and the ground objects in the actual scene. To alleviate this problem, an edge-aware feature extractor called contour structural profiles (CSPs) is proposed to extract the discriminative features for hyperspectral images classification (HSIC). The proposed classification method comprises three components. First, the spectral dimension of the HSI is reduced with an averaging-based method. Then, an edge-aware total variation (TV) model is constructed to extract the contour structural profile, in which a learned contour probability map is served as one of the major cues in the feature extraction process. Next, multiscale structural profiles (MSSPs) are constructed using the edge-aware TV model with different parameters so as to fully characterize ground objects with different scales. Finally, the MSSPs are fused with a kernel principal component analysis (KPCA) followed by a spectral classifier to obtain the final classification map. Experimental results on several publicly available hyperspectral datasets illustrate that the proposed method obtains superior classification performance over several state-of-the-art classification approaches, especially when the number of training samples is insufficient.

OCT fluid segmentation is a crucial task for diagnosis and therapy in ophthalmology. The current convolutional neural networks (CNNs) supervised by pixel-wise annotated masks achieve great success in OCT fluid segmentation. However, requiring pixel-wise masks from OCT images is time-consuming, expensive and expertise needed. This paper proposes an Intra- and inter-Slice Contrastive Learning Network (ISCLNet) for OCT fluid segmentation with only point supervision. Our ISCLNet learns visual representation by designing contrastive tasks that exploit the inherent similarity or dissimilarity from unlabeled OCT data. Specifically, we propose an intra-slice contrastive learning strategy to leverage the fluid-background similarity and the retinal layer-background dissimilarity. Moreover, we construct an inter-slice contrastive learning architecture to learn the similarity of adjacent OCT slices from one OCT volume. Finally, an end-to-end model combining intra- and inter-slice contrastive learning processes learns to segment fluid under the point supervision. The experimental results on two public OCT fluid segmentation datasets (i.e., AI Challenger and RETOUCH) demonstrate that the ISCLNet bridges the gap between fully-supervised and weakly-supervised OCT fluid segmentation and outperforms other well-known point-supervised segmentation methods.

Semantic segmentation methods based on deep neural networks have achieved great success in recent years. However, training such deep neural networks relies heavily on a large number of images with accurate pixel-level labels, which requires a huge amount of human effort, especially for large-scale remote sensing images. In this paper, we propose a point-based weakly supervised learning framework called the deep bilateral filtering network (DBFNet) for the semantic segmentation of remote sensing images. Compared with pixel-level labels, point annotations are usually sparse and cannot reveal the complete structure of the objects; they also lack boundary information, thus resulting in incomplete prediction within the object and the loss of object boundaries. To address these problems, we incorporate the bilateral filtering technique into deeply learned representations in two respects. First, since a target object contains smooth regions that always belong to the same category, we perform deep bilateral filtering (DBF) to filter the deep features by a nonlinear combination of nearby feature values, which encourages the nearby and similar features to become closer, thus achieving a consistent prediction in the smooth region. In addition, the DBF can distinguish the boundary by enlarging the distance between the features on different sides of the edge, thus preserving the boundary information well. Experimental results on two widely used datasets, the ISPRS 2-D semantic labeling Potsdam and Vaihingen datasets, demonstrate that our proposed DBFNet can achieve a highly competitive performance compared with state-of-the-art fully-supervised methods. Code is available at https://github.com/Luffy03/DBFNet.

Deep learning-based methods have produced significant gains for hyperspectral image (HSI) classification in recent years, leading to high impact academic achievements and industrial applications. Despite the success of deep learning-based methods in HSI classification, they still lack the robustness of handling unknown object in open-set environment (OSE). Open-set classification is to deal with the problem of unknown classes that are not included in the training set, while in closed-set environment (CSE), unknown classes will not appear in the test set. The existing open-set classifiers almost entirely rely on the supervision information given by the known classes in the training set, which leads to the specialization of the learned representations into known classes, and makes it easy to classify unknown classes as known classes. To improve the robustness of HSI classification methods in OSE and meanwhile maintain the classification accuracy of known classes, a spectral-spatial latent reconstruction framework which simultaneously conducts spectral feature reconstruction, spatial feature reconstruction and pixel-wise classification in OSE is proposed. By reconstructing the spectral and spatial features of HSI, the learned feature representation is enhanced, so as to retain the spectral-spatial information useful for rejecting unknown classes and distinguishing known classes. The proposed method uses latent representations for spectral-spatial reconstruction, and achieves robust unknown detection without compromising the accuracy of known classes. Experimental results show that the performance of the proposed method outperforms the existing state-of-the-art methods in OSE.

In the past decade, the object detection in remote sensing has become a research focus with the development of aerial images' acquirement technology. However, it is difficult to accurately describe rotated rigid objects with large aspect ratio through axis-aligned convolutional features. To solve this problem, we propose an oriented spatial correlative feature alignment Net (OSCFA-Net) with a correlative feature alignment module (CFAM). CFAM aligns features by rotating object features and considering the spatial correlation of rigid objects to ensure the feature alignment between the internal feature points. Experiment results demonstrate that OSCFA-Net can achieve state-of-the-art performance in remote sensing object detection.

Remote sensing scene classification aims to assign automatically each aerial image a specific sematic label. In this letter, we propose a new method, called self-attention-based deep feature fusion (SAFF), to aggregate deep layer features and emphasize the weights of the complex objects of remote sensing scene images for remote sensing scene classification. First, the pretrained convolutional neural network (CNN) model is applied to extract the abstract multilayer feature maps from the original aerial imagery. Then, a nonparametric self-attention layer is proposed for spatial-wise and channel-wise weightings, which enhances the effects of the spatial responses of the representative objects and uses the infrequently occurring features more sufficiently. Thus, it can extract more discriminative features. Finally, the aggregated features are fed into a support vector machine (SVM) for classification. The proposed method is experimented on several data sets, and the results prove the effectiveness and efficiency of the scheme for remote sensing scene classification.

Hyperspectral image (HSI) clustering is a challenging task due to the complex characteristics in HSI data, such as spatial-spectral structure, high-dimension, and large spectral variability. In this paper, we propose a novel deep spatial-spectral subspace clustering network (DS3C-Net), which explores spatial-spectral information via the multi-scale auto-encoder and collaborative constraint. Considering the structure correlations of HSI, the multi-scale auto-encoder is first designed to extract spatial-spectral features with different-scale pixel blocks which are selected as the inputs. Then, the collaborative constrained self-expressive layers are introduced between the encoder and decoder, to capture the self-expressive subspace structures. By designing a self-expressiveness similarity constraint, the proposed network is trained collaboratively, and the affinity matrices of the feature representation are learned in an end-to-end manner. Based on the affinity matrices, the spectral clustering algorithm is utilized to obtain the final HSI clustering result. Experimental results on three widely used hyperspectral image datasets demonstrate that the proposed method outperforms state-of-the-art methods.

Convolutional neural network (CNN) has demonstrated to be a powerful tool for hyperspectral images (HSIs) classification. Previous CNN-based HSI classification methods only adopt the fixed-size patches to train the CNN model, and such single scale patches may not reflect the complex spatial structural information in the HSIs. In addition, although different layers of CNN can extract features of multiple scales, the traditional CNN model can only utilize features from the highest level for the classification task. These features, however, do not fully consider the strong complementary yet correlated information among different layers. To address these issues, in this paper, a multiscale densely-connected convolutional network (MS-DenseNet) framework is proposed to sufficiently exploit multiple scales information for the HSIs classification. Specifically, for each pixel, the MS-DenseNet, first, extracts its surrounding patches of multiple scales. These patches can separately constitute multiple scale training and testing samples. Within each specific scale sample, instead of using the forward convolutional layers, the MS-DenseNet adopts the dense blocks, which can connect each layer to other layers in a feed-forward fashion and thus can exploit the information among different layers for training and testing. Furthermore, since high correlations exist in patches of different scales, the MS-DenseNet introduces several dense blocks to fuse the multiscale information among different layers for the final HSI classification. Experimental results on several real HSIs demonstrate the superiority of the proposed MS-DenseNet over single scale-based CNN classification model and several well-known classification methods.

Recently, automatic diagnostic approaches have been widely used to classify ocular diseases. Most of these approaches are based on a single imaging modality (e.g., fundus photography or optical coherence tomography (OCT)), which usually only reflect the oculopathy to a certain extent, and neglect the modality-specific information among different imaging modalities. This paper proposes a novel modality-specific attention network (MSAN) for multi-modal retinal image classification, which can effectively utilize the modality-specific diagnostic features from fundus and OCT images. The MSAN comprises two attention modules to extract the modality-specific features from fundus and OCT images, respectively. Specifically, for the fundus image, ophthalmologists need to observe local and global pathologies at multiple scales (e.g., from microaneurysms at the micrometer level, optic disc at millimeter level to blood vessels through the whole eye). Therefore, we propose a multi-scale attention module to extract both the local and global features from fundus images. Moreover, large background regions exist in the OCT image, which is meaningless for diagnosis. Thus, a region-guided attention module is proposed to encode the retinal layer-related features and ignore the background in OCT images. Finally, we fuse the modality-specific features to form a multi-modal feature and train the multi-modal retinal image classification network. The fusion of modality-specific features allows the model to combine the advantages of fundus and OCT modality for a more accurate diagnosis. Experimental results on a clinically acquired multi-modal retinal image (fundus and OCT) dataset demonstrate that our MSAN outperforms other well-known single-modal and multi-modal retinal image classification methods.

Due to its noninvasive character, optical coherence tomography (OCT) has become a popular diagnostic method in clinical settings. However, the low-coherence interferometric imaging procedure is inevitably contaminated by heavy speckle noise, which impairs both visual quality and diagnosis of various ocular diseases. Although deep learning has been applied for image denoising and achieved promising results, the lack of well-registered clean and noisy image pairs makes it impractical for supervised learning-based approaches to achieve satisfactory OCT image denoising results. In this paper, we propose an unsupervised OCT image speckle reduction algorithm that does not rely on well-registered image pairs. Specifically, by employing the ideas of disentangled representation and generative adversarial network, the proposed method first disentangles the noisy image into content and noise spaces by corresponding encoders. Then, the generator is used to predict the denoised OCT image with the extracted content features. In addition, the noise patches cropped from the noisy image are utilized to facilitate more accurate disentanglement. Extensive experiments have been conducted, and the results suggest that our proposed method is superior to the classic methods and demonstrates competitive performance to several recently proposed learning-based approaches in both quantitative and qualitative aspects. Code is available at: https://github.com/tsmotlp/DRGAN-OCT.

As the labeled anomalous medical images are usually difficult to acquire, especially for rare diseases, the deep learning based methods, which heavily rely on the large amount of labeled data, cannot yield a satisfactory performance. Compared to the anomalous data, the normal images without the need of lesion annotation are much easier to collect. In this paper, we propose an anomaly detection framework, namely [Formula: see text], extracting [Formula: see text]elf-supervised and tr [Formula: see text]ns [Formula: see text]ation-consistent features for [Formula: see text]nomaly [Formula: see text]etection. The proposed SALAD is a reconstruction-based method, which learns the manifold of normal data through an encode-and-reconstruct translation between image and latent spaces. In particular, two constraints (i.e., structure similarity loss and center constraint loss) are proposed to regulate the cross-space (i.e., image and feature) translation, which enforce the model to learn translation-consistent and representative features from the normal data. Furthermore, a self-supervised learning module is engaged into our framework to further boost the anomaly detection accuracy by deeply exploiting useful information from the raw normal data. An anomaly score, as a measure to separate the anomalous data from the healthy ones, is constructed based on the learned self-supervised-and-translation-consistent features. Extensive experiments are conducted on optical coherence tomography (OCT) and chest X-ray datasets. The experimental results demonstrate the effectiveness of our approach.

In this paper, in order to extract efficient spectral-spatial features for hyperspectral image classification, a Gaussian curvature (GC) filter based feature extraction method with multiscale segmentation constraint is proposed. The method consists of the following major steps: First, the maximum noise fraction (MNF) method is applied on the hyperspectral images (HSIs) to reduce the noise and computational complexity. The GC features are extracted from the dimension reduced HSIs via the GC filter. Next, a multiscale segmentation strategy is applied on the HSIs, and the multiscale spatial features are extracted by applying the weighted mean operations within and among superpixels. Finally, the GC features and dimension reduced multiscale spatial features are fused to form the final multiscale Gaussian curvature features (MGCFs) for classification purposes. To verify the effectiveness of the proposed method, we conduct experiments on the Indian Pines data set. Experimental results demonstrate that the proposed method can significantly improve the classification accuracies compared to several standard classification methods.

Fully supervised methods for hyperspectral image (HSI) classification usually require a considerable number of training samples to obtain high classification accuracy. However, it is time-consuming and difficult to collect the training samples. Under this context, semisupervised learning, which can effectively augment the number of training samples and extract the underlying information among the unlabeled samples, gained much attention. In this letter, we propose a Multiscale convolutional neural networks (CNNs) Ensemble Based Self-Learning (MCE-SL) method for semisupervised HSI classification. Generally, the proposed MCE-SL method consists of the following two stages. In the first stage, the spatial information of different scales from limited labeled training samples are extracted to train several CNN models. In the second stage, the trained multiscale CNNs are used to classify the unlabeled samples. After error correction, the problem of label partially incorrect is alleviated, and unlabeled samples with high confidence will be added to the original training data set for the next training iteration. We conduct comprehensive experiments on two real HSI data sets, and the experimental results show that the proposed MCE-SL can obtain better classification performance compared with several traditional semisupervised methods in few iterations.

Recently, deep learning has been demonstrated to be an effective tool to detect changes in bitemporal remote sensing images. However, most existing methods based on deep learning obtain the ultimate change map by analyzing the difference image (DI) or the stacked feature vectors of input images, which cannot sufficiently capture the relationship between the two input images to obtain the change information. In this letter, a new method named bilinear convolutional neural networks (BCNNs) is proposed to detect changes in bitemporal multispectral images. The model can be trained end to end with two symmetric convolutional neural networks (CNNs), which are capable of learning the feature representation from bitemporal images and utilizing the relations between the two input images by a linear outer product operation in an effective way. Specifically, two sets of patches obtained from two multispectral images of different times are first input into two CNNs to extract deep features, respectively. Then, the matrix outer product is applied on the output feature maps to obtain the combined bilinear features. Finally, the ultimate change detected result can be produced by applying the softmax classifier on the combined features. Experimental results on real multispectral data sets demonstrate the superiority of the proposed method over several well-known change-detection approaches.

Due to the limitation of the satellite sensor, it is difficult to acquire a high-resolution (HR) multispectral (HRMS) image directly. The aim of pansharpening (PNN) is to fuse the spatial in panchromatic (PAN) with the spectral information in multispectral (MS). Recently, deep learning has drawn much attention, and in the field of remote sensing, several pioneering attempts have been made related to PNN. However, the big size of remote sensing data will produce more training samples, which require a deeper neural network. Most current networks are relatively shallow and raise the possibility of detail loss. In this letter, we propose a residual encoder-decoder conditional generative adversarial network (RED-cGAN) for PNN to produce more details with sharpened images. The proposed method combines the idea of an autoencoder with generative adversarial network (GAN), which can effectively preserve the spatial and spectral information of the PAN and MS images simultaneously. First, the residual encoder-decoder module is adopted to extract the multiscale features from the last step to yield pansharpened images and relieve the training difficulty caused by deepening the network layers. Second, to further enhance the performance of the generator to preserve more spatial information, a conditional discriminator network with the input of PAN and MS images is proposed to encourage that the estimated MS images share the same distribution as that of the referenced HRMS images. The experiments conducted on the Worldview2 (WV2) and Worldview3 (WV3) images demonstrate that our proposed method provides better results than several state-of-the-art PNN methods.

Combining a high-spatial-resolution multispectral image (HR-MSI) with a low-spatial-resolution hyperspectral image (LR-HSI) has become a common way to enhance the spatial resolution of the HSI. The existing state-of-the-art LR-HSI and HR-MSI fusion methods are mostly based on the matrix factorization, where the matrix data representation may be hard to fully make use of the inherent structures of 3-D HSI. We propose a nonlocal sparse tensor factorization approach, called the NLSTF_SMBF, for the semiblind fusion of HSI and MSI. The proposed method decomposes the HSI into smaller full-band patches (FBPs), which, in turn, are factored as dictionaries of the three HSI modes and a sparse core tensor. This decomposition allows to solve the fusion problem as estimating a sparse core tensor and three dictionaries for each FBP. Similar FBPs are clustered together, and they are assumed to share the same dictionaries to make use of the nonlocal self-similarities of the HSI. For each group, we learn the dictionaries from the observed HR-MSI and LR-HSI. The corresponding sparse core tensor of each FBP is computed via tensor sparse coding. Two distinctive features of NLSTF_SMBF are that: 1) it is blind with respect to the point spread function (PSF) of the hyperspectral sensor and 2) it copes with spatially variant PSFs. The experimental results provide the evidence of the advantages of the NLSTF_SMBF method over the existing state-of-the-art methods, namely, in semiblind scenarios.

Most of the existing hyperspectral anomaly detectors are designed based on a single pixel-level feature. These detectors may not adequately utilize spectral-spatial information in hyperspectral images (HSIs) for detecting anomalies. To overcome this problem, this article introduces a novel subpixel-pixel-superpixel guided fusion (SPSGF) method for hyperspectral anomaly detection. This approach comprises three main steps. First, subpixel-, pixel-, and superpixel-level features are extracted from an HSI by employing the spectral unmixing, morphological operation, and superpixel segmentation techniques, respectively. Then, based on the spatial consistency of three features, a guided filtering-based weight optimization technique is developed to construct weight maps for fusion. Finally, a simple yet effective decision fusion method is adopted to utilize the complemental information of three features, and then generates a fused detection result. The performance of the proposed approach is evaluated on three real-scene HSIs and one synthetic HSI. Experimental results validate the advantages of the SPSGF method.

The convolutional neural networks (CNNs) have recently demonstrated to be a powerful tool for object detection. However, with the complex scenes in remote sensing images, feature extraction of the object in the CNN will be seriously affected by background information. To address this issue, in this article, a region-enhanced CNN (RECNN) is proposed for the object detection of remote sensing images. The RECNN introduces the saliency constraint and multilayer fusion strategy into the CNN model, which can effectively enhance the object regions for better detection. Specifically, the saliency map is extracted and utilized to guide the training of the proposed model to strengthen saliency regions in feature maps. In addition, since different layers can reflect the object regions in varied resolutions, a multilayer fusion strategy is introduced to connect different convolutional layers and explore the context, where the feature maps of object regions are further enhanced. Experimental results on a publicly available ten-class object detection data set demonstrate the superiority of the RECNN over several competitive object detection methods.

This paper proposes a novel end-to-end learning model, called skip-connected covariance (SCCov) network, for remote sensing scene classification (RSSC). The innovative contribution of this paper is to embed two novel modules into the traditional convolutional neural network (CNN) model, i.e., skip connections and covariance pooling. The advantages of newly developed SCCov are twofold. First, by means of the skip connections, the multi-resolution feature maps produced by the CNN are combined together, which provides important benefits to address the presence of large-scale variance in RSSC data sets. Second, by using covariance pooling, we can fully exploit the second-order information contained in such multi-resolution feature maps. This allows the CNN to achieve more representative feature learning when dealing with RSSC problems. Experimental results, conducted using three large-scale benchmark data sets, demonstrate that our newly proposed SCCov network exhibits very competitive or superior classification performance when compared with the current state-of-the-art RSSC techniques, using a much lower amount of parameters. Specifically, our SCCov only needs 10% of the parameters used by its counterparts.

Convolutional neural networks (CNNs) have recently shown remarkable performance in remote sensing scene image classification. However, long-range dependencies (e.g. non-local similarities) within the scene are often ignored by CNNs. To address this issue, in this paper we develop a new high-order self-attention network (HoSA) for remote sensing scene classification. Specifically, we embed two novel modules, i.e., a self-attention module and a high order pooling module, into off-the-shelf CNN models and then fine-tune the whole network. The advantages of our newly proposed HoSA network are twofold. Firstly, with the self-attention module, the HoSA can capture long-range dependencies within the scenes for high-level semantic feature extraction. Secondly, by means of its high-order pooling mechanism, our newly developed HoSA can further explore high-order information contained in the features. Our experiments with a widely used remote sensing scene data set demonstrate that the proposed HoSA network exhibits better classification performance than the baseline and several well-known methods.

Recently, deep learning has been recognized as a powerful tool to extract hierarchical features of hyperspectral images (HSIs). The existing deep learning-based methods exploit label information of land classes as the supervised information to train deep networks. However, considering that HSIs exhibit very complex spectral-spatial characteristic, e.g., the large intraclass variations and small interclass variations, these semantic information (i.e., label information)-based deep networks may not effectively cope with the above problem. In this letter, we propose a novel deep model, named deep hashing neural network (DHNN), to learn similarity-preserving deep features (SPDFs) for HSI classification. First, a well-pretrained network is introduced to simultaneously extract features of a pair of input samples. Second, a novel hashing layer is inserted after the last fully connected layer to transfer the real-value features into binary features, which can significantly speed up the computation for feature distance. Then, a loss function is elaborately designed to minimize the feature distance of similar pairs and maximize the feature distance of dissimilar pairs in Hamming space. Finally, the SPDF extracted by propagating the samples through the trained DHNN are fed into a support vector machine (SVM) classifier for HSI classification. Experimental results on two real HSIs demonstrate that the proposed feature extraction method in conjunction with a linear SVM classifier outperforms other feature extraction methods and competitive classifiers.

Optical coherence tomography (OCT) enables instant and direct imaging of morphological retinal tissue and has become an essential imaging modality for ophthalmology diagnosis. As one of the important morphological retinal characteristics, the structural information of retinal layers provides meaningful diagnostic information and is closely related to several retinal diseases. In this letter, we propose a novel layer guided convolutional neural network (LGCNN) to identify normal retina and three common types of macular pathologies, namely, diabetic macular edema, drusen, and choroidal neovascularization. Specifically, an efficient segmentation network is first employed to generate the retinal layer segmentation maps, which can delineate two lesion-related retinal layers associated with the meaningful retinal lesions. Then, two well-designed subnetworks in LGCNN are utilized to integrate the information of two lesion-related layers. Consequently, LGCNN can efficiently focus on the meaningful lesion-related layer regions to improve OCT classification. The experimental results conducted on two clinically acquired datasets demonstrate the effectiveness of the proposed method.

Tensor completion (TC), aiming to recover original high-order data from its degraded observations, has recently drawn much attention in hyperspectral images (HSIs) domain. Generally, the widely used TC methods formulate the rank minimization problem with a convex trace norm penalty, which shrinks all singular values equally, and may generate a much biased solution. Besides, these TC methods assume the whole high-order data is of low-rank, which may fail to recover the detail information in high-order data with diverse and complex structures. In this paper, a novel nonlocal low-rank regularization-based TC (NLRR-TC) method is proposed for HSIs, which includes two main steps. In the first step, an initial completion result is generated by the proposed low-rank regularization-based TC (LRR-TC) model, which combines the logarithm of the determinant with the tensor trace norm. This model can more effectively approximate the tensor rank, since the logarithm function values can be adaptively tuned for each input. In the second step, the nonlocal spatial-spectral similarity is integrated into the LRR-TC model, to obtain the final completion result. Specifically, the initial completion result is first divided into groups of nonlocal similar cubes (each group forms a 3-D tensor), and then the LRR-TC is applied to each group. Since similar cubes within each group contain similar structures, each 3-D tensor should have low-rank property, and thus further improves the completion result. Experimental results demonstrate that the proposed NLRR-TC method outperforms state-of-the-art HSIs completion techniques.

A convolutional neural network (CNN) has recently demonstrated its outstanding capability for the classification of hyperspectral images (HSIs). Typical CNN-based methods usually adopt image patches as inputs to the network. However, a fixed-size image patch in HSI with complex spatial contexts may contain multiple ground objects of different classes, which will deteriorate the classification performance of the CNN. In addition, traditional convolutional layers adopted in the CNN have a huge amount of parameters needed to be tuned, which will cause high computational cost. To address the above-mentioned issues, a novel squeeze multibias network (SMBN) is proposed for HSI classification. Specifically, the proposed SMBN first introduces the multibias module (MBM), which incorporates multibias into the rectified linear unit layers. The MBM can decouple the feature maps of input patches into multiple response maps (corresponding to different ground objects) and adaptively select the meaningful maps for classification. Furthermore, the proposed SMBN replaces the traditional convolutional layer with a squeeze convolution module, which can greatly reduce the number of parameters in the network, thus saving the running time, while still maintaining high classification accuracy. Experimental results on three real HSIs demonstrate the superiority of the proposed SMBN method over several state-of-the-art classification approaches.

The classification of hyperspectral images (HSIs) using convolutional neural networks (CNNs) has recently drawn significant attention. However, it is important to address the potential overfitting problems that CNN-based methods suffer when dealing with HSIs. Unlike common natural images, HSIs are essentially three-order tensors which contain two spatial dimensions and one spectral dimension. As a result, exploiting both spatial and spectral information is very important for HSI classification. This paper proposes a new hand-crafted feature extraction method, based on multiscale covariance maps (MCMs), that is specifically aimed at improving the classification of HSIs using CNNs. The proposed method has the following distinctive advantages. First, with the use of covariance maps, the spatial and spectral information of the HSI can be jointly exploited. Each entry in the covariance map stands for the covariance between two different spectral bands within a local spatial window, which can absorb and integrate the two kinds of information (spatial and spectral) in a natural way. Second, by means of our multiscale strategy, each sample can be enhanced with spatial information from different scales, increasing the information conveyed by training samples significantly. To verify the effectiveness of our proposed method, we conduct comprehensive experiments on three widely used hyperspectral data sets, using a classical 2-D CNN (2DCNN) model. Our experimental results demonstrate that the proposed method can indeed increase the robustness of the CNN model. Moreover, the proposed MCMs+2DCNN method exhibits better classification performance than other CNN-based classification strategies and several standard techniques for spectral-spatial classification of HSIs.

Hyperspectral image (HSI) classification has become a hot topic in the field of remote sensing. In general, the complex characteristics of hyperspectral data make the accurate classification of such data challenging for traditional machine learning methods. In addition, hyperspectral imaging often deals with an inherently nonlinear relation between the captured spectral information and the corresponding materials. In recent years, deep learning has been recognized as a powerful feature-extraction tool to effectively address nonlinear problems and widely used in a number of image processing tasks. Motivated by those successful applications, deep learning has also been introduced to classify HSIs and demonstrated good performance. This survey paper presents a systematic review of deep learning-based HSI classification literatures and compares several strategies for this topic. Specifically, we first summarize the main challenges of HSI classification which cannot be effectively overcome by traditional machine learning methods, and also introduce the advantages of deep learning to handle these problems. Then, we build a framework that divides the corresponding works into spectral-feature networks, spatial-feature networks, and spectral-spatial-feature networks to systematically review the recent achievements in deep learning-based HSI classification. In addition, considering the fact that available training samples in the remote sensing field are usually very limited and training deep networks require a large number of samples, we include some strategies to improve classification performance, which can provide some guidelines for future studies on this topic. Finally, several representative deep learning-based classification methods are conducted on real HSIs in our experiments.

Fine-tuning of pretrained convolutional neural networks (CNNs) has been proven to be an effective strategy for remote sensing image scene classification, particularly when a limited number of labeled data sets are available for training purposes. However, such a fine-tuning process often needs that the input images are resized into a fixed size to generate input vectors of the size required by fully connected layers (FCLs) in the pretrained CNN model. Such a resizing process often discards key information in the scenes and thus deteriorates the classification performance. To address this issue, in this paper, we introduce a scale-free CNN (SF-CNN) for remote sensing scene classification. Specifically, the FCLs in the CNN model are first converted into convolutional layers, which not only allow the input images to be of arbitrary sizes but also retain the ability to extract discriminative features using a traditional sliding-window-based strategy. Then, a global average pooling (GAP) layer is added after the final convolutional layer so that input images of arbitrary size can be mapped to feature maps of uniform size. Finally, we utilize the resulting feature maps to create a new FCL that is fed to a softmax layer for final classification. Our experimental results conducted using several real data sets demonstrate the superiority of the proposed SF-CNN method over several well-known classification methods, including pretrained CNN-based ones.

Natural images often contain patches with high similarity. In this paper, to effectively utilize the local and nonlocal self-similarity for low-rank models, we propose a novel weighted tensor rank-1 decomposition method (terms as WTR1) for nonlocal image denoising. Although the low-rank approximation problem has been well studied for matrices, it remains elusive of the theoretically extension to tensors due to the NPhard tensor decomposition. To tackle this problem, the proposed WTR1 method designs a new efficient CANDECOMP/PARAFAC (CP) decomposition algorithm and constructs a straightforward low-rank tensor approximation strategy. This is achieved by elegantly manipulating the CP-rank, called intrinsic low-rank tensor approximation. Specifically, the WTR1 method first groups similar patches into a 3-D stack and converts the stack into a finite sum of rank-1 products. Then, we deploy the intrinsic low-rank tensor approximation to produce the final denoised image. The proposed WTR1 method can jointly exploit the local and nonlocal self-similarity, thus improving the nonlocal image denoising quality. Experimental results have shown that the proposed WTR1 outperforms several state-of-the-art denoising methods.

Automatic and accurate classification of retinal optical coherence tomography (OCT) images is essential to assist ophthalmologist in the diagnosis and grading of macular diseases. Clinically, ophthalmologists usually diagnose macular diseases according to the structures of macular lesions, whose morphologies, size, and numbers are important criteria. In this paper, we propose a novel lesion-aware convolutional neural network (LACNN) method for retinal OCT image classification, in which retinal lesions within OCT images are utilized to guide the CNN to achieve more accurate classification. The LACNN simulates the ophthalmologists' diagnosis that focuses on local lesion-related regions when analyzing the OCT image. Specifically, we first design a lesion detection network to generate a soft attention map from the whole OCT image. The attention map is then incorporated into a classification network to weight the contributions of local convolutional representations. Guided by the lesion attention map, the classification network can utilize the information from local lesion-related regions to further accelerate the network training process and improve the OCT classification. Our experimental results on two clinically acquired OCT datasets demonstrate the effectiveness and efficiency of the proposed LACNN method for retinal OCT image classification.

Hyperspectral images (HSIs) with high spectral resolution only have the low spatial resolution. On the contrary, multispectral images (MSIs) with much lower spectral resolution can be obtained with higher spatial resolution. Therefore, fusing the high-spatial-resolution MSI (HR-MSI) with low-spatial-resolution HSI of the same scene has become the very popular HSI super-resolution scheme. In this paper, a novel low tensor-train (TT) rank (LTTR)-based HSI super-resolution method is proposed, where an LTTR prior is designed to learn the correlations among the spatial, spectral, and nonlocal modes of the nonlocal similar high-spatial-resolution HSI (HR-HSI) cubes. First, we cluster the HR-MSI cubes as many groups based on their similarities, and the HR-HSI cubes are also clustered according to the learned cluster structure in the HR-MSI cubes. The HR-HSI cubes in each group are much similar to each other and can constitute a 4-D tensor, whose four modes are highly correlated. Therefore, we impose the LTTR constraint on these 4-D tensors, which can effectively learn the correlations among the spatial, spectral, and nonlocal modes because of the well-balanced matricization scheme of TT rank. We formulate the super-resolution problem as TT rank regularized optimization problem, which is solved via the scheme of alternating direction method of multipliers. Experiments on HSI data sets indicate the effectiveness of the LTTR-based method.

Remotely sensed hyperspectral images (HSIs) usually have high spectral resolution but low spatial resolution. A way to increase the spatial resolution of HSIs is to solve a fusion inverse problem, which fuses a low spatial resolution HSI (LR-HSI) with a high spatial resolution multispectral image (HR-MSI) of the same scene. In this paper, we propose a novel HSI super-resolution approach (called LRSR), which formulates the fusion problem as the estimation of a spectral dictionary from the LR-HSI and the respective regression coefficients from both images. The regression coefficients are estimated by formulating a variational regularization problem which promotes local (in the spatial sense) low-rank and sparse regression coefficients. The local regions, where the spectral vectors are low-rank, are estimated by segmenting the HR-MSI. The formulated convex optimization is solved with SALSA. Experiments provide evidence that LRSR is competitive with respect to the state-of-the-art methods.

In this paper, a covariance matrix based feature fusion (CMF-F) framework is proposed to combine two low-level visual features i.e., the Gabor feature and color feature for scene classification. Generally, the proposed method consists of following three steps. Firstly, the Gabor feature and color feature are extracted from original image and stacked together. Then, a covariance matrix is extracted to fuse these two low-level visual features. Each nondiagonal entry in the covariance matrix stands for the correlation of two different feature dimensions. Finally, the obtained covariance matrix is handled by a kernel linear discriminative analysis algorithm followed with nearest neighboring classifier for label assignment. The proposed method is tested on a public 21-classes UC Merced land use data set and compared with mid-level visual feature oriented method and the high-level feature oriented methods. The experimental results demonstrate that the proposed CMFF framework can not only improve the classification performance of the low-level visual feature (the Gabor feature and the color feature), but also can outperform the conventional mid-level visual feature oriented methods.

In this paper, a novel fusion framework, which is called as subpixel, pixel and superpixel fusion based anomaly detection (SPSF-AD), is proposed for hyperspectral anomaly detection. Most existing methods are based on single pixel-level data representation, which fail to fully exploit spectral-spatial information in hyperspectral image for anomaly detection. For the proposed SPSF-AD method, the first step is to separately extract subpixel, pixel, and superpixel features from HSI via the spectral unmixing, morphological operation, and super-pixel segmentation technique. Then, three sparse anomalies detection results based on different features can be generated via low-rank decomposition (LRD) technique. Finally, since the different features contain highly complementary information, an effective decision fusion technique, i.e., average operator, is employed to estimate the final anomaly detection result. Experiments on three real HSI data sets can demonstrate the effectiveness of the proposed method.

Recently, deep learning has been introduced to extract hierarchical features of hyperspectral images (HSls) and achieved good classification performance. However, the previous deep learning based methods only consider the semantic information of individual pixel, which cannot effectively deal with the complex spectral-spatial characteristic of HSls. In this paper, we propose a novel deep learning based framework to learn the similarity-preserving deep features (SPDF) for HSI classification. Specifically, we firstly introduce a deep network that can take pairs of image patches as training samples, and then a loss function is elaborately designed to minimize the feature distance of similar pairs and maximize the feature distance of dissimilar pairs in feature space. Once the deep network is well trained, the SPDF can be obtained by propagating the samples through the trained network. Finally, these features are fed into the support vector machines (SVM) for HSI classification. Experimental results demonstrate the pro-nosed method outperforms other competitive methods.

Recently, robust principal component analysis (RPCA) based methods have been used for hyperspectral images (HSIs) restoration to simultaneously remove several types of noise, including Gaussian noise, impulse noise, stripes, and so on. However, most of these RPCA methods formulate the optimization problem with a convex l1-norm penalty, which over-penalizes large entries of vectors, and results in a biased solution. In this paper, a novel nonconvex sparsity regularizer (NonSR) for measuring the clean HSI low rank structure and noise sparsity structure is proposed, which can effectively approximate rank function and noise sparsity instead of the convex l1-norm. By embedding the sparsity regularizer into the RPCA framework, we formulate a new model, which enhance the capability in simultaneously removing several types of noise. In addition, an iterative algorithm based on the alternative direction multiplier method (ADMM) is developed to effectively solve the proposed model. Experimental results demonstrate that the proposed NonSR method outperforms state-of-the-art HSIs restoration techniques.

Convolutional neural networks (CNNs) have recently been demonstrated to be a powerful tool for hyperspectral image (HSI) classification, since they adopt deep convolutional layers whose kernels can effectively extract high-level spatial-spectral features. However, sampling locations of traditional convolutional kernels are fixed and cannot be changed according to complex spatial structures in HSIs. In addition, the typical pooling layers (e.g., average or maximum operations) in CNNs are also fixed and cannot be learned for feature downsampling in an adaptive manner. In this letter, a novel deformable CNN-based HSI classification method is proposed, which is called deformable HSI classification networks (DHCNet). The proposed network, DHCNet, introduces the deformable convolutional sampling locations, whose size and shape can be adaptively adjusted according to HSIs' complex spatial contexts. Specifically, to create the deformable sampling locations, 2-D offsets are first calculated for each pixel of input images. The sampling locations of each pixel with calculated offsets can cover the locations of other neighboring pixels with similar characteristics. With the deformable sampling locations, deformable feature images are then created by compressing neighboring similar structural information of each pixel into fixed grids. Therefore, applying the regular convolutions on the deformable feature images can reflect complex structures more effectively. Moreover, instead of adopting the pooling layers, the strided convolution is further introduced on the feature images, which can be learned for feature downsampling according to spatial contexts. Experimental results on two real HSI data sets demonstrate that DHCNet can obtain better classification performance than can several well-known classification methods.

An extinction profile (EP) is an effective spatial-spectral feature extraction method for hyperspectral images (HSIs), which has recently drawn much attention. However, the existing methods utilize the EPs in a stacking way, which is hard to fully explore the information in EPs for HSI classification. In this paper, a novel fusion framework termed EPs-fusion (EPs-F) is proposed to exploit the information within and among EPs for HSI classification. In general, EPs-F includes the following two stages. In the first stage, by extracting the EPs from three independent components of an HSI, three complementary groups of EPs can be constructed. For each EP, an adaptive superpixel-based composite kernel strategy is proposed to explore the spatial information within an EP. The weights to create the composite kernel and the number of superpixels are automatically determined based on the spatial information of each EP. In the second stage, since the different EPs contain highly complementary information, a simple yet effective decision fusion method is further applied to obtain the final classification result. Experiments on three real HSI data sets verify the qualitative and quantitative superiority of the proposed EPs-F method over several state-of-the-art HSI classifiers.

In this paper, a novel local covariance matrix (CM) representation method is proposed to fully characterize the correlation among different spectral bands and the spatial-contextual information in the scene when conducting feature extraction (FE) from hyperspectral images (HSIs). Specifically, our method first projects the HSI into a subspace, using the maximum noise fraction method. Then, for each test pixel in the subspace, its most similar neighboring pixels (within a local spatial window) are clustered using the cosine distance measurement. The test pixel and its neighbors are used to calculate a local CM for FE purposes. Each nondiagonal entry in the matrix characterizes the correlation between different spectral bands. Finally, these matrices are used as spatial-spectral features and fed to a support vector machine for classification purposes. The proposed method offers a new strategy to characterize the spatial-spectral information in the HSI prior to classification. Experimental results have been conducted using three publicly available hyperspectral data sets for classification, indicating that the proposed method can outperform several state-of-the-art techniques, especially when the training samples available are limited.

Sparse representation (SR) has been successfully used in the classification of hyperspectral images (HSIs) by representing HSI pixels over a dictionary and yielding discriminative sparse coefficients. Most of SR-based classification methods construct the dictionary by directly using some labeled pixels as atoms. Such dictionary can lead to inefficient SR for large-sized HSIs, and may be incomplete when the number of labeled pixels is less than the number of spectral bands. This paper proposes a contextual online dictionary learning (DL) method for HSIs classification, which learns a dictionary over the whole image rather than few labeled pixels. The proposed method can effectively and efficiently improve the adaptive representation capability of different pixels with an online learning mechanism. Specifically, the contextual characteristics of the HSI are integrated with discriminative spectral information for online DL, i.e., pushing similar pixels in neighborhood to share similar sparse coefficients with respect to the well-learned dictionary. By this way, the obtained sparse coefficients are structured and discriminative. Finally, a traditional classifier, i.e., the linear support vector machine, is applied to the sparse coefficients, and the final classification results are obtained. Experimental results on real HSIs show the effectiveness of the proposed method.

This paper proposes a new method, called multilayer stacked covariance pooling (MSCP), for remote sensing scene classification. The innovative contribution of the proposed method is that it is able to naturally combine multilayer feature maps, obtained by pretrained convolutional neural network (CNN) models. Specifically, the proposed MSCP-based classification framework consists of the following three steps. First, a pretrained CNN model is used to extract multilayer feature maps. Then, the feature maps are stacked together, and a covariance matrix is calculated for the stacked features. Each entry of the resulting covariance matrix stands for the covariance of two different feature maps, which provides a natural and innovative way to exploit the complementary information provided by feature maps coming from different layers. Finally, the extracted covariance matrices are used as features for classification by a support vector machine. The experimental results, conducted on three challenging data sets, demonstrate that the proposed MSCP method can not only consistently outperform the corresponding single-layer model but also achieve better classification performance than other pretrained CNN-based scene classification methods.

Recently, deep learning has been introduced to classify hyperspectral images (HSIs) and achieved good performance. In general, deep models adopt a large number of hierarchical layers to extract features. However, excessively increasing network depth will result in some negative effects (e.g., overfitting, gradient vanishing, and accuracy degrading) for conventional convolutional neural networks. In addition, the previous networks used in HSI classification do not consider the strong complementary yet correlated information among different hierarchical layers. To address the above two issues, a deep feature fusion network (DFFN) is proposed for HSI classification. On the one hand, the residual learning is introduced to optimize several convolutional layers as the identity mapping, which can ease the training of deep network and benefit from increasing depth. As a result, we can build a very deep network to extract more discriminative features of HSIs. On the other hand, the proposed DFFN model fuses the outputs of different hierarchical layers, which can further improve the classification accuracy. Experimental results on three real HSIs demonstrate that the proposed method outperforms other competitive classifiers.

Fusing a low spatial resolution hyperspectral image (LR-HSI) with a high spatial resolution multispectral image (HR-MSI) to obtain a high spatial resolution hyperspectral image (HR-HSI) has attracted increasing interest in recent years. In this paper, we propose a coupled sparse tensor factorization (CSTF) based approach for fusing such images. In the proposed CSTF method, we consider an HR-HSI as a three-dimensional tensor and redefine the fusion problem as the estimation of a core tensor and dictionaries of the three modes. The high spatial-spectral correlations in the HR-HSI are modeled by incorporating a regularizer which promotes sparse core tensors. The estimation of the dictionaries and the core tensor are formulated as a coupled tensor factorization of the LR-HSI and of the HR-MSI. Experiments on two remotely sensed HSIs demonstrate the superiority of the proposed CSTF algorithm over current state-of-the-art HSI-MSI fusion approaches.

Hyperspectral image (HSI) sharpening, which aims at fusing an observable low spatial resolution (LR) HSI (LR-HSI) with a high spatial resolution (HR) multispectral image (HR-MSI) of the same scene to acquire an HR-HSI, has recently attracted much attention. Most of the recent HSI sharpening approaches are based on image priors modeling, which are usually sensitive to the parameters selection and time-consuming. This paper presents a deep HSI sharpening method (named DHSIS) for the fusion of an LR-HSI with an HR-MSI, which directly learns the image priors via deep convolutional neural network-based residual learning. The DHSIS method incorporates the learned deep priors into the LR-HSI and HR-MSI fusion framework. Specifically, we first initialize the HR-HSI from the fusion framework via solving a Sylvester equation. Then, we map the initialized HR-HSI to the reference HR-HSI via deep residual learning to learn the image priors. Finally, the learned image priors are returned to the fusion framework to reconstruct the final HR-HSI. Experimental results demonstrate the superiority of the DHSIS approach over existing state-of-the-art HSI sharpening approaches in terms of reconstruction accuracy and running time.

Hyperspectral image (HSI) super-resolution, which fuses a low-resolution (LR) HSI with a high-resolution (HR) multispectral image (MSI), has recently attracted much attention. Most of the current HSI super-resolution approaches are based on matrix factorization, which unfolds the three-dimensional HSI as a matrix before processing. In general, the matrix data representation obtained after the matrix unfolding operation makes it hard to fully exploit the inherent HSI spatial-spectral structures. In this paper, a novel HSI super-resolution method based on non-local sparse tensor factorization (called as the NLSTF) is proposed. The sparse tensor factorization can directly decompose each cube of the HSI as a sparse core tensor and dictionaries of three modes, which reformulates the HSI super-resolution problem as the estimation of sparse core tensor and dictionaries for each cube. To further exploit the non-local spatial self-similarities of the HSI, similar cubes are grouped together, and they are assumed to share the same dictionaries. The dictionaries are learned from the LR-HSI and HR-MSI for each group, and corresponding sparse core tensors are estimated by spare coding on the learned dictionaries for each cube. Experimental results demonstrate the superiority of the proposed NLSTF approach over several state-of-the-art HSI super-resolution approaches.

Sparse representation (SR) based hyperspectral image (HSI) classification is a rapidly evolving research topic. How to construct an optimized dictionary to better characterize spectral-spatial features of HSI is an important problem. In this paper, a novel spectral-spatial online dictionary learning (SSODL) method for HSI classification is proposed. The main idea is to learn a complete and discriminative dictionary by exploiting both spatial and spectral information all over the whole image. Rather than only using training samples for dictionary construction, the online dictionary learning (ODL) mechanism can effectively improve the adaptive representation capability of different pixels. Specifically, the contextual characteristics of HSI are integrated with discriminative spectral information for the ODL, i.e., pushing similar pixels in neighborhood to share similar sparse coefficients w.r.t. the well learnt dictionary. By this way, the yielding sparse coefficients are structured and discriminative. Finally, a traditional classifier, i.e., linear support vector mechine (SVM), is applied to the sparse coefficients and the final classification results are obtained. Experimental results on real HSIs show the effectiveness of the proposed method.

Extinction profile (EP) is an effective feature extraction method which can well preserve the geometrical characteristics of a hyperspectral image (HSI) and by extracting the EP from first three independent components (ICs) of an HSI, three correlated and complementary groups of EP features can be constructed. In this paper, an EPs fusion (EPs-F) strategy is proposed for HSI classification by exploring spatial-spectral information within and among three EP features. In general, the EPs-F method includes two stages. In the first stage, within each EP feature, a superpixel-based composite kernel strategy is proposed to adaptively fuse the spatial information of EP and the spectral feature of HSI. Then, the obtained adaptive composite kernel is used to create a classification map for each EP. In the second stage, decision fusion is further applied on different classification maps to create the final classification result. Experiments on two real HSIs verify the effectiveness of the proposed EPs-F algorithm.

Sparse representation (SR) can transform spectral signatures of hyperspectral pixels into sparse coefficients with very few nonzero entries, which can efficiently be used for compression. In this paper, a spectral-spatial adaptive SR (SSASR) method is proposed for hyperspectral image (HSI) compression by taking advantage of the spectral and spatial information of HSIs. First, we construct superpixels, i.e., homogeneous regions with adaptive sizes and shapes, to describe HSIs. Since homogeneous regions usually consist of similar pixels, pixels within each superpixel will be similar and share similar spectral signatures. Then, the spectral signatures of each superpixel can be simultaneously coded in the SR model to exploit their joint sparsity. Since different superpixels generally have different performances of SR, their rate-distortion performances in the sparse coding will be different. To achieve the best possible overall rate-distortion performance, an adaptive coding scheme is introduced to adaptively assign distortions to superpixels. Finally, the obtained sparse coefficients are quantized and entropy coded and constitute the final bitstream with the coded superpixel map. The experimental results over several HSIs show that the proposed SSASR method outperforms some state-of-the-art HSI compression methods in terms of the rate-distortion and spectral fidelity performances.

Supervised classification of hyperspectral images (HSI) is a very challenging task due to the existence of noisy and mixed spectral characteristics. Recently, the widely developed spectral unmixing techniques offer the possibility to extract spectral mixture information at a subpixel level, which can contribute to the categorization of seriously mixed spectral pixels. Besides, it has been demonstrated that the discrimination between different materials will be improved by integrating the geometry and structure information, which can be derived from the variance between neighboring pixels. Furthermore, by incorporating the spatial context, the superpixel-based spectral-spatial similarity information can be used to smooth classification results in homogeneous regions. Therefore, a novel fusion framework for HSI classification that combines subpixel, pixel, and superpixel-based complementary information is proposed in this paper. Here, both feature fusion and decision fusion schemes are introduced. For the feature fusion scheme, the first step is to extract subpixel-level, pixel-level, and superpixel-level features from HSI, respectively. Then, the multiple feature-induced kernels are fused to form one composite kernel, which is incorporated with a support vector machine (SVM) classifier for label assignment. For the decision fusion scheme, class probabilities based on three different features are estimated by the probabilistic SVM classifier first. Then, the class probabilities are adaptively fused to form a probabilistic decision rule for classification. Experimental results tested on different real HSI images can demonstrate the effectiveness of the proposed fusion schemes in improving discrimination capability, when compared with the classification results relied on each individual feature.

We demonstrate the usefulness of utilizing a segmentation step for improving the performance of sparsity based image reconstruction algorithms. In specific, we will focus on retinal optical coherence tomography (OCT) reconstruction and propose a novel segmentation based reconstruction framework with sparse representation, termed segmentation based sparse reconstruction (SSR). The SSR method uses automatically segmented retinal layer information to construct layer-specific structural dictionaries. In addition, the SSR method efficiently exploits patch similarities within each segmented layer to enhance the reconstruction performance. Our experimental results on clinical-grade retinal OCT images demonstrate the effectiveness and efficiency of the proposed SSR method for both denoising and interpolation of OCT images.

In this paper, a non-local based sparse representation (called as the NLSR) is proposed for the super-resolution of hyperspectral image. Specifically, the NLSR firstly uses the non-local Kmeans to partition pixels of low spatial resolution hyperspectral image into several classes. The non-local Kmeans can exploit the similar patterns and structures of the low spatial resolution image to enhance the efficiency of the sparse solution. Then, the sparse representation is independently applied on each class of the low resolution hyperspectral image and high spatial resolution multispectral image to obtain the high resolution hyperspectral image. Experimental results demonstrate the superiority of the proposed NLSR method over several well-known super-resolution methods.

In this paper, a decision fusion of pixel-level and superpixel-level classifiers (DFPSC) for the HSI is proposed. First, the support vector machine based classification probability combined with the local spatial information is introduced to classify the HSI in a pixel-by-pixel manner. Then, the HSI is over-segmented into non-overlapping superpixels. Each superpixel contains spatially-connected and spectrally-similar pixels, which are assigned to the same label via joint sparse regularization. Finally, a guided map is generated based on the edge map and superpixel map, which is used to guide the fusion of both pixel-level and superpixel-level classification results. With the proposed decision fusion scheme, the classification results in homogeneous and structural areas can be better balanced, leading to the improvement of the overall classification accuracy. The experimental results demonstrate the superiority of the proposed method over some well-known classification methods.

A novel hyperspectral image (HSI) classification method by the probabilistic fusion of pixel-level and superpixel-level classifiers is proposed. Generally, pixel-level classifiers based on spectral information only may generate “salt and pepper” result in the classification map since spatial correlation is not considered. By incorporating spatial information in homogeneous regions, the superpixel-level classifiers can effectively eliminate the noisy appearance. However, the classification accuracy will be deteriorated if undersegmentation cannot be fully avoided in superpixel-based approaches. Therefore, it is proposed to adaptively combine both the pixel-level and superpixel-level classifiers, to improve the classification performance in both homogenous and structural areas. In the proposed method, a support vector machine classifier is first applied to estimate the pixel-level class probabilities. Then, superpixel-level class probabilities are estimated based on a joint sparse representation. Finally, the two levels of class probabilities are adaptively combined in a maximum a posteriori estimation model, and the classification map is obtained by solving the maximum optimization problem. Experimental results on real HSI images demonstrate the superiority of the proposed method over several well-known classification approaches in terms of classification accuracy.

A novel set-to-set distance-based spectral-spatial classification method for hyperspectral images (HSIs) is proposed. In HSIs, the spatially connected and spectrally similar pixels within each homogeneous region can be considered as one set of test samples, i.e., a test set, which should belong to the same class. In addition, each class of labeled pixels can be regarded as one set of training samples, i.e., a training set. Therefore, it is a natural consideration in the proposed method to measure the similarity between test and training sets via specific set-based distance criteria and then decide the classification label for each test set, accordingly. Specifically, the superpixel algorithm-based oversegmentation technique jointly exploits both the spatial similarity and structural information to first divide the HSI into multiple perceptually uniform regions. As a result, each segmented region corresponds to one test set. Then, each test/training set is represented with an affine hull (AH) model, which utilizes both the similarity and variance of pixels within each set to adaptively characterize the set. Finally, the class label for each test set is determined based on the closest geometry distance between test and training AHs. Experimental results on real HSI data sets demonstrate the superiority of the proposed algorithm over several well-known classification approaches, in terms of classification accuracy and computational speed.

In this paper, a novel spectral–spatial adaptive sparse representation (SSASR) method is proposed for hyperspectral image (HSI) denoising. The proposed SSASR method aims at improving noise-free estimation for noisy HSI by making full use of highly correlated spectral information and highly similar spatial information via sparse representation, which consists of the following three steps. First, according to spectral correlation across bands, the HSI is partitioned into several nonoverlapping band subsets. Each band subset contains multiple continuous bands with highly similar spectral characteristics. Then, within each band subset, shape-adaptive local regions consisting of spatially similar pixels are searched in spatial domain. This way, spectral–spatial similar pixels can be grouped. Finally, the highly correlated and similar spectral–spatial information in each group is effectively used via the joint sparse coding, in order to generate better noise-free estimation. The proposed SSASR method is evaluated by different objective metrics in both real and simulated experiments. The numerical and visual comparison results demonstrate the effectiveness and superiority of the proposed method.

We propose a superpixel-based composite kernel framework for hyperspectral image (HSI) classification. Composite kernel methods can utilize both the spectral and the spatial information for the HSI classification. However, setting the optimal spatial neighborhood for different spatial structures is a non-trivial issue. In order to adaptively exploit the spatial contextual information, we utilize superpixel to obtain spatial information. A superpixel can be regarded as a local neighborhood, whose size and shape can be adaptively adjusted according to the spatial structures in the HSI. Then, the spatial features are extracted by computing the mean of the spectral pixels within each superpixel. Finally, composite kernel with support vector machine is implemented on real HSI. Experiments on two real HSIs demonstrate the outstanding performance of the proposed method.

Recently, the superpixel segmentation is introduced into the hyperspectral image (HSI) classification to exploit the spatial information. However, the size of superpixels influences the classification significantly because small superpixels can not provide enough spatial information and large superpixels generally result in error segmentation. The error segmentation is irreversible and intolerable, so the size of superpixels tends to be small. This paper proposes a hyperspectral unmixing based superpixel merging criterion to merge small su-perpixels and thus make use of the spatial information. The spatial information is then incorporated into the joint sparsity model for the spectral-spatial classification. Experimental results demonstrate the superiority of the proposed method over some widely used classification methods.

The 2014 data fusion contest organized by IGARSS 2014 has shown an example that dual images with quite different spatial resolutions may have quite different ground coverages. Specifically, due to the technology limit of optical sensor and the incomplete flight path, the high spatial resolution visible image used in the data fusion contest can only provide a sparse ground coverage. To fill the missing area in this visible image, a simple yet effective image completion method is introduced in this paper which consists of the following two steps: First, through performing patch matching on the hyperspectral image, the most common recurring offsets between patches in the known region and those in the missing region are estimated. Second, the pixels in the missing region is completed by fusing pixels of the shifted visible images (obtained using the above estimated offsets). Experimental results show that the proposed method performs much better than multiple general image completion methods for this data set.

For the classification of hyperspectral images (HSIs), this paper presents a novel framework to effectively utilize the spectral-spatial information of superpixels via multiple kernels, which is termed as superpixel-based classification via multiple kernels (SC-MK). In the HSI, each superpixel can be regarded as a shape-adaptive region, which consists of a number of spatial neighboring pixels with very similar spectral characteristics. First, the proposed SC-MK method adopts an oversegmentation algorithm to cluster the HSI into many superpixels. Then, three kernels are separately employed for the utilization of the spectral information, as well as spatial information, within and among superpixels. Finally, the three kernels are combined together and incorporated into a support vector machine classifier. Experimental results on three widely used real HSIs indicate that the proposed SC-MK approach outperforms several well-known classification methods.

A novel superpixel-based discriminative sparse model (SBDSM) for spectral-spatial classification of hyperspectral images (HSIs) is proposed. Here, a superpixel in a HSI is considered as a small spatial region whose size and shape can be adaptively adjusted for different spatial structures. In the proposed approach, the SBDSM first clusters the HSI into many superpixels using an efficient oversegmentation method. Then, pixels within each superpixel are jointly represented by a set of common atoms from a dictionary via a joint sparse regularization. The recovered sparse coefficients are utilized to determine the class label of the superpixel. In addition, instead of directly using a large number of sampled pixels as dictionary atoms, the SBDSM applies a discriminative K-SVD learning algorithm to simultaneously train a compact representation dictionary, as well as a discriminative classifier. Furthermore, by utilizing the class label information of training pixels and dictionary atoms, a class-labeled orthogonal matching pursuit is proposed to accelerate the K-SVD algorithm while still enforcing high discriminability on sparse coefficients when training the classifier. Experimental results on four real HSI datasets demonstrate the superiority of the proposed SBDSM algorithm over several well-known classification approaches in terms of both classification accuracies and computational speed.

In this paper, a novel feature extraction method based on intrinsic image decomposition (IID) is proposed for hyperspectral image classification. The proposed method consists of the following steps. First, the spectral dimension of the hyperspectral image is reduced with averaging-based image fusion. Then, the dimension reduced image is partitioned into several subsets of adjacent bands. Next, the reflectance and shading components of each subset are estimated with an optimization-based IID technique. Finally, pixel-wise classification is performed only on the reflectance components, which reflect the material-dependent properties of different objects. Experimental results show that, with the proposed feature extraction method, the support vector machine classifier is able to obtain much higher classification accuracy even when the number of training samples is quite small. This demonstrates that IID is indeed an effective way for feature extraction of hyperspectral images.

This paper introduces a novel spectral-spatial classification method for hyperspectral images based on extended random walkers (ERWs), which consists of two main steps. First, a widely used pixelwise classifier, i.e., the support vector machine (SVM), is adopted to obtain classification probability maps for a hyperspectral image, which reflect the probabilities that each hyperspectral pixel belongs to different classes. Then, the obtained pixelwise probability maps are optimized with the ERW algorithm that encodes the spatial information of the hyperspectral image in a weighted graph. Specifically, the class of a test pixel is determined based on three factors, i.e., the pixelwise statistics information learned by a SVM classifier, the spatial correlation among adjacent pixels modeled by the weights of graph edges, and the connectedness between the training and test samples modeled by random walkers. Since the three factors are all well considered in the ERW-based global optimization framework, the proposed method shows very good classification performances for three widely used real hyperspectral data sets even when the number of training samples is relatively small.

We present a novel general-purpose compression method for tomographic images, termed 3D adaptive sparse representation based compression (3D-ASRC). In this paper, we focus on applications of 3D-ASRC for the compression of ophthalmic 3D optical coherence tomography (OCT) images. The 3D-ASRC algorithm exploits correlations among adjacent OCT images to improve compression performance, yet is sensitive to preserving their differences. Due to the inherent denoising mechanism of the sparsity based 3D-ASRC, the quality of the compressed images are often better than the raw images they are based on. Experiments on clinical-grade retinal OCT images demonstrate the superiority of the proposed 3D-ASRC over other well-known compression methods.

This paper proposes a new spectral-spatial hyperspectral classification method named the shape-adaptive sparse representation (SASR). The fixed window is not suitable for all pixels of hyperspectral image (HSI) to search local similar regions. In order to overcome the drawback, we propose to apply the shape-adaptive algorithm to exploit the contextual spatial information of HSI. Furthermore, the hyperspectral classification is implemented by incorporating the spatial contextual information of HSI into the sparse representation classification model. Experimental results demonstrate the superiority of the proposed SASR method over both classical and state-of-the-art approaches.

Sparse representation has been demonstrated to be a powerful tool in classification of hyperspectral images (HSIs). The spatial context of an HSI can be exploited by first defining a local region for each test pixel and then jointly representing pixels within each region by a set of common training atoms (samples). However, the selection of the optimal region scale (size) for different HSIs with different types of structures is a nontrivial task. In this paper, considering that regions of different scales incorporate the complementary yet correlated information for classification, a multiscale adaptive sparse representation (MASR) model is proposed. The MASR effectively exploits spatial information at multiple scales via an adaptive sparse strategy. The adaptive sparse strategy not only restricts pixels from different scales to be represented by training atoms from a particular class but also allows the selected atoms for these pixels to be varied, thus providing an improved representation. Experiments on several real HSI data sets demonstrate the qualitative and quantitative superiority of the proposed MASR algorithm when compared to several well-known classifiers.

Remote sensing image fusion can integrate the spatial detail of panchromatic (PAN) image and the spectral information of a low-resolution multispectral (MS) image to produce a fused MS image with high spatial resolution. In this paper, a remote sensing image fusion method is proposed with sparse representations over learned dictionaries. The dictionaries for PAN image and low-resolution MS image are learned from the source images adaptively. Furthermore, a novel strategy is designed to construct the dictionary for unknown high-resolution MS images without training set, which can make our proposed method more practical. The sparse coefficients of the PAN image and low-resolution MS image are sought by the orthogonal matching pursuit algorithm. Then, the fused high-resolution MS image is calculated by combining the obtained sparse coefficients and the dictionary for the high-resolution MS image. By comparing with six well-known methods in terms of several universal quality evaluation indexes with or without references, the simulated and real experimental results on QuickBird and IKONOS images demonstrate the superiority of our method.

In this paper, we present a novel technique, based on compressive sensing principles, for reconstruction and enhancement of multi-dimensional image data. Our method is a major improvement and generalization of the multi-scale sparsity based tomographic denoising (MSBTD) algorithm we recently introduced for reducing speckle noise. Our new technique exhibits several advantages over MSBTD, including its capability to simultaneously reduce noise and interpolate missing data. Unlike MSBTD, our new method does not require an a priori high-quality image from the target imaging subject and thus offers the potential to shorten clinical imaging sessions. This novel image restoration method, which we termed sparsity based simultaneous denoising and interpolation (SBSDI), utilizes sparse representation dictionaries constructed from previously collected datasets. We tested the SBSDI algorithm on retinal spectral domain optical coherence tomography images captured in the clinic. Experiments showed that the SBSDI algorithm qualitatively and quantitatively outperforms other state-of-the-art methods.

In this letter, we propose an unsupervised approach for change detection in multitemporal satellite images based on a novel detail-enhancing algorithm. The multitemporal source images are first used to generate the difference image, which is decomposed into low-pass approximation and high-pass directional subbands by the nonsubsampled contourlet transform. The coefficients from the directional subbands are fused at intrascale and interscale to extract the meaningful details of the difference image. After that, the extracted details are injected into one base image selected from the approximation subbands, which results in a detail-enhanced difference image. For each pixel in the enhanced difference image, a dimension-reduced feature vector is created using the principal component analysis (PCA). The final change detection map is achieved by clustering the feature vectors using a PCA-guidedk-means algorithm into “changed” and “unchanged” classes. Experimental results demonstrate the superior performance of the proposed approach compared with several well-known change detection techniques.

In this paper, we propose a novel feature vector clustering method for unsupervised change detection in multitemporal satellite images. A feature vector for each pixel is extracted using the compressed sparse representation of the difference image which is obtained by comparing a pair of co-registered images acquired at different times on the same area. The compressed sparse representation is achieved by taking two stages: compressed sampling and sparse representation. The compressed sampling is first employed in order to reduce the dimensionality of the feature vectors. Then, the sparse representation is applied to extract the meaningful change information and to combat the noise interference. The final change detection is obtained by clustering the extracted feature vectors using k-means algorithm into “changed” and “unchanged” classes. Experimental results clearly show that the proposed approach consistently yields superior performance compared to several well-known change detection techniques on both noise-free and noisy satellite images.

This paper proposes a new learned dictionary for sparse representation of given data and suggests a way to apply it to non-local denoising. First, a quad-tree structure is efficiently embedded into a sparse dictionary model. This enables the dictionary to discover the complex structures in the given data and to be easily employed to high dimensional data. Besides, we propose a joint 3-D operation to exploit the correlations among the similar blocks, as the non-local denoising model assumes that there exist mutually similar blocks in nature images. This 3-D operation is achieved by a simple concatenation of the similar patches to a single vector and training the proposed dictionary on it. The experimental results indicate that our approach is competitive with several well known denoising techniques in terms of both PSNR and visual quality.