Due to the substantial domain gaps in Remote Sensing (RS) images that are characterized by variabilities such as location, wavelength, and sensor type, Remote Sensing Domain Generalization (RSDG) has emerged as a critical and valuable research frontier, focusing on developing models that generalize effectively across diverse scenarios. However, research in this area remains underexplored: (1) Current cross-domain methods primarily focus on Domain Adaptation (DA), which adapts models to predefined domains rather than to unseen ones; (2) Few studies target the RSDG issue, especially for semantic segmentation tasks. Existing related models are developed for specific unknown domains, struggling with issues of underfitting on other unseen scenarios; (3) Existing RS foundation models tend to prioritize in-domain performance over cross-domain generalization. To this end, we introduce the first vision foundation model for RSDG semantic segmentation, CrossEarth. CrossEarth demonstrates strong cross-domain generalization through a specially designed data-level Earth-Style Injection pipeline and a model-level Multi-Task Training pipeline. In addition, for the semantic segmentation task, we have curated an RSDG benchmark comprising 32 semantic segmentation scenarios across various regions, spectral bands, platforms, and climates, providing comprehensive evaluations of the generalizability of future RSDG models. Extensive experiments on this collection demonstrate the superiority of CrossEarth over existing state-of-the-art methods.

Fine-grained ship recognition in remote sensing imagery is essential for maritime applications. However, its development is hindered by two challenges: 1) the limited granularity of existing ship detection datasets, and 2) the disturbance of complex maritime conditions as well as the arbitrary ship orientations and distributions. To address the first issue, we annotated a large-scale fine-grained ship instance detection dataset (LAFI), comprising 48,717 ship instances worldwide with 49 categories. To tackle the challenges of marine disturbance and diverse ship status, we proposed a controllable generative knowledge-driven ship detection framework (COSD). It employs a controllable diffusion model guided by ship-marine textual prompt to generate millions of synthetic images that not only preserve ship structures but also cover diverse sea and weather conditions for robust pretraining. The pretraining stage then utilizes masked reconstruction to learn component-level cues under occlusion, clutter, fog, and illumination changes. Furthermore, a heterogeneous feature alignment decoder is designed to align multi-modal metrics of orientation and distribution features in the latent space, allowing for accurate representation of diverse ship status. Extensive experiments on two benchmark datasets showed that our method respectively increased 0.011 and 0.030 mean average precision (mAP@50) over SOTA methods, particularly in scenarios involving small, densely packed and arbitrary oriented ships.

Multi-temporal hyperspectral imagery (HSI) has become a powerful tool for change detection (CD) owing to its rich spectral signatures and detailed spatial information. Nevertheless, the application of paired HSIs is constrained by the scarcity of annotated training data. While unsupervised domain adaptation (UDA) offers a potential solution by transferring change detection knowledge from source to target domains, two critical limitations persist: 1) the labor-intensive process of acquiring and annotating source-domain paired samples, and 2) the suboptimal transfer performance caused by substantial cross-domain distribution discrepancies. To address these challenges, we present a Temporal Self-Construction Cross-Domain learning (TSCCD) framework for UDA-based HSI-CD. Our TSCCD framework introduces an innovative temporal self-construction mechanism that synthesizes bi-temporal source-domain data from existing HSI classification datasets while simultaneously performing initial data-level alignment. Furthermore, we develop a reweighted amplitude maximum mean discrepancy (MMD) metric to enhance feature-level domain adaptation. The proposed architecture incorporates an attention-based Kolmogorov-Arnold network (KAN) with high-frequency feature augmentation within an encoder-decoder structure to effectively capture change characteristics. Comprehensive experiments conducted on three benchmark HSI datasets demonstrate that TSCCD achieves superior performance compared to current state-of-the-art methods in HSI change detection tasks. Codes are available at https://github.com/Zhoutya/TSCCD.

Mamba has demonstrated exceptional performance in visual tasks due to its powerful global modeling capabilities and linear computational complexity, offering considerable potential in hyperspectral image super-resolution (HSISR). However, in HSISR, Mamba faces challenges as transforming images into 1D sequences neglects the spatial-spectral structural relationships between locally adjacent pixels, and its performance is highly sensitive to input order, which affects the restoration of both spatial and spectral details. In this paper, we propose HSRMamba, a contextual spatial-spectral modeling state space model for HSISR, to address these issues both locally and globally. Specifically, a local spatial-spectral partitioning mechanism is designed to establish patch-wise causal relationships among adjacent pixels in 3D features, mitigating the local forgetting issue. Furthermore, a global spectral reordering strategy based on spectral similarity is employed to enhance the causal representation of similar pixels across both spatial and spectral dimensions. Finally, experimental results demonstrate our HSRMamba outperforms the state-of-the-art methods in quantitative quality and visual results. Code is available at: https://github.com/Tomchenshi/HSRMamba.

Spiking neural networks (SNNs) are emerging as a promising alternative to traditional artificial neural networks (ANNs), offering biological plausibility and energy efficiency. Despite these merits, SNNs are frequently hampered by limited capacity and insufficient representation power, yet remain underexplored in remote sensing image (RSI) super-resolution (SR) tasks. In this paper, we first observe that spiking signals exhibit drastic intensity variations across diverse textures, highlighting an active learning state of the neurons. This observation motivates us to apply SNNs for efficient SR of RSIs. Inspired by the success of attention mechanisms in representing salient information, we devise the spiking attention block (SAB), a concise yet effective component that optimizes membrane potentials through inferred attention weights, which, in turn, regulates spiking activity for superior feature representation. Our key contributions include: 1) we bridge the independent modulation between temporal and channel dimensions, facilitating joint feature correlation learning, and 2) we access the global self-similar patterns in large-scale remote sensing imagery to infer spatial attention weights, incorporating effective priors for realistic and faithful reconstruction. Building upon SAB, we proposed SpikeSR, which achieves state-of-the-art performance across various remote sensing benchmarks such as AID, DOTA, and DIOR, while maintaining high computational efficiency. Code of SpikeSR will be available at https://github.com/XY-boy/SpikeSR.

Accurate hyperspectral image (HSI) interpretation is critical for providing valuable insights into various earth observation-related applications such as urban planning, precision agriculture, and environmental monitoring. However, existing HSI processing methods are predominantly task-specific and scene-dependent, which severely limits their ability to transfer knowledge across tasks and scenes, thereby reducing the practicality in real-world applications. To address these challenges, we present HyperSIGMA, a vision transformer-based foundation model that unifies HSI interpretation across tasks and scenes, scalable to over one billion parameters. To overcome the spectral and spatial redundancy inherent in HSIs, we introduce a novel sparse sampling attention (SSA) mechanism, which effectively promotes the learning of diverse contextual features and serves as the basic block of HyperSIGMA. HyperSIGMA integrates spatial and spectral features using a specially designed spectral enhancement module. In addition, we construct a large-scale hyperspectral dataset, HyperGlobal-450K, for pre-training, which contains about 450 K hyperspectral images, significantly surpassing existing datasets in scale. Extensive experiments on various high-level and low-level HSI tasks demonstrate HyperSIGMA's versatility and superior representational capability compared to current state-of-the-art methods. Moreover, HyperSIGMA shows significant advantages in scalability, robustness, cross-modal transferring capability, real-world applicability, and computational efficiency.

Our understanding of the temporal dynamics of the Earth's surface has been significantly advanced by deep vision models, which often require a massive amount of labeled multi-temporal images for training. However, collecting, preprocessing, and annotating multi-temporal remote sensing images at scale is non-trivial since it is expensive and knowledge-intensive. In this paper, we present scalable multi-temporal change data generators based on generative models, which are cheap and automatic, alleviating these data problems. Our main idea is to simulate a stochastic change process over time. We describe the stochastic change process as a probabilistic graphical model, namely the generative probabilistic change model (GPCM), which factorizes the complex simulation problem into two more tractable sub-problems, i.e., condition-level change event simulation and image-level semantic change synthesis. To solve these two problems, we present Changen2, a GPCM implemented with a resolution-scalable diffusion transformer which can generate time series of remote sensing images and corresponding semantic and change labels from labeled and even unlabeled single-temporal images. Changen2 is a "generative change foundation model" that can be trained at scale via self-supervision, and is capable of producing change supervisory signals from unlabeled single-temporal images. Unlike existing "foundation models", our generative change foundation model synthesizes change data to train task-specific foundation models for change detection. The resulting model possesses inherent zero-shot change detection capabilities and excellent transferability. Comprehensive experiments suggest Changen2 has superior spatiotemporal scalability in data generation, e.g., Changen2 model trained on 256 pixel single-temporal images can yield time series of any length and resolutions of 1,024 pixels. Changen2 pre-trained models exhibit superior zero-shot performance (narrowing the performance gap to 3% on LEVIR-CD and approximately 10% on both S2Looking and SECOND, compared to fully supervised counterpart) and transferability across multiple types of change tasks, including ordinary and off-nadir building change, land-use/land-cover change, and disaster assessment.

Integrating low-resolution hyperspectral images with high-resolution multispectral images is an effective approach to derive high-resolution hyperspectral images. Recently, numerous deep learning-based approaches have been employed to model the mapping relationships for the fusion directly. However, these methods often neglect the spectral characteristics and fail to facilitate comprehensive interactions among global features from heterogeneous modalities. In this paper, we propose a novel cyclic Transformer based on the cross-modality spatial-spectral interaction, exploiting diverse interaction modes to explore the similarity and complementarity among cross-modality features. Specifically, we design a cyclic interactive architecture to fully exploit the abundant spectral prior information in low-resolution hyperspectral images and the rich spatial prior information in high-resolution multispectral images. By incorporating spatial and spectral priors into the attention mechanisms in Transformer modules, we explore the long-range dependency information within the cross-modality features. Furthermore, to enhance interaction among features from different modalities, we devise the cross-modality adaptive interaction mechanisms in both spatial and spectral dimensions to facilitate information reciprocity between different modalities. Extensive experiments demonstrate that the proposed approach outperforms the state-of-the-art fusion methods both quantitatively and visually. The code is available athttps://github.com/Tomchenshi/CYformer.

Hyperspectral images (HSIs) often suffer from various noises, such as Gaussian noise, stripe noise, impulse noise, and deadlines due to the influence of sensors and external environments. These noises significantly degrade the quality of HSI and hinder subsequent applications. While most current transformer-based methods can effectively remove certain types of noise, they struggle with wide stripe noise. In addition, transformers are typically applied within local windows due to the limitation of computational complexity. Although windowshifting operations enhance the interaction between windows to a certain extent, this interaction remains insufficient for comprehensive global modeling. In light of these limitations, we propose a Bidirectional-aware network combining Transformer and Mamba (BTMnet), which consists of Bidirectional Long-Short Distance Attention (BLSDA) and Channel-Split Mamba (CSM). To better remove wide stripe noise, BLSDA is designed with two rectangular windows adapted to wide stripes in both vertical and horizontal directions, utilizing transformers to compute attention relationships within windows and across different windows. To further integrate global information and enhance the interaction of features between adjacent windows, CSM extracts global features by scanning in four directions across different feature channels. In BLSDA, we applied bidirectional windows in vertical and horizontal directions, and in CSM, we conducted bidirectional scanning in vertical and horizontal directions. The combination of these techniques allows for the simultaneous extraction of bidirectional features from HSI. By evaluating the metrics and visualization, the experimental results on simulated and real experiments prove that our method can achieve better results.

Hyperspectral unmixing aims to resolve the prevalent issue of mixed pixels in hyperspectral imagery and serves as an effective technique for sub-pixel level image interpretation. Recent years have seen the emergence of advanced unmixing algorithms that integrate both spatial and spectral information. However, existing methods mainly focus on spatial context and lack depth in modeling spatial correlations. Both relevant and irrelevant spatial information is introduced into the spectral mixing model for local pixels, with the irrelevant information acting as noise that impacts the unmixing accuracy. To address these challenges, we propose an advanced spectral mixing model, Adaptive Correlation Refined Network (ACR-Net) which integrates refined spatial correlation based on self-attention. A key component of ACR-Net is the Adaptive Correlation Aggregated Decoder (ACAD), which extracts affinity information from the encoder’s feature map and adaptively amplifies the influence of highly correlated regions in the unmixing process. Additionally, the Composite Active Spatial Attention (CASA) Module emphasizes unique spectral characteristics across bands, improving spatial distribution representation and enabling more accurate abundance estimation. We conducted abundant evaluations on six datasets, including three real-world and two synthetic hyperspectral unmixing datasets as well as a large benchmark classification dataset. Extensive experiments have demonstrated that the proposed algorithm can achieve exceptional or comparable unmixing results among various state-of-the-art algorithms. Moreover, the proposed method achieved optimal classification performance on the classic PaviaU dataset, indicating its strong potential for hyperspectral classification task.

Due to the difficulty of obtaining labeled data for hyperspectral images (HSIs), cross-scene classification has emerged as a widely adopted approach in the remote sensing community. It involves training a model using labeled data from a source domain (SD) and unlabeled data from a target domain (TD), followed by inference on the TD. However, variations in the reflectance spectrum of the same object between the SD and the TD, as well as differences in the feature distribution of the same land cover class, pose significant challenges to the performance of cross-scene classification. To address this issue, we propose a dual classification head self-training network (DHSNet). This method aligns class-wise features across domains, ensuring that the trained classifier can accurately classify TD data of different classes. We introduce a dual classification head self-training strategy for the first time in the cross-scene HSI classification field and design a self-training loss based on the prediction of the two classification heads. The proposed approach mitigates the domain gap while preventing the accumulation of incorrect pseudo-labels in the model. Additionally, we incorporate a novel central feature attention mechanism to enhance the model’s capacity to learn scene-invariant features across domains. DHSNet significantly outperforms state-of-the-art methods on three cross-scene HSI datasets, achieving 80.23±1.92% OA on the Houston dataset. The code for DHSNet will be available at https://github.com/liurongwhm.

Image super-resolution (SR) is an effective way to enhance the spatial resolution and detail information of remote sensing images to obtain a superior visual quality. As SR is severely ill-conditioned, effective image priors are necessary to regularize the solution space and generate the corresponding high-resolution (HR) image. In this article, we propose a novel gradient-guided multiframe SR (MFSR) framework for remote sensing imagery reconstruction. The framework integrates a learned gradient prior as the regularization term into a model-based optimization method. Specifically, the local gradient regularization (LGR) prior is derived from the deep residual attention network (DRAN) through gradient profile transformation (GPT). The nonlocal total variation (NLTV) prior is characterized using the spatial structure similarity of the gradient patches with the maximum a posteriori (MAP) model. The modeled prior performs well in preserving edge smoothness and suppressing visual artifacts, while the learned prior is effective in enhancing sharp edges and recovering fine structures. By incorporating the two complementary priors into an adaptive norm-based reconstruction framework, the mixed L1 and L2 regularization minimization problem is optimized to achieve the required HR remote sensing image. Extensive experimental results on remote sensing data demonstrate that the proposed method can produce visually pleasant images and is superior to several of the state-of-the-art SR algorithms in terms of the quantitative evaluation.

Disasters occur continuously around the world, causing significant damage to human life and property. In practice, there are inevitably situations that require the use of heterogeneous remote sensing images, for instant optical and synthetic aperture radar (SAR) images, for change detection (CD) and disaster recognition. However, the significant differences between heterogeneous remote sensing images make direct comparison difficult. Although some methods use image translation (IT) to reduce these differences, most employ a two-stage ”translation followed by change detection” strategy, which can lead to feature degradation. Furthermore, these methods often require adversarial training or complex loss functions that are sensitive to hyperparameters. This paper proposes a change detection network for optical and SAR images, named Diffpurifier. First, optical images are translated into SAR images using pre-trained denoising diffusion probabilistic models (DDPMs) and ordinary differential equations (ODEs) while simultaneously extracting multi-scale features. Then, change detection is performed under superpixel enhancement to improve the homogeneity of the change detection maps. Diffpurifier not only integrates image translation and feature extraction, simplifying the workflow, but also maintains high accuracy, stable training, and generalization to different types of data without the need for additional translation constraints. In comparative experiments on four public datasets, Diffpurifier outperforms the second-best method by an average of approximately 5% in terms of F1-score, validating the effectiveness and robustness of the method.

Detection of targets for hyperspectral images (HSIs) persists as a fundamental task in remote sensing image processing. Exploring the discriminative ability of deep Siamese networks to distinguish targets from backgrounds is the mainstream method for target detection in HSIs. Nevertheless, these methods enhance the discriminative ability of networks by learning the separability distance between the target and the overall backgrounds, where the backgrounds are considered as a single category. As a result, they may struggle to effectively suppress backgrounds with solely subtle spectral differences from the target, resulting in a limited separability performance, and the inability to accurately detect the targets. To alleviate this problem, we propose a novel subtle spectral difference discriminative deep metric learning-based target detector for HSIs (denoted as S2D3ML) in this work. The proposed S2D3ML constructs a deep metric learning framework embedded with a discriminative constraint to learn a deep metric feature space for addressing limited separability, in which the subtle feature differences between targets and different ground objects can be distinguished. In addition, we investigate a new multi-block sparse representation score-based strategy to obtain sufficient samples and spectral centers of backgrounds for training the S2D3ML framework. Finally, the detection of targets is executed within the learned metric space. A comprehensive suite of experiments is rigorously conducted on four benchmark datasets, and the results indicate that the S2D3ML achieves superior performance in HSIs target detection.

Hyperspectral video (HSV) provides rich spectral-spatial-temporal information, enabling the capture of complex object dynamics beyond the limitations of conventional single- and multi-modal tracking. However, current HSV tracking methods face challenges such as data scarcity, band gaps, spectral fragmentation, temporal underutilization, and high computational load, which constrain performance. In this article, we present SpectralTrack, a novel HSV tracking framework with spectral-spatial fusion and memory enhancement. SpectralTrack incorporates an explicit visual prompting module to mitigate band gaps and spectral fragmentation. We further introduce an extraction-matching-interaction module, which leverages a template-bridging search adapter and a multi-layer perceptron adapter within a multi-modal Transformer architecture for efficient cross-modal feature extraction-matching-interaction. Additionally, a memory perception module enhances state reasoning by injecting temporal prompts to refine spectral and spatial cues. SpectralTrack follows parameter-efficient fine-tuning and feature-level fusion to alleviate data scarcity and reduce computational overhead. We instantiate two variants, SpectralTrack and SpectralTrack+, across nine HSV tracking datasets, demonstrating superior effectiveness over extensive trackers. Implementations and results will be available at https://github.com/YZCU/SpectralTrack.

Weakly-Supervised Change Detection (WSCD) aims to distinguish specific object changes (e.g., objects appearing or disappearing) from background variations (e.g., environmental changes due to light, weather, or seasonal shifts) in paired satellite images, relying only on paired image (i.e., image-level) classification labels. This technique significantly reduces the need for dense annotations required in fully-supervised change detection. However, as image-level supervision only indicates whether objects have changed in a scene, WSCD methods often misclassify background variations as object changes, especially in complex remote-sensing scenarios. In this work, we propose an Adversarial Class Prompting (AdvCP) method to address this co-occurring noise problem, including two phases: a) Adversarial Prompt Mining: After each training iteration, we introduce adversarial prompting perturbations, using incorrect one-hot image-level labels to activate erroneous feature mappings. This process reveals co-occurring adversarial samples under weak supervision, namely background variation features that are likely to be misclassified as object changes. b) Adversarial Sample Rectification: We integrate these adversarially prompt-activated pixel samples into training by constructing an online global prototype. This prototype is built from an exponentially weighted moving average of the current batch and all historical training data. Serving as an unbiased anchor, the global prototype guides the rectification of adversarial pixel samples. Our AdvCP can be seamlessly integrated into current WSCD methods without adding additional inference cost. Experiments on ConvNet, Transformer, and Segment Anything Model (SAM)-based baselines demonstrate significant performance enhancements, achieving up to 7.37%, 7.46%, and 6.56% IoU improvements on the WHU-CD, LEVIR-CD, and DSIFN-CD datasets. Furthermore, we demonstrate the generalizability of AdvCP to other multi-class weakly-supervised dense prediction scenarios. Code is available at https://github.com/zhenghuizhao/AdvCP.

Recently, there has been a surge in interest in Large Language Models (LLMs), with ChatGPT standing out for its exceptional capabilities in language comprehension, reasoning, and interactive communication. These models have garnered attention from a diverse array of users and researchers across various disciplines. While LLMs have demonstrated remarkable proficiency in mimicking human task execution through natural language, their application in remote sensing interpretation remains largely uncharted. Furthermore, the current lack of automation in remote sensing task planning limits the accessibility of these sophisticated interpretation techniques, especially for non-specialists in the field. To bridge this gap, we introduce Remote Sensing ChatGPT, an innovative LLM-driven agent that integrates ChatGPT with a suite of AI-powered remote sensing models to tackle complex interpretation challenges. This system is designed to interpret user requests, delineate task planning based on the functionalities required, execute each subtask sequentially, and compile the final output by synthesizing the results from each stage. Given that LLMs, trained predominantly on natural language, do not inherently comprehend visual elements present in remote sensing imagery, we have devised a method to incorporate visual cues, effectively embedding the visual context of remote sensing images into the ChatGPT framework. With Remote Sensing ChatGPT, users can effortlessly submit a remote sensing image alongside their query and promptly receive detailed interpretation outcomes along with comprehensive linguistic feedback. Experiments and case studies demonstrate that our method is adept at handling a diverse range of remote sensing tasks and has the potential to be expanded to encompass an even wider array of applications with the integration of more advanced models, such as remote sensing foundation model. The code and demo of Remote Sensing ChatGPT is publicly available at https://github.com/HaonanGuo/Remote-Sensing-ChatGPT.

The escalating intensity and frequency of floods, exacerbated by global climate change, emphasize the urgent need to address the growing risks of floods. Rapid and precise flood detection is paramount for efficiently responding to emergencies and executing disaster relief measures, enabling swift reactions to flood disasters and minimizing the losses incurred thus. The 2024 IEEE GRSS Data Fusion Contest Track 1 is centered on leveraging multi-source remote sensing data, particularly synthetic aperture radar (SAR) data, to classify flood and non-flood areas. In this contest, we acknowledge the significance of managing uncertain predictions and present an efficient Uncertainty-Aware Fusion Network (UAFNet). Specifically, we build on the traditional encoder-decoder architecture, initially employing the pyramid visual transformer (PVT) as a feature extractor. Subsequently, we apply a typical decoding strategy, namely the feature pyramid network, to obtain a flood extraction map with relatively high uncertainty. Furthermore, leveraging the uncertain extraction map, we introduce an Uncertainty Rank Algorithm to quantify the uncertainty level of each pixel of the foreground and background. We seamlessly integrate this algorithm with our proposed Uncertainty-Aware Fusion Module, enabling level-by-level feature refinement and ultimately yielding a refined extraction map with minimal uncertainty. Employing the proposed UAFNet, we utilize diverse versions of PVT as encoders to train multiple UAFNets. Additionally, we enhance our approach with online testing augmentation and multi-model fusion strategy, aiming to enhance the final flood extraction accuracy. Our technical solution has exhibited outstanding performance, earning the first-place ranking in the 2024 IEEE GRSS Data Fusion Contest Track 1 and achieving an impressive F1 score of 82.985% on the official test set.

Crop mapping is an important task for agriculture-related activities and economic development. Most present studies focus on the crop mapping of staple crops, i.e., soybean, maize, and wheat, few concentrate on the multi-level fine-grained crop classification, which requires knowing finer crop type, i.e., barley or rye, winter wheat or spring wheat for different application. Deep learning methods with the strong ability to extract features automatically have great potential in fine-grained crop mapping. However, the classification of multi-level finer crops is challenged by the extremely similar phenological characters. In this paper, a multi-level fine-grained crop classification method based on multi-expert knowledge distill is proposed to learn the phenological features with different distinction degrees. Specifically, it uses three expert models to distinguish crop types with obvious, similar, and confusing phenological features. Then through a self-paced learning module, the student model first learns the knowledge from three expert models in the early stage and then learns to excavate the phenological features actively during the learning process. The experiment was carried on in Nordrhein Westfalen, Germany based on the EUROCROPS and time-series Sentinel-2 dataset and achieved great performance compared with popular deep learning methods.

As global climate change worsens, floods are becoming more severe and frequent, urgently demanding effective flood risk mitigation strategies. Timely and precise flood inundation mapping is crucial for emergency response and relief. The 2024 IEEE GRSS Data Fusion Contest Track 2 aims to pioneer innovative algorithms for accurate flood extraction using multi-source optical remote sensing (RS) data. However, data diversity introduces aleatoric uncertainty, especially with synthetic, non-real data. Meanwhile, the vast coverage of RS imagery and the small proportion of flood areas cause a significant class imbalance, leading to epistemic uncertainty. In this paper, we propose an Uncertainty-aware Detail-Preserving Network (UADPNet) for rapid flood mapping of multi-source optical data. Firstly, we design an Aleatoric Uncertainty Estimator to model aleatoric uncertainty in multi-source data. Secondly, we introduce a Multi-Scale Convolution Block to extract multi-scale information without downsampling. Thirdly, we utilize a multi-level supervised strategy to quantify epistemic uncertainty and highlight uncertain pixels via the Uncertainty-Aware Fusion Module. With UADPNet, we adopt a multi-model fusion and post-processing strategy to enhance the final flood extraction accuracy. Our outstanding experimental results in the official test set showcase the superiority of our method, which secured the first-place ranking in the 2024 IEEE GRSS Data Fusion Contest Track 2, boasting an impressive F1 score of 89.843%.

Recent studies demonstrate the effectiveness of deep learning (DL) methods in crop mapping using Satellite Image Time Series (SITS). Despite this progress, the existing crop mapping methods often neglect prior knowledge, leading to unsatisfactory performance, especially in scenarios with limited data. To address this limitation, this paper introduces the Knowledge-Guided Crop Mapping Network (KGCMNet) as an innovative solution for crop mapping. Beyond a spatiotemporal encoder and a crop type prediction decoder, KGCMNet incorporates a knowledge reconstruction (KR) module for introducing essential crop-related prior knowlege. The KR module, utilizing NDVI as the chosen knowledge, aids the encoder in learning discriminative phenology from the SITS data. Experimental results demonstrate the outperformance of KGCMNet compared to other methods, showcasing the effectiveness of KR module in improving the accuracy across various crop types.

Visual foundation models have achieved remarkable results in zero-shot image classification and segmentation, but zero-shot change detection remains an open problem. In this paper, we propose the segment any change models (AnyChange), a new type of change detection model that supports zero-shot prediction and generalization on unseen change types and data distributions. AnyChange is built on the segment anything model (SAM) via our training-free adaptation method, bitemporal latent matching. By revealing and exploiting intra-image and inter-image semantic similarities in SAM's latent space, bitemporal latent matching endows SAM with zero-shot change detection capabilities in a training-free way. We also propose a point query mechanism to enable AnyChange's zero-shot object-centric change detection capability. We perform extensive experiments to confirm the effectiveness of AnyChange for zero-shot change detection. AnyChange sets a new record on the SECOND benchmark for unsupervised change detection, exceeding the previous SOTA by up to 4.4\% F$_1$ score, and achieving comparable accuracy with negligible manual annotations (1 pixel per image) for supervised change detection. Code is available at https://github.com/Z-Zheng/pytorch-change-models.

Optical-flow-based and kernel-based approaches have been extensively explored for temporal compensation in satellite Video Super-Resolution (VSR). However, these techniques are less generalized in large-scale or complex scenarios, especially in satellite videos. In this paper, we propose to exploit the well-defined temporal difference for efficient and effective temporal compensation. To fully utilize the local and global temporal information within frames, we systematically modeled the short-term and long-term temporal discrepancies since we observe that these discrepancies offer distinct and mutually complementary properties. Specifically, we devise a Short-term Temporal Difference Module (S-TDM) to extract local motion representations from RGB difference maps between adjacent frames, which yields more clues for accurate texture representation. To explore the global dependency in the entire frame sequence, a Long-term Temporal Difference Module (L-TDM) is proposed, where the differences between forward and backward segments are incorporated and activated to guide the modulation of the temporal feature, leading to a holistic global compensation. Moreover, we further propose a Difference Compensation Unit (DCU) to enrich the interaction between the spatial distribution of the target frame and temporal compensated results, which helps maintain spatial consistency while refining the features to avoid misalignment. Rigorous objective and subjective evaluations conducted across five mainstream video satellites demonstrate that our method performs favorably against state-of-the-art approaches. Code will be available athttps://github.com/XY-boy/LGTD.

Aircraft recognition is crucial in both civil and military fields, and high-spatial resolution remote sensing has emerged as a practical approach. However, existing data-driven methods fail to locate discriminative regions for effective feature extraction due to limited training data, leading to poor recognition performance. To address this issue, we propose a knowledge-driven deep learning method called the explicable aircraft recognition framework based on a part parsing prior (APPEAR). APPEAR explicitly models the aircraft's rigid structure as a pixel-level part parsing prior, dividing it into five parts: 1) the nose; 2) left wing; 3) right wing; 4) fuselage; and 5) tail. This fine-grained prior provides reliable part locations to delineate aircraft architecture and imposes spatial constraints among the parts, effectively reducing the search space for model optimization and identifying subtle interclass differences. A knowledge-driven aircraft part attention (KAPA) module uses this prior to achieving a geometric-invariant representation for identifying discriminative features. Part features are generated by part indexing in a specific order and sequentially embedded into a compact space to obtain a fixed-length representation for each part, invariant to aircraft orientation and scale. The part attention module then takes the embedded part features, adaptively reweights their importance to identify discriminative parts, and aggregates them for recognition. The proposed APPEAR framework is evaluated on two aircraft recognition datasets and achieves superior performance. Moreover, experiments with few-shot learning methods demonstrate the robustness of our framework in different tasks. Ablation analysis illustrates that the fuselage and wings of the aircraft are the most effective parts for recognition.

Hyperspectral (HS) video is able to capture abundant spectral, spatial, and temporal information about objects, which overcomes the limitations of common red-green-blue (RGB) video in complex scenarios such as similar appearances and background clutters (BCs). However, most trackers apply hand-crafted features extracted from manually selected bands instead of deep features for object representations due to limited HS data and the band gap problem. Each HS image consists of many bands, and it is challenging to fully interact with the band information while maintaining tracking speed. To this end, this article proposes a novel end-to-end spectral awareness interaction network with a dynamic template (SPIRIT) for HS video object tracking. First, a spectral awareness module (SAM) is proposed to learn band contributions with consideration of nonlinear and global interactions between HS bands. It can also cooperate with the feature extraction module pretrained with RGB data to attenuate the band gap and data-hungry. Second, an interaction module (IM) is proposed to achieve inter and intraband feature interactions to enhance tracking performance while improving efficiency. Furthermore, the proposed method contains a novel update module (UM) that evaluates the tracking confidence of the current state to adapt to object changes and attenuate tracking drifts. Extensive experiments demonstrate the superiority of our approach compared to state-of-the-arts (SOTAs) while meeting real-time demands.

Hyperspectral image classification plays an important role in the field of remote sensing. Even though we can easily acquire hyperspectral remote sensing images, obtaining a large number of labeled hyperspectral samples remains challenging, especially in high-altitude or uninhabited areas. In this paper, we propose a hyperspectral classification scheme for scenarios with insufficient labeled samples. This scheme is based on a variant of the ResBlock and a non-learned classifier. First, we introduce a new and simplified backbone network for feature extraction. This network primarily consists of an attention-enhanced ResBlock-in-ResBlock module, which utilizes nested residual modules to enhance nonlinear expression and further optimizes the network using channel attention. Building upon this foundation, we address the challenge of achieving optimal classification with limited labeled training samples, a scenario described by the neural collapse theory. To address this, we introduce the Equiangular Tight Frame (ETF) classifier and the dot-regression loss into hyperspectral classification. We conducted extensive comparative experiments using three hyperspectral image datasets. The experimental results demonstrate that our algorithm achieves superior classification accuracy, especially when the training sample size is small, outperforming other state-of-the-art algorithms. Furthermore, our algorithm maintains a low number of parameters and an overall complexity level.

Hyperspectral Image (HSI) classification is a key technique in remote sensing. Despite the increasing availability of high-quality HSI data, obtaining a large number of labeled samples remains challenging in certain cases. Consequently, HSI classification often faces the issue of insufficiently labeled samples. To address this challenge, sample augmentation techniques can be used to generate additional training samples. However, because of the significant differences between hyperspectral images and ordinary natural images, some augmentation techniques are not suitable for hyperspectral classification scenarios. In this paper, considering phenomena such as spectral aliasing in hyperspectral image classification and imaging processes, we propose a novel online augmentation technique for hyperspectral samples. During training, we apply random gains to the center pixel of labeled samples to increase the number of usable samples. Additionally, since augmented samples may still be insufficient, using overly complex networks can lead to overfitting. Therefore, we introduce a hyperspectral image classification network called Attention-enhancing Residual and Spatial-Channel Attention-based network (ReSC-net). In ReSC-net, we observe that the spatial dimension of hyperspectral blocks is much smaller than the channel dimension, and the limited sample size can lead to overfitting when using complex networks. Thus, we propose a channel attention-enhanced residual module to extract low-level features. Furthermore, ReSC-net introduces new spatial-channel attention to further optimize the extracted deep features for better classification. We conduct experiments on four commonly used HSI datasets. The experimental results demonstrate that our algorithm achieves favorable results on multiple HSI classification evaluation metrics.

Buildings and roads are the two most basic man-made environments that carry and interconnect human society. Building and road information has important application value in the frontier fields of regional coordinated development, disaster prevention, auto-driving, etc. Mapping buildings and roads from very high-resolution (VHR) remote sensing images has become a hot research topic. However, the existing methods often extract buildings and roads with separate models, ignoring their strong spatial correlation. To fully utilize their complementary relation, we propose a method that simultaneously extracts buildings and roads from remote sensing images. The accuracy of both tasks can be improved using our proposed multi-task feature interaction and cross-scale feature interaction modules. To be specific, a multi-task interaction module is proposed to interact information across building extraction and road extraction tasks while preserving the unique information of each task. Furthermore, a cross-scale interaction module is designed to automatically learn the optimal reception field for buildings and roads under varied appearances and structures. Compared with existing methods that train individual models for each task separately, the proposed collaborative extraction method can utilize the complementary advantages between buildings and roads and reduce the inference time by half using a single model. Experiments on a wide range of urban and rural scenarios show that the proposed algorithm can achieve building-road extraction with outstanding performance and efficiency.

Crop mapping in a cloudy area is always a challenge due to the lack of time-series clear optical satellite imagery. Making use of time-series synthetic aperture radar (SAR) imagery that is immune to cloud contamination is essential and promising for seamless and large-area crop mapping. However, existing deep learning (DL)-based crop classification methods give the extracted phenological features equal weights, without considering the different contributions of phenological features of the different crop growth periods. In this article, a phenology-based crop mapping network (PhenoCropNet) is proposed to extract the discriminative features from the two levels, including the key phenological dates in the phenological periods and key phenological periods in the whole growth stages. PhenoCropNet includes a phenological calendar information injection (PAI) module that divides the satellite imagery time series (SITS) into multiple sequences according to the phenological calendar information, and a hierarchical attention network structure that uses the two-level bidirectional gated recurrent unit-based self-attention (BiGRUA) modules to automatically extract the features containing the most important phenological information of key phenological dates and key phenological periods. The proposed PhenoCropNet was verified in Hubei province in China, around 185 933 km2, a typical cloudy area in China, for rapid winter crop mapping based on temporal Sentinel-1 SAR imagery. The mapping result shows that the$F1$-score of PhenoCropNet for winter crop mapping could achieve 0.90, showing great potential in large-scale and seamless crop mapping. The code is available on request:https://github.com/LL0912/PhenoCropNet.

Building extraction aims to segment building pixels from remote sensing images and plays an essential role in many applications, such as city planning and urban dynamic monitoring. Over the past few years, deep learning methods with encoder–decoder architectures have achieved remarkable performance due to their powerful feature representation capability. Nevertheless, due to the varying scales and styles of buildings, conventional deep learning models always suffer from uncertain predictions and cannot accurately distinguish the complete footprints of the building from the complex distribution of ground objects, leading to a large degree of omission and commission. In this paper, we realize the importance of uncertain prediction and propose a novel and straightforward Uncertainty-Aware Network (UANet) to alleviate this problem. Specifically, we first apply a general encoder–decoder network to obtain a building extraction map with relatively high uncertainty. Second, in order to aggregate the useful information in the highest-level features, we design a Prior Information Guide Module to guide the highest-level features in learning the prior information from the conventional extraction map. Third, based on the uncertain extraction map, we introduce an Uncertainty Rank Algorithm to measure the uncertainty level of each pixel belonging to the foreground and the background. We further combine this algorithm with the proposed Uncertainty-Aware Fusion Module to facilitate level-by-level feature refinement and obtain the final refined extraction map with low uncertainty. To verify the performance of our proposed UANet, we conduct extensive experiments on three public building datasets, including the WHU building dataset, the Massachusetts building dataset, and the Inria aerial image dataset. Results demonstrate that the proposed UANet outperforms other state-of-the-art algorithms by a large margin. The source code of the proposed UANet is available at https://github.com/Henryjiepanli/Uncertainty-aware-Network.

The combined sparse and collaborative representation-based algorithm is one of the most effective methods among hyperspectral target detection methods based on representation and dictionary learning. It encourages target atoms to compete with each other and background atoms to collaborate in the representation. However, this method suffers from several drawbacks. In sparse representation, an overcomplete dictionary is necessary, whereas, in collaborative representation, non-negative coefficients are required. Besides, the local dual window approach may result in impure background dictionaries obtained from the outer window. To address these issues, we propose a novel approach for hyperspectral target detection, referred to as the global overcomplete dictionary-based sparse and nonnegative collaborative representation (GODSNCR) detector. First, a hierarchical density clustering algorithm is used to complete the dictionary atom extraction to construct a joint overcomplete dictionary to satisfy the dictionary overcompleteness problem required for sparse representation. Second, a nonnegative constraint on the coefficient matrix and a “sum to one” constraint for the joint representation are incorporated to make it more consistent with the physical meaning. Finally, the limitation of the local dual window approach is overcome by substituting the local background dictionary with a global background dictionary. Through the aforementioned strategies, we can use a joint overcomplete dictionary for achieving the sparse representation of targets and utilize a global background dictionary for the collaborative representation of background, the final detection results are obtained by calculating the residuals. The experimental results clearly demonstrate that the proposed algorithm has significant improvement in detection accuracy and strong robustness compared to other typical representation-based hyperspectral target detection methods. Our model will be available at https://github.com/Chenxing-Li/GODSNCR.

High spatial, high spectral, and high temporal ($\text {H}^{3}$) information of the objects of interest can be provided by hyperspectral video, which makes it possible to track objects in complex scenarios. However, during motion, changes in the target’s appearance, background, and spectral information can degrade the performance of existing hyperspectral trackers due to insufficient training data. Consequently, this results in weak generalization of these trackers. In this article, to solve the above problems, a deep temporal-spectral–spatial anchor-free Siamese tracking network for hyperspectral video object tracking, namely HA-Net, is proposed. In HA-Net, a Siamese spectral enhancement tracker module based on an RGB tracker (pseudo-color tracker) is designed, which uses the powerful feature expression capabilities of the deep network to learn more discriminative deep spectral features for identifying objects in complex scenarios. The pseudo-color tracker is introduced to solve the problem of model performance limitation due to insufficient training data. By introducing the temporal-spectral–spatial online discrimination learning module, the temporal-spectral–spatial information of the target can be dynamically modeled to adapt to new targets and the dynamic changes of targets. Benefiting from the double Siamese network architecture, the model can be effectively trained from scratch with less than 20 000 training samples. Online learning of temporal-spectral–spatial information for the target, particularly in cases of insufficient training data, can alleviate the issue of model degradation. This approach enhances the model’s robustness when tracking the target in complex scenes. In the 2021 IEEE WHISPERS Hyperspectral Object Tracking (HOT) Challenge, HA-Net obtained the best performance, with a distance precision (DP) score of 0.948 and an area under the curve (AUC) score of 0.688. The running speed is also 14 frames/s, which is superior to the existing hyperspectral object trackers for hyperspectral video. The source code is available athttps://github.com/zhenliuzhenqi/HOT.

Subpixel mapping (SPM) aims to reconstruct a subpixel-level class distribution map from the pixel-level abundance maps, which is an under-determined problem that has nonunique solutions. To address this, the spatiotemporal SPM uses the abundance, spatial, and temporal constraints to reduce the uncertainty of the mapping solutions, so the spatiotemporal SPM is essentially a constrained optimization problem. However, it is hard to find the optimal weighting parameters to combine the three joint constraints. In addition, the existing spatiotemporal SPM methods mainly use the temporal information either for the unchanged subpixels detection or for the subpixel classification, which is insufficient in the utilization of the temporal information. In this article, a novel spatiotemporal SPM algorithm based on multiobjective optimization (STSPM_MO) is proposed. STSPM_MO is composed of an unchanged subpixels detection stage and a multiobjective spatiotemporal mapping stage. In the former stage, the historical thematic map is used for identifying the unchanged subpixels. In the latter stage, the historical thematic map is further used for providing the temporal dependence, so that the temporal information can be more fully utilized. Moreover, to solve the constrained optimization problem of the spatiotemporal SPM, the abundance, spatial, and temporal constraints are modeled as three objectives and are dynamically fused through the subfitness-based multiobjective evolution, to generate the optimal subpixel classification map. Both synthetic and real-data experiments have been conducted, and the results show the proposed method, and its two variants are superior, stable, and effective.

Deep learning (DL) has recently shown considerable potential in the field of Hyperspectral unmixing (HU), thanks to its advanced capabilities in feature representation. Most joint spectral-spatial DL-based unmixing methods extract spatial features at a single scale, with basic feature fusion techniques. Given the heterogeneity in ground object sizes, these methods, with their singular receptive fields, may capture only incomplete features or blend features from disparate objects, leading to inadequate spatial comprehension. To tackle this challenge, we propose the multiscale aggregation transformer network (MAT-Net), which utilizes an encoder-decoder architecture designed to harness both spectral and spatial data comprehensively. Within this structure, we have devised a dual-stream, multibranch convolutional neural network (CNN) encoder to extract both spectral and multiscale spatial information. The spatial stream includes three CNN branches with different receptive fields. A block-by-block branching strategy is employed in this stream to maintain the continuity of multiscale spatial information and reduce computational costs. To effectively integrate intricate multiscale features, we introduce a transformer encoder that incorporates multihead self-scale-aggregation attention (MSsaA) blocks. These blocks are designed to adaptively modulate feature weights based on the scale characteristics of the image, enabling a more nuanced feature integration process. Experimental results on synthetic and real datasets demonstrate the effectiveness of MAT-Net.

Scene change detection (SCD) involves recognizing the high-level semantic change types for a bitemporal remote sensing image scene pair. Previous methods have implicitly considered joint bitemporal features as a change representation to perform SCD. However, they have not effectively enhanced the discriminative power of the change representation. Consequently, these methods struggle to recognize scene changes with both intraclass variation and interclass similarity. In this paper, scene change contrastive (SCC) learning based on contrastive learning is proposed to ensure that bitemporal features are discriminative change representations for SCD recognition. Contrastive learning can learn specific discriminative features by gathering predefined specific positives and separating negatives in the projection space. In the SCC learning, the change representations, which are represented by the joint bitemporal features, are mapped to real and pseudo multi-view change projections by the proposed multi-view change (MVC) projector and the pseudo change augmentation (PeCA) strategy. The change projections are then guided to be discriminative, exploiting both local spatial and global information. By doing so, the joint bitemporal scene features become more discriminative change representations, which enable accurate recognition of scene changes. The extensive experimental results obtained on two public datasets consistently demonstrate the effectiveness of the proposed method. The code is available at https://github.com/wdczs/SCC.

Semantic change detection (SCD) for high-resolution remote sensing imagery involves simultaneously locating the changed regions and identifying the semantic change categories. Recently, a series of multitask Siamese networks have been proposed to model the SCD task by merging the binary change detection (BCD) results and the bitemporal land-cover classification (LCC) results. However, due to the large reflectance variability of the land cover in bitemporal high-resolution images, the land-cover feature clusters extracted by these methods are still considerably mixed, leading to the misidentification of change and their semantic changes types. In this article, to handle this problem, the SiamContrast method is proposed to learn temporally invariant discriminative land-cover features for SCD. As part of SiamContrast, a novel SCD contrastive loss (SCD-CL) is proposed to enhance the temporally invariant feature discrimination across bitemporal images. SCD-CL utilizes supervised contrastive learning and consists of two complementary components: mono-temporal contrastive loss (MCL) and cross-temporal contrastive loss (CCL). In particular, MCL contrasts the land-cover features within each temporal image, to enhance the mono-temporal feature discrimination. Meanwhile, CCL with a change-aware hard anchor sampling (CHAS) strategy contrasts the land-cover features across bitemporal images, to align the land cover features of the same category. To validate the effectiveness of SCD-CL, the SiamContrast method incorporates a difference feature pyramid (DFP) decoder, which leverages feature distance to model changes, allowing it to directly benefit from the discriminative features learned by SCD-CL. The comprehensive experimental results consistently confirm the effectiveness of the proposed SiamContrast in improving SCD performance, compared with the existing SCD methods. The code is available athttps://github.com/wdczs/SiamContrast.

Recently, convolutional networks have achieved remarkable development in remote sensing image (RSI) super-resolution (SR) by minimizing the regression objectives, e.g., MSE loss. However, despite achieving impressive performance, these methods often suffer from poor visual quality with oversmooth issues. Generative adversarial networks (GANs) have the potential to infer intricate details, but they are easy to collapse, resulting in undesirable artifacts. To mitigate these issues, in this article, we first introduce diffusion probabilistic model (DPM) for efficient RSI SR, dubbed efficient diffusion model for RSI SR (EDiffSR). EDiffSR is easy to train and maintains the merits of DPM in generating perceptual-pleasant images. Specifically, different from previous works using heavy UNet for noise prediction, we develop an efficient activation network (EANet) to achieve favorable noise prediction performance by simplified channel attention and simple gate operation, which dramatically reduces the computational budget. Moreover, to introduce more valuable prior knowledge into the proposed EDiffSR, a practical conditional prior enhancement module (CPEM) is developed to help extract an enriched condition. Unlike most DPM-based SR models that directly generate conditions by amplifying LR images, the proposed CPEM helps to retain more informative cues for accurate SR. Extensive experiments on four remote sensing datasets demonstrate that EDiffSR can restore visual-pleasant images on simulated and real-world RSIs, both quantitatively and qualitatively. The code of EDiffSR will be available athttps://github.com/XY-boy/EDiffSR.

Road occlusion seriously affects the connectivity of extracted roads, and has a negative effect in practical applications. The dense road occlusion problem is caused by high-rise buildings and street trees, and is a more serious and unique problem than simple occlusion caused by low buildings and scattered trees. The existing methods mainly solve the road occlusion problem by enhancing the encoder ability to capture the long connectivity feature of roads. Unfortunately, the existing methods can only solve small and sparse road occlusion situations, and they cannot deal with the dense road occlusions caused by dense high-rise buildings or trees. In this article, to solve the dense road occlusion problem, the occlusion-aware road extraction network, namely OARENet, is proposed for road extraction from high-resolution remote sensing imagery. In OARENet, an occlusion-aware decoder (OADecoder) is designed by explicit modeling the texture feature for road regions with dense occlusions. The OADecoder is made up of a regular occlusion-aware (ROA) module and a stochastic occlusion-aware (SOA) module. The ROA module is implemented by adopting different dilation rates to fit the texture feature in the semantic feature maps. The SOA module is proposed by designing stochastic convolutions to adaptively fit the spatial details of road regions with dense occlusions. In order to evaluate the dense occlusion problem, a dense occlusion road dataset (JHWV) was built and annotated. The experimental results obtained on the DeepGlobe dataset, the newly built JHWV dataset, and large-scale urban images demonstrate the superiority of OARENet, especially when faced with a dense road occlusion situation. Code has been made available at: https://github.com/WanderRainy/OARENet.

The existing deep learning based methods have shown great potential for the aurora image classification problem. However, there are many differences in the morphology and distribution patterns of aurora images from different observation stations, and the differences between Antarctic and Arctic aurora images are particularly obvious. Currently, there are 76 research stations in 31 countries in Antarctica and more than 100 land-based stations in the Arctic. In the face of the difference in morphology and distribution patterns between Antarctic and Arctic auroras, the current popular methods cannot maintain a consistent classification ability. At the same time, it is important to effectively use both historical and real-time information to enable continual learning of aurora classification models to take full advantage of the high temporal resolution of streaming aurora image data. In this paper, a cross-station continual (CSC) aurora image classification framework is proposed to tackle these problems. To simulate a cross-station aurora image data stream, aurora images from three observation stations located in the Antarctic and the Arctic were selected and split into mini-batches in chronological order to form the cross-station streaming (CSS) aurora image dataset. Based on the vision transformer model, the CSC framework sequentially learns the semantic representation of streaming aurora data by learning dynamic prompts in the prompt bank selected by the average cosine distance. For the cross-station aurora discrepancy phenomenon, a local-global enhancement (LGE) module is designed, by organically combining the local and global semantics of aurora images to reduce the microscopic intra-class similarity and macroscopic inter-class confusion. Extensive experiments conducted on the CSS dataset show that the proposed method can achieve efficient continual learning of streaming aurora data and a competitive classification accuracy under the condition of joint training of data from multiple observation stations.

In recent years, great progress has been made in the field of cross-scene classification. However, most existing cross-scene methods assume that the source domain has massive and carefully annotated data, which is time-consuming and labor-intensive in practice. Datasets in real-life applications usually contain a large number of noisy labels, which will significantly affect cross-scene classification performance. How to perform more discriminative and generalized cross-scene classification in the presence of noisy samples needs to be urgently addressed. Apart from that, existing methods tend to implement global matching between domains, causing problems such as unbalanced adaptation and negative transfer, limiting the cross-scene performance of the model. For more effective and reliable cross-scene classification under noisy environment, robust adaptation with noise (RAN) is proposed in this article. RAN explores which samples are noiseless and transferable to enable positive and robust cross-scene transfer. The curriculum learning strategy is used to filter out noisy samples for better source supervised learning and cross-domain matching. To further improve the stability and effectiveness of cross-scene adaptation, the class weighting factor and the public weighting factor are introduced to consider the class information of the source and target domains. RAN is an efficient plug-and-play adaptation framework, which is easily implemented and can be embedded in existing methods. Experimental results demonstrate that the proposed RAN can achieve remarkable performance on cross-scene classification tasks in noisy environments.

Hyperspectral image super-resolution has attained widespread prominence to enhance the spatial resolution of hyperspectral images. However, convolution-based methods have encountered challenges in harnessing the global spatial-spectral information. The prevailing transformer-based methods have not adequately captured the long-range dependencies in both spectral and spatial dimensions. To alleviate this issue, we propose a novel cross-scope spatial-spectral Transformer (CST) to efficiently investigate long-range spatial and spectral similarities for single hyperspectral image super-resolution. Specifically, we devise cross-attention mechanisms in spatial and spectral dimensions to comprehensively model the long-range spatial-spectral characteristics. By integrating global information into the rectangle-window self-attention, we first design a cross-scope spatial self-attention to facilitate long-range spatial interactions. Then, by leveraging appropriately characteristic spatial-spectral features, we construct a cross-scope spectral self-attention to effectively capture the intrinsic correlations among global spectral bands. Finally, we elaborate a concise feed-forward neural network to enhance the feature representation capacity in the Transformer structure. Extensive experiments over three hyperspectral datasets demonstrate that the proposed CST is superior to other state-of-the-art methods both quantitatively and visually. The code is available at https://github.com/Tomchenshi/CST.git.

Change detection (CD) is a fundamental and important task for monitoring the land surface dynamics in the earth observation field. Existing deep learning-based CD methods typically extract bi-temporal image features using a weight-sharing Siamese encoder network and identify change regions using a decoder network. These CD methods, however, still perform far from satisfactorily as we observe that 1) deep encoder layers focus on irrelevant background regions; and 2) the models' confidence in the change regions is inconsistent at different decoder stages. The first problem is because deep encoder layers cannot effectively learn from imbalanced change categories using the sole output supervision, while the second problem is attributed to the lack of explicit semantic consistency preservation. To address these issues, we design a novel similarity-aware attention flow network (SAAN). SAAN incorporates a similarity-guided attention flow module with deeply supervised similarity optimization to achieve effective change detection. Specifically, we counter the first issue by explicitly guiding deep encoder layers to discover semantic relations from bi-temporal input images using deeply supervised similarity optimization. The extracted features are optimized to be semantically similar in the unchanged regions and dissimilar in the changing regions. The second drawback can be alleviated by the proposed similarity-guided attention flow module, which incorporates similarity-guided attention modules and attention flow mechanisms to guide the model to focus on discriminative channels and regions. We evaluated the effectiveness and generalization ability of the proposed method by conducting experiments on a wide range of CD tasks. The experimental results demonstrate that our method achieves excellent performance on several CD tasks, with discriminative features and semantic consistency preserved.

Transformer-based method has demonstrated promising performance in image super-resolution tasks, due to its long-range and global aggregation capability. However, the existing Transformer brings two critical challenges for applying it in large-area earth observation scenes: (1) redundant token representation due to most irrelevant tokens; (2) single-scale representation which ignores scale correlation modeling of similar ground observation targets. To this end, this paper proposes to adaptively eliminate the interference of irreverent tokens for a more compact self-attention calculation. Specifically, we devise a Residual Token Selective Group (RTSG) to grasp the most crucial token by dynamically selecting the top- k keys in terms of score ranking for each query. For better feature aggregation, a Multi-scale Feed-forward Layer (MFL) is developed to generate an enriched representation of multi-scale feature mixtures during feed-forward process. Moreover, we also proposed a Global Context Attention (GCA) to fully explore the most informative components, thus introducing more inductive bias to the RTSG for an accurate reconstruction. In particular, multiple cascaded RTSGs form our final Top- k Token Selective Transformer (TTST) to achieve progressive representation. Extensive experiments on simulated and real-world remote sensing datasets demonstrate our TTST could perform favorably against state-of-the-art CNN-based and Transformer-based methods, both qualitatively and quantitatively. In brief, TTST outperforms the state-of-the-art approach (HAT-L) in terms of PSNR by 0.14 dB on average, but only accounts for 47.26% and 46.97% of its computational cost and parameters. The code and pre-trained TTST will be available at https://github.com/XY-boy/TTST for validation.

Recent years have witnessed the superiority of deep learning-based algorithms in the field of HSI classification. However, a prerequisite for the favorable performance of these methods is a large number of refined pixel-level annotations. Due to atmospheric changes, sensor differences, and complex land cover distribution, pixel-level labeling of high-dimensional hyperspectral image (HSI) is extremely difficult, time-consuming, and laborious. To overcome the above hurdle, an Image-To-pixEl Representation (ITER) approach is proposed in this paper. To the best of our knowledge, this is the first time that image-level annotation is introduced to predict pixel-level classification maps for HSI. The proposed model is along the lines of subject modeling to boundary refinement, corresponding to pseudo-label generation and pixel-level prediction. Concretely, in the pseudo-label generation part, the spectral/spatial activation, spectral-spatial alignment loss, and geographic element enhancement are sequentially designed to locate discriminate regions of each category, optimize multi-domain class activation map (CAM) collaborative training, and refine labels, respectively. For the pixel-level prediction portion, a high frequency-aware self-attention in a high-enhanced transformer is put forward to achieve detailed feature representation. With the two-stage pipeline, ITER explores weakly supervised HSI classification with image-level tags, bridging the gap between image-level annotation and dense prediction. Extensive experiments in three benchmark datasets with state-of-the-art (SOTA) works show the performance of the proposed approach.

Remote sensing image scene classification methods based on deep learning have been widely studied and discussed. However, most of the network architectures are directly reliant on natural image processing methods and are fixed. A few studies have focused on automatic search mechanisms, but they cannot weigh the interpretation accuracy and the parameter quantity for practical application. As a result, automatic global search methods based on multiobjective evolutionary computation have more advantages. However, in the ranking process, the network individuals with large parameter quantities are easy to eliminate, but a higher accuracy may be obtained after full training. In addition, evolutionary neural architecture search methods often take several days. In this article, in order to solve the above concerns, we propose an efficient multiobjective evolutionary automatic search framework for remote sensing image scene classification deep learning network architectures (E2SCNet). In E2SCNet, eight kinds of lightweight operators are used to build a diversified search space, and the coding connection mode is flexible. In the search process, a large model retention mechanism is implemented through two-step multiobjective modeling and evolutionary search, where one step involves the "parameter quantity and accuracy," and the other step involves the "parameter quantity and accuracy growth quantity." Moreover, a super network is constructed to share the weight in the process of individual network evaluation and promote the search speed. The effectiveness of E2SCNet is proven by comparison with several networks designed by human experts and networks obtained by gradient and evolutionary computing-based search methods.

Convolutional neural networks (CNNs) have been widely applied to hyperspectral image classification (HSIC). However, traditional convolutions can not effectively extract features for objects with irregular distributions. Recent methods attempt to address this issue by performing graph convolutions on spatial topologies, but fixed graph structures and local perceptions limit their performances. To tackle these problems, in this article, different from previous approaches, we perform the superpixel generation on intermediate features during network training to adaptively produce homogeneous regions, obtain graph structures, and further generate spatial descriptors, which are served as graph nodes. Besides spatial objects, we also explore the graph relationships between channels by reasonably aggregating channels to generate spectral descriptors. The adjacent matrices in these graph convolutions are obtained by considering the relationships among all descriptors to realize global perceptions. By combining the extracted spatial and spectral graph features, we finally obtain a spectral-spatial graph reasoning network (SSGRN). The spatial and spectral parts of SSGRN are separately called spatial and spectral graph reasoning subnetworks. Comprehensive experiments on four public datasets demonstrate the competitiveness of the proposed methods compared with other state-of-the-art graph convolution-based approaches.

Recent neural architecture search (NAS)-based approaches have made great progress in the hyperspectral image (HSI) classification tasks. However, the architectures are usually optimized independently of the network weights, increasing searching time, and restricting model performances. To tackle these issues, in this article, different from previous methods that extra define structural parameters, we propose to directly generate structural parameters by utilizing the specifically designed hyper kernels, ingeniously converting the original complex dual optimization problem into easily implemented one-tier optimizations, and greatly shrinking searching costs. Then, we develop a hierarchical multimodule search space whose candidate operations only contain convolutions, and these operations can be integrated into unified kernels. Using the above searching strategy and searching space, we obtain three kinds of networks to separately conduct pixel-level or image-level classifications with 1-D or 3-D convolutions. In addition, by combining the proposed hyper kernel searching scheme with the 3-D convolution decomposition mechanism, we obtain diverse architectures to simulate 3-D convolutions, greatly improving network flexibilities. A series of quantitative and qualitative experiments on six public datasets demonstrate that the proposed methods achieve state-of-the-art results compared with other advanced NAS-based HSI classification approaches.

Recent research has shown the great potential of deep learning algorithms in the hyperspectral image (HSI) classification task. Nevertheless, training these models usually requires a large amount of labeled data. Since the collection of pixel-level annotations for HSI is laborious and time-consuming, developing algorithms that can yield good performance in the small sample size situation is of great significance. In this study, we propose a robust self-ensembling network (RSEN) to address this problem. The proposed RSEN consists of two subnetworks including a base network and an ensemble network. With the constraint of both the supervised loss from the labeled data and the unsupervised loss from the unlabeled data, the base network and the ensemble network can learn from each other, achieving the self-ensembling mechanism. To the best of our knowledge, the proposed method is the first attempt to introduce the self-ensembling technique into the HSI classification task, which provides a different view on how to utilize the unlabeled data in HSI to assist the network training. We further propose a novel consistency filter to increase the robustness of self-ensembling learning. Extensive experiments on three benchmark HSI datasets demonstrate that the proposed algorithm can yield competitive performance compared with the state-of-the-art methods.

Deep learning-based methods have shown promising outcomes in many fields. However, the performance gain is always limited to a large extent in classifying hyperspectral image (HSI). We discover that the reason behind this phenomenon lies in the incomplete classification of HSI, i.e., existing works only focus on a certain stage that contributes to the classification, while ignoring other equally or even more significant phases. To address the above issue, we creatively put forward three elements needed for complete classification: the extensive exploration of available features, adequate reuse of representative features, and differential fusion of multidomain features. To the best of our knowledge, these three elements are being established for the first time, providing a fresh perspective on designing HSI-tailored models. On this basis, an HSI classification full model (HSIC-FM) is proposed to overcome the barrier of incompleteness. Specifically, a recurrent transformer corresponding to Element 1 is presented to comprehensively extract short-term details and long-term semantics for local-to-global geographical representation. Afterward, a feature reuse strategy matching Element 2 is designed to sufficiently recycle valuable information aimed at refined classification using few annotations. Eventually, a discriminant optimization is formulized in accordance with Element 3 to distinctly integrate multidomain features for the purpose of constraining the contribution of different domains. Numerous experiments on four datasets at small-, medium-, and large-scale demonstrate that the proposed method outperforms the state-of-the-art (SOTA) methods, such as convolutional neural network (CNN)-, fully convolutional network (FCN)-, recurrent neural network (RNN)-, graph convolutional network (GCN)-, and transformer-based models (e.g., accuracy improvement of more than 9% with only five training samples per class). The code will be available soon at https://github.com/jqyang22/ HSIC-FM.

Due to the limitation of target size and spatial resolution, targets of interest in hyperspectral images (HSIs) often appear as subpixel targets, which makes hyperspectral target detection still faces an important bottleneck, that is, subpixel target detection. In this article, we propose a new detector by learning single spectral abundance for hyperspectral subpixel target detection (denoted as LSSA). Different from most existing hyperspectral detectors that are designed based on a match of the spectrum assisted by spatial information or focusing on the background, the proposed LSSA addresses the problem of detecting subpixel targets by learning a spectral abundance of the target of interest directly. In LSSA, the abundance of the prior target spectrum is updated and learned, while the prior target spectrum is fixed in a nonnegative matrix factorization (NMF) model. It turns out that such a way is quite effective to learn the abundance of subpixel targets and contributes to detecting subpixel targets in hyperspectral imagery (HSI). Numerous experiments are conducted on one simulated dataset and five real datasets, and the results indicate that the LSSA yields superior performance in hyperspectral subpixel target detection and outperforms its counterparts.

Diffusion models have recently received a surge of interest due to their impressive performance for image restoration, especially in terms of noise robustness. However, existing diffusion-based methods are trained on a large amount of training data and perform very well in-distribution, but can be quite susceptible to distribution shift. This is especially inappropriate for data-starved hyperspectral image (HSI) restoration. To tackle this problem, this work puts forth a self-supervised diffusion model for HSI restoration, namely Denoising Diffusion Spatio-Spectral Model (DDS2M), which works by inferring the parameters of the proposed Variational Spatio-Spectral Module (VS2M) during the reverse diffusion process, solely using the degraded HSI without any extra training data. In VS2M, a variational inference-based loss function is customized to enable the untrained spatial and spectral networks to learn the posterior distribution, which serves as the transitions of the sampling chain to help reverse the diffusion process. Benefiting from its self-supervised nature and the diffusion process, DDS2M enjoys stronger generalization ability to various HSIs compared to existing diffusion-based methods and superior robustness to noise compared to existing HSI restoration methods. Extensive experiments on HSI denoising, noisy HSI completion and super-resolution on a variety of HSIs demonstrate DDS2M’s superiority over the existing task-specific state-of-the-arts. Code is available at: https://github.com/miaoyuchun/DDS2M.

Understanding the temporal dynamics of Earth’s surface is a mission of multi-temporal remote sensing image analysis, significantly promoted by deep vision models with its fuel—labeled multi-temporal images. However, collecting, preprocessing, and annotating multi-temporal remote sensing images at scale is non-trivial since it is expensive and knowledge-intensive. In this paper, we present a scalable multi-temporal remote sensing change data generator via generative modeling, which is cheap and automatic, alleviating these problems. Our main idea is to simulate a stochastic change process over time. We consider the stochastic change process as a probabilistic semantic state transition, namely generative probabilistic change model (GPCM), which decouples the complex simulation problem into two more trackable sub-problems, i.e., change event simulation and semantic change synthesis. To solve these two problems, we present the change generator (Changen), a GAN-based GPCM, enabling controllable object change data generation, including customizable object property, and change event. The extensive experiments suggest that our Changen has superior generation capability, and the change detectors with Changen pre-training exhibit excellent transferability to real-world change datasets.

The success of the Segment Anything Model (SAM) demonstrates the significance of data-centric machine learning. However, due to the difficulties and high costs associated with annotating Remote Sensing (RS) images, a large amount of valuable RS data remains unlabeled, particularly at the pixel level. In this study, we leverage SAM and existing RS object detection datasets to develop an efficient pipeline for generating a large-scale RS segmentation dataset, dubbed SAMRS. SAMRS totally possesses 105,090 images and 1,668,241 instances, surpassing existing high-resolution RS segmentation datasets in size by several orders of magnitude. It provides object category, location, and instance information that can be used for semantic segmentation, instance segmentation, and object detection, either individually or in combination. We also provide a comprehensive analysis of SAMRS from various aspects. Moreover, preliminary experiments highlight the importance of conducting segmentation pre-training with SAMRS to address task discrepancies and alleviate the limitations posed by limited training data during fine-tuning. The code and dataset will be available at https://github.com/ViTAE-Transformer/SAMRS

The synthetic aperture radar (SAR) image is widely used in many remote sensing applications. In order to store and transmit the increasing SAR image data, more efficient compression algorithms are needed. The purpose of this letter is to introduce a new framework for compressing SAR images. First, we propose a novel analysis and synthesis transform based on multi-Resblocks for transforming the original SAR image into a compact latent representation. Then, a Gaussian mixture model (GMM) is used to estimate the latent representation’s distribution. In order to explore the redundancy within the latent representation, the entropy model parameter is estimated by combining the local context, global context, and hyperprior information. In order to evaluate the performance of the proposed algorithm, we conduct experiments on a dataset of SAR images. The results show that the proposed algorithm outperforms JPEG2000 and some state-of-the-art learned image compression schemes in terms of compression performance.

Deep learning for change detection is one of the current hot topics in the field of remote sensing. However, most end-to-end networks are proposed for supervised change detection, and unsupervised change detection models depend on traditional pre-detection methods. Therefore, we proposed a fully convolutional change detection framework with generative adversarial network, to unify unsupervised, weakly supervised, regional supervised, and fully supervised change detection tasks into one end-to-end framework. A basic Unet segmentor is used to obtain change detection map, an image-to-image generator is implemented to model the spectral and spatial variation between multi-temporal images, and a discriminator for changed and unchanged is proposed for modeling the semantic changes in weakly and regional supervised change detection task. The iterative optimization of segmentor and generator can build an end-to-end network for unsupervised change detection, the adversarial process between segmentor and discriminator can provide the solutions for weakly and regional supervised change detection, the segmentor itself can be trained for fully supervised task. The experiments indicate the effectiveness of the propsed framework in unsupervised, weakly supervised and regional supervised change detection. This article provides new theorical definitions for unsupervised, weakly supervised and regional supervised change detection tasks with the proposed framework, and shows great potentials in exploring end-to-end network for remote sensing change detection (https://github.com/Cwuwhu/FCD-GAN-pytorch).

This article presents Holistically-Attracted Wireframe Parsing (HAWP), a method for geometric analysis of 2D images containing wireframes formed by line segments and junctions. HAWP utilizes a parsimonious Holistic Attraction (HAT) field representation that encodes line segments using a closed-form 4D geometric vector field. The proposed HAWP consists of three sequential components empowered by end-to-end and HAT-driven designs: 1) generating a dense set of line segments from HAT fields and endpoint proposals from heatmaps, 2) binding the dense line segments to sparse endpoint proposals to produce initial wireframes, and 3) filtering false positive proposals through a novel endpoint-decoupled line-of-interest aligning (EPD LOIAlign) module that captures the co-occurrence between endpoint proposals and HAT fields for better verification. Thanks to our novel designs, HAWPv2 shows strong performance in fully supervised learning, while HAWPv3 excels in self-supervised learning, achieving superior repeatability scores and efficient training (24 GPU hours on a single GPU). Furthermore, HAWPv3 exhibits a promising potential for wireframe parsing in out-of-distribution images without providing ground truth labels of wireframes.

Geospatial object segmentation, a fundamental Earth vision task, always suffers from scale variation, the larger intra-class variance of background, and foreground-background imbalance in high spatial resolution (HSR) remote sensing imagery. Generic semantic segmentation methods mainly focus on the scale variation in natural scenarios. However, the other two problems are insufficiently considered in large area Earth observation scenarios. In this paper, we propose a foreground-aware relation network (FarSeg++) from the perspectives of relation-based, optimization-based, and objectness-based foreground modeling, alleviating the above two problems. From the perspective of the relations, the foreground-scene relation module improves the discrimination of the foreground features via the foreground-correlated contexts associated with the object-scene relation. From the perspective of optimization, foreground-aware optimization is proposed to focus on foreground examples and hard examples of the background during training to achieve a balanced optimization. Besides, from the perspective of objectness, a foreground-aware decoder is proposed to improve the objectness representation, alleviating the objectness prediction problem that is the main bottleneck revealed by an empirical upper bound analysis. We also introduce a new large-scale high-resolution urban vehicle segmentation dataset to verify the effectiveness of the proposed method and push the development of objectness prediction further forward. The experimental results suggest that FarSeg++ is superior to the state-of-the-art generic semantic segmentation methods and can achieve a better trade-off between speed and accuracy.

Planning a practical three-dimensional (3-D) flight path for unmanned aerial vehicles (UAVs) is a key challenge for the follow-up management and decision making in disaster emergency response. The ideal flight path is expected to balance the total flight path length and the terrain threat, to shorten the flight time and reduce the possibility of collision. However, in the traditional methods, the tradeoff between these concerns is difficult to achieve, and practical constraints are lacking in the optimized objective functions, which leads to inaccurate modeling. In addition, the traditional methods based on gradient optimization lack an accurate optimization capability in the complex multimodal objective space, resulting in a nonoptimal path. Thus, in this article, an accurate UAV 3-D path planning approach in accordance with an enhanced multiobjective swarm intelligence algorithm is proposed (APPMS). In the APPMS method, the path planning mission is converted into a multiobjective optimization task with multiple constraints, and the objectives based on the total flight path length and degree of terrain threat are simultaneously optimized. In addition, to obtain the optimal UAV 3-D flight path, an accurate swarm intelligence search approach based on improved ant colony optimization is introduced, which can improve the global and local search capabilities by using the preferred search direction and random neighborhood search mechanism. The effectiveness of the proposed APPMS method was demonstrated in three groups of simulated experiments with different degrees of terrain threat, and a real-data experiment with 3-D terrain data from an actual emergency situation.

Deep learning-based hyperspectral image super-resolution (SR) methods have achieved remarkable success, which can improve the spatial resolution of hyperspectral images with abundant spectral information. However, most of them utilize 2D or 3D convolutions to extract local features while ignoring the rich global spatial-spectral information. In this paper, we propose a novel method called the Multi-Scale Deformable Transformer (MSDformer) for single hyperspectral image super-resolution (SR). The proposed method incorporates the strengths of the convolutional neural network for local spatial-spectral information and the Transformer structure for global spatial-spectral information. Specifically, a multi-scale spectral attention module based on dilated convolution is designed to extract local multi-scale spatial-spectral information, which leverages shared module parameters to exploit the intrinsic spatial redundancy and spectral attention mechanism to accentuate the subtle differences between different spectral groups. Then a deformable convolution-based Transformer module is proposed to further extract the global spatial-spectral information from the local multi-scale features of the previous stage, which can explore the diverse long-range dependencies among all spectral bands. Extensive experiments on three hyperspectral datasets demonstrate that the proposed method achieves excellent SR performance and outperforms the state-of-the-art methods in terms of quantitative quality and visual results. The code is available at https://github.com/Tomchenshi/MSDformer.git.

Deep neural networks have shown huge potential in hyperspectral unmixing (HU). However, the large function space increases the difficulty of obtaining the optimal solution with limited unmixing data. The autoencoder-based blind unmixing methods are sensitive to the hyperparameters, and the optimal solution can be difficult to obtain. Algorithm unrolling, which integrates deep learning and iterative algorithms, can shrink the search space and improve the efficiency of obtaining optimal results. Based on this, a model-driven deep neural network named the group sparsity regularized unmixing unrolling (GSUU) network, which unrolls a regularized matrix factorization objective function for blind HU, is proposed in this paper. Based on the nonnegative matrix factorization (NMF) optimization rules, the GSUU network contains two sub-networks—the A-Block and the S-Block—for alternately and iteratively estimating the optimal endmember spectra and abundance maps. The GSUU method incorporates the spatial group sparsity prior of the abundances, i.e., the fact that spatially adjacent mixed pixels share similar sparse abundances, into a deep unrolling network. The experimental results obtained with both synthetic and real hyperspectral data illustrate that the proposed algorithm can obtain a superior accuracy, compared to the other state-of-the-art unmixing algorithms.

Mixed pixels that contain more than one material type are common in mid/low spatial resolution remote sensing imagery. Hyperspectral unmixing is aimed at decomposing the mixed pixels into endmembers and abundances. However, there are few datasets that are suitable for quantitatively evaluating unmixing accuracies, and the ground-truth abundances of the existing datasets are often generated in an approximate way. To address the lack of real unmixing datasets for quantitative evaluation, we built the realistic mixing miniature scenes (RMMS) dataset, which can be used to quantitatively evaluate the unmixing accuracy of different algorithms. The RMMS dataset consists of a simple mixture scene with homogeneous flat materials and a complex mixture scene with 3-D structural features. The features of the RMMS dataset also take point, line, and polygon characteristics into consideration, and the spectral similarity of the materials increases the challenge of the spectral unmixing. In the RMMS dataset, due to the multiscale observation characteristics of the spatiotemporal scanning modality, it can avoid the registration error between RGB and hyperspectral data, and it can ensure that the endmembers are pure pixels. Most of the autonomous hyperspectral unmixing algorithms focus on solving some of the unmixing problems and have difficulty achieving fully autonomous hyperspectral unmixing (FAHU). In this article, to overcome this shortcoming, a fully autonomous hyperspectral unmixing method called FAHU is proposed to take advantage of the spatial information. Some of the state-of-the-art autonomous hyperspectral unmixing algorithms are used to evaluate the performance with the RMMS dataset, and the experimental results show the advantages and disadvantages of the different autonomous unmixing algorithms.

Very high-resolution (VHR) earth observation systems provide an ideal data source for man-made structure detection such as building footprint extraction. Manually delineating building footprints from the remotely sensed VHR images, however, is laborious and time-intensive; thus, automation is needed in the building extraction process to increase productivity. Recently, many researchers have focused on developing building extraction algorithms based on the encoder-decoder architecture of convolutional neural networks. However, we observe that this widely adopted architecture cannot well preserve the precise boundaries and integrity of the extracted buildings. Moreover, features obtained by shallow convolutional layers contain irrelevant background noises that degrade building feature representations. This paper addresses these problems by presenting a feature decoupling network (FD-Net) that exploits two essential building information from the input image, including semantic information that concerns building integrity and edge information that improves building boundaries. The proposed FD-Net improves the existing encoder-decoder framework by decoupling image features into the edge subspace and the semantic subspace; the decoupled features are then integrated by a supervision-guided fusion process considering the heterogeneity between edge and semantic features. Furthermore, a lightweight and effective global context attention module is introduced to capture contextual building information and thus enhance feature representations. Comprehensive experimental results on three real-world datasets confirm the effectiveness of FD-Net in large-scale building mapping. We applied the proposed method to various encoder-decoder variants to verify the generalizability of the proposed framework. Experimental results show remarkable accuracy improvements with less computational cost.

Hyperspectral image (HSI) classification with limited training samples has been well studied in recent years. Among them, the few-shot learning (FSL) technique demonstrates excellent processing capability under limited labeled samples. Nevertheless, the current FSL-based works provide scarce attention to effective class prototypes and metric types, resulting in high generalization error and poor interpretation during the cross-domain testing phase. A dual-adjustment mode-based cross-domain meta-learning (DMCM) method for few-shot HSI classification is proposed to tackle this issue. Specifically, a three-dimensional ghost attention network (TGAN) with strong learning capability without massive parameters is first constructed. Meanwhile, a dual-adjustment mode comprising intra-correction (IC) and inter-alignment (IA) learning strategies is then adopted to solve domain shift issue via episode-level meta tasks, where IC and IA focus on effective class prototypes and data distribution differences between domains, respectively. Afterward, considering that the traditional Euclidean distance metric is insensitive to the distribution of within-class samples, the class-covariance metric is employed to account for the distribution in feature space of each class to optimize decision boundary and alleviate the misclassification problem. Extensive experiments on three publicly available target hyperspectral datasets demonstrate the effectiveness of the proposed method in comparison with other state-of-the-art methods. The codes will be available on the website: https://github.com/HlEvag/DMCM.git.

Soil moisture is one of the important parameters in Earth system models. In recent years, the retrieval based on machine learning and data fusion of multi-source satellite observation data has become one of the effective methods to obtain soil moisture information at a large scale. However, most retrieval studies need to download remote sensing original data first, then preprocess, train the retrieval models, and finally generate products in the offline environment. In order to meet the requirements of long temporal series of large-scale area retrieval, and with the widespread use of machine learning in retrieval studies, the amount of remote sensing data and necessary computing resources are gradually increasing. Moreover, studies usually use a single machine learning retrieval model for the entire study area, which lacks the consideration of geographical differences and spatial heterogeneity of soil moisture. Therefore, we established a geo-intelligent soil moisture retrieval framework completely based on the cloud environment. In this study, a variety of machine learning algorithms were used to fuse multi-source observation data mainly including MODIS data and other auxiliary data, and the Continental United States (CONUS) was taken as the experimental area to generate soil moisture data with a resolution of 500m. In addition, this study combines geographical correlation with machine learning models to cope with the spatial heterogeneity of surface soil moisture. Overall, on the basis of site-based validation, the retrieval model trained under the framework performed well, with estimation accuracy of 0.716 and 0.0383 m3·m-3in terms of coefficient of determination (R2) and unbiased root mean square error (ubRMSE). The establishment of the cloud retrieval framework provides convenience for the whole retrieval process and also provides a new idea for other retrieval studies of geoscience parameters.

Shadow correction is an important task in the analysis of high-resolution remote sensing images, as the existence of shadows reduces radiometric information and causes changes in the energy distribution. This is especially the case when the spatial resolution is very high, as the shadows disturb the subsequent processing and applications, such as image mosaicking, classification, segmentation, etc. Traditional shadow correction methods are limited by the shadow detection accuracy and the available non-shaded samples in the imagery. In this paper, we propose an evolutionary shadow correction network (ESCNet) and describe how we built a benchmark unmanned aerial vehicle (UAV) image dataset to achieve shadow correction directly, without shadow detection. The proposed ESCNet is made up of two sub-networks with an evolutionary relationship: a shadow removal network (SRNet) followed by a radiation adjustment network (RANet). The shadows are first removed by SRNet trained on the UAV image dataset to achieve the primary shadow-corrected image, and the global radiation is then adjusted to a sunlit-like status by RANet. Shadow detection is not required in the proposed method, which effectively overcomes the error accumulation and shadow edge artifact problem of the traditional methods. Experiments were carried out and the results were compared with those of both traditional and deep learning-based shadow correction methods, for which both qualitative and quantitative evaluations were performed. The results suggest that the proposed method shows obvious advantages in information recovery for shadow regions, and the global brightness of the corrected imagery is consistent with that of sunlit conditions.

Nowadays, non-corner reflectors (CRs) calibration techniques are attractive to relieve the workload of polarimetric calibration. Yet, it is challenging to derive the complex co-pol channel imbalance (CCI) precisely and conveniently when CRs are unavailable. Our previous research provided the general unitary zero helix (gUZH) and phase-optimized UZH (pUZH) methods to estimate CCI by the bare soil pixels. However, the amplitude is still overestimated compared with CRs. Thus, this paper proposes to optimize the goal function by L2 normalization and derive the amplitude-optimized UZH (aUZH) to estimate the CCI amplitude robustly. The new aUZH performs well in resisting errors from improper reference picking. We also develop a signal-to-noise ratio (SNR) filter that selects the soil pixels with high SNR into aUZH to reduce the influence of the noise floor. Furthermore, we combine aUZH and the SNR filter to develop a method, i.e., HybridC, to process the massive data for a more precise solution. This paper validates the new algorithm through the external calibration of the Gaofen-3 satellite from 2017 to 2020. The result shows that CR error in HH/VV is better than 0.26 dB after aUZH calibration. Furthermore, we process the proposed HybridC as a tool to monitor the sensor quality of the Gaofen-3 in over 40,000 images. We find that the imbalance phase exceeds the designed specification at some beams and that our algorithm can calibrate the bias precisely.

Hyperspectral object tracking is aimed at tracking targets by using both the spatial information and abundant spectral information, overcoming the drawbacks of traditional RGB tracking in complex scenarios, such as the low resolution or background clutter. However, the current hyperspectral object tracking methods usually have a high computational complexity, due to the huge data volume, making them difficult to apply to real-time applications on edge devices (e.g., robots, unmanned aerial vehicles, and satellites) with limited computational resources. In this paper, a lightweight dual Siamese network for onboard hyperspectral object tracking—termed SiamOHOT—is proposed for real-time and onboard tracking. Specifically, a joint spatial-spectral knowledge distillation method is proposed to teach a lightweight dual Siamese tracker to learn from a deep tracker— SiamHYPER—so that the number of parameters can be compressed to improve the computational efficiency. In addition, a deep learning inference optimizer is introduced to fuse the layers with similar functions and quantify the parameters of the network, to further promote the processing speed when deployed on an embedded platform. The proposed lightweight model was verified using the 2021 WHISPERS Hyperspectral Object Tracking Challenge dataset, and achieved a superior efficiency and accuracy. In addition, a prototype system was built integrating a snapshot hyperspectral imager, the SiamOHOT tracking algorithm, and an artificial intelligence edge device (NVIDIA Jetson Xavier NX), to realize real-time imaging and tracking. The inference speed of the optimized SiamOHOT network is nearly doubled when compared to the teacher model on the prototype system.

Hyperspectral remote sensing band selection picks out characteristic feature combination to weaken the strong correlation caused by spectral continuity. However, it is difficult for traditional methods with fixed strategies to search the entire space and make adjustments for the optimization process. Thus, the solutions obtained can be mostly local optima. In this paper, a novel adaptive multi-strategy particle swarm optimization for hyperspectral image remote sensing band selection (AMSPSO_BS) is introduced to obtain a subset solution suitable for classification. The problem is modeled as an effective fitness function, and the quotient of the linear discriminant value and the mean mutual information (LD/MMI) is used to remove the redundancy between bands. The randomly generated solutions are then encoded to form a population, which rely on various particle update strategies (PUS) with different reference positions for updating. During the particle motion, the effect of each strategy on population evolution is considered comprehensively and reflected in the change of selection probability. And the motion parameters are dynamically adjusted to balance the global and local capabilities. Four hyperspectral remote sensing image datasets were utilized to conduct band selection experiments, to confirm the effectiveness of AMSPSO_BS.

Large-scale vision foundation models have made significant progress in visual tasks on natural images, with vision transformers (ViTs) being the primary choice due to their good scalability and representation ability. However, large-scale models in remote sensing (RS) have not yet been sufficiently explored. In this article, we resort to plain ViTs with about 100 million parameters and make the first attempt to propose large vision models tailored to RS tasks and investigate how such large models perform. To handle the large sizes and objects of arbitrary orientations in RS images, we propose a new rotated varied-size window attention to replace the original full attention in transformers, which can significantly reduce the computational cost and memory footprint while learning better object representation by extracting rich context from the generated diverse windows. Experiments on detection tasks show the superiority of our model over all state-of-the-art models, achieving 81.24% mean average precision (mAP) on the DOTA-V1.0 dataset. The results of our models on downstream classification and segmentation tasks also show competitive performance compared to existing advanced methods. Further experiments show the advantages of our models in terms of computational complexity and data efficiency in transferring. The code and models will be released athttps://github.com/ViTAE-Transformer/Remote-Sensing-RVSA.

Change detection (CD) is a challenging task on high-resolution bitemporal remote sensing images. Many recent studies of CD have focused on designing fully convolutional Siamese network architectures. However, most of these methods initialize their encoders by random values or an ImageNet pretrained model, without any prior for the CD task, thus limiting the performance of the CD model. In this article, the novel supervised contrastive pretraining and fine-tuning CD (SCPFCD) framework, which is made up of two cascaded stages, is presented to train a CD network based on a pretrained encoder. In the first supervised contrastive pretraining stage, the encoder of the Siamese network is asked to solve a joint pretext task introduced by the proposed CDContrast pretraining method on labeled CD data. The proposed CDContrast pretraining method includes land contrastive learning (LCL), which is based on supervised contrastive learning, and proxy CD learning. The LCL focuses on learning the spatial relationships among the land cover from bitemporal images by solving a land contrast task, while the proxy CD learning performs a proxy CD task on the top of the upsampling projector to avoid local optima for the LCL and learn features for the CD. Then, in the second fine-tuning stage, the whole Siamese network initialized with the pretrained encoder is fine-tuned to perform the CD task in an end-to-end manner. The proposed SCPFCD framework was verified with three CD datasets of high-resolution remote sensing images. The extensive experimental results consistently show that the proposed framework can effectively improve the ability to extract change information for Siamese networks.

Recent efforts have witnessed remarkable progress in Satellite Video Super-Resolution (SVSR). However, most SVSR methods usually assume the degradation is fixed and known,e.g., bicubicdownsampling, which makes them vulnerable in real-world scenes with multiple and unknown degradations. To alleviate this issue, blind SR has thus become a research hotspot. Nevertheless, existing approaches are mainly engaged in blur kernel estimation while losing sight of another critical aspect for VSR tasks: temporal compensation, especially compensating for blurry and smooth pixels with vital sharpness from severely degraded satellite videos. Therefore, this paper proposes a practical Blind SVSR algorithm (BSVSR) to explore more sharp cues by considering the pixel-wise blur levels in a coarse-to-fine manner. Specifically, we employed multi-scale deformable convolution to coarsely aggregate the temporal redundancy into adjacent frames by window-slid progressive fusion. Then the adjacent features are finely merged into mid-feature using deformable attention, which measures the blur levels of pixels and assigns more weights to the informative pixels, thus inspiring the representation of sharpness. Moreover, we devise a pyramid spatial transformation module to adjust the solution space of sharp mid-feature, resulting in flexible feature adaptation in multi-level domains. Quantitative and qualitative evaluations on both simulated and real-world satellite videos demonstrate that our BSVSR performs favorably against state-of-the-art non-blind and blind SR models. Code will be available at https://github.com/XY-boy/Blind-Satellite-VSR.

Cross-scene classification has recently gained increasing interest, which improves the classification performance on label-scarce domains by transferring knowledge learned from label-rich domains. Domain adaptation (DA) attempts to solve the domain gap problem and it is widely used in the cross-scene applications. The priori knowledge that the label space between the source and target domains is identical is a key requirement for existing cross-scene approaches to work successfully. However, label sets of different domains can always be different in real applications, that is to say, there will be common as well as private categories for different domains. Universal DA (UniDA) has been proposed to deal with the above difficulty by relaxing all constraints on the label sets. In order to complete more general remote sensing cross-scene classification tasks regardless of label sets, we propose a UniDA cross-scene classification approach, adversarial divergence training (ADT), to simultaneously classify the target common categories and detect the target private categories based on the divergence of different classifiers. ADT attempts to train the classifier and feature extractor against each other (in adversarial) in order to extract more domain-invariant and discriminative features. At the same time, divergence optimization of different classifiers is used to distinguish the target private class. The former makes it capable of cross-scene tasks, while the latter weakens the effect of the label set on the performance of the algorithm. Experiments show that ADT outperforms baselines in the UniDA setting and even in other settings.

Characterized by tremendous spectral information, hyperspectral image is able to detect subtle changes and discriminate various change classes for change detection. The recent research works dominated by hyperspectral binary change detection, however, cannot provide fine change classes information. And most methods incorporating spectral unmixing for hyperspectral multiclass change detection (HMCD), yet suffer from the neglection of temporal correlation and error accumulation. In this study, we proposed an unsupervised Binary Change Guided hyperspectral multiclass change detection Network (BCG-Net) for HMCD, which aims at boosting the multiclass change detection result and unmixing result with the mature binary change detection approaches. In BCG-Net, a novel partial-siamese united-unmixing module is designed for multi-temporal spectral unmixing, and a groundbreaking temporal correlation constraint directed by the pseudo-labels of binary change detection result is developed to guide the unmixing process from the perspective of change detection, encouraging the abundance of the unchanged pixels more coherent and that of the changed pixels more accurate. Moreover, an innovative binary change detection rule is put forward to deal with the problem that traditional rule is susceptible to numerical values. The iterative optimization of the spectral unmixing process and the change detection process is proposed to eliminate the accumulated errors and bias from unmixing result to change detection result. The experimental results demonstrate that our proposed BCG-Net could achieve comparative or even outstanding performance of multiclass change detection among the state-of-the-art approaches and gain better spectral unmixing results at the same time.

Hyperspectral image (HSI) classification is challenging due to spatial variability caused by complex imaging conditions. Prior methods suffer from limited representation ability, as they train specially designed networks from scratch on limited annotated data. We propose a tri-spectral image generation pipeline that transforms HSI into high-quality tri-spectral images, enabling the use of off-the-shelf ImageNet pretrained backbone networks for feature extraction. Motivated by the observation that there are many homogeneous areas with distinguished semantic and geometric properties in HSIs, which can be used to extract useful contexts, we propose an end-to-end segmentation network named DCN-T. It adopts transformers to effectively encode regional adaptation and global aggregation spatial contexts within and between the homogeneous areas discovered by similarity-based clustering. To fully exploit the rich spectrums of the HSI, we adopt an ensemble approach where all segmentation results of the tri-spectral images are integrated into the final prediction through a voting scheme. Extensive experiments on three public benchmarks show that our proposed method outperforms state-of-the-art methods for HSI classification. The code will be released at https://github.com/DotWang/DCN-T.

In the past few years, deep learning-based methods have shown commendable performance for hyperspectral image (HSI) classification. Many works focus on designing independent spectral and spatial branches and then fusing the output features from two branches for category prediction. In this way, the correlation that exists between spectral and spatial information is not completely explored, and spectral information extracted from one branch is always not sufficient. Some studies also try to directly extract spectral-spatial features using 3D convolutions but are accompanied by the severe over-smoothing phenomenon and poor representation ability of spectral signatures. Unlike the above-mentioned approaches, in this paper, we propose a novel online spectral information compensation network (OSICN) for HSI classification, which consists of a candidate spectral vector mechanism, progressive filling process, and multi-branch network. To the best of our knowledge, this paper is the first to online supplement spectral information into the network when spatial features are extracted. The proposed OSICN makes the spectral information participate in network learning in advance to guide spatial information extraction, which truly processes spectral and spatial features in HSI as a whole. Accordingly, OSICN is more reasonable and more effective for complex HSI data. Experimental results on three benchmark datasets demonstrate that the proposed approach has more outstanding classification performance compared with the state-of-the-art methods, even with a limited number of training samples.

Deep neural networks (DNNs) have greatly contributed to the performance gains in semantic segmentation. Nevertheless, training DNNs generally requires large amounts of pixel-level labeled data, which is expensive and time-consuming to collect in practice. To mitigate the annotation burden, this paper proposes a self-ensembling generative adversarial network (SE-GAN) exploiting cross-domain data for semantic segmentation. In SE-GAN, a teacher network and a student network constitute a self-ensembling model for generating semantic segmentation maps, which together with a discriminator, forms a GAN. Despite its simplicity, we find SE-GAN can significantly boost the performance of adversarial training and enhance the stability of the model, the latter of which is a common barrier shared by most adversarial training-based methods. We theoretically analyze SE-GAN and provide an$\mathcal {O}(1/\sqrt{N})$generalization bound ($N$is the training sample size), which suggests controlling the discriminator's hypothesis complexity to enhance the generalizability. Accordingly, we choose a simple network as the discriminator. Extensive and systematic experiments in two standard settings demonstrate that the proposed method significantly outperforms current state-of-the-art approaches.

In hyperspectral target detection, the conventional metric learning-based algorithms provide unique advantages in detecting targets as they do not require specific assumptions and adapt to the condition of limited training samples. Nevertheless, they usually learn a linear transformation for metric space, which is unable to capture nonlinear mapping where the hyperspectral imageries possess, especially occurs in the spectra variability and nonlinear mixing problems. To alleviate this limitation, this study investigates a new spatial-spectral adaptive sample generation and deep metric learning-based method for hyperspectral target detection (denoted as DMLTD). The proposed DMLTD employs a spatial-spectral adaptive sample generation strategy and subpixel synthetic method for background sample generation and target sample augmentation, respectively. With sufficient samples, the proposed DMLTD trains a deep discriminative metric learning network to learn hierarchical nonlinear mappings, so that to address the spectra variability and nonlinear mixing problems, thus exploiting discriminative information between targets and backgrounds for detection. Experiments and analyses conducted on three real-world hyperspectral datasets indicate that our DMLTD yields competitive performance in hyperspectral image target detection.

Change detection (CD), as one of the central problems in Earth observation, has attracted a lot of research interest over recent decades. Due to the rapid development of satellite sensors in recent years, we have witnessed an enrichment of the CD source data with the availability of very-high-resolution (VHR) multispectral imagery, which provides abundant change clues. However, precisely locating real changed areas still remains a challenge. In this article, we propose an end-to-end superpixel-enhanced CD network (ESCNet) for VHR images, which combines differentiable superpixel segmentation and a deep convolutional neural network (DCNN). Two weight-sharing superpixel sampling networks (SSNs) are tailored for the feature extraction and superpixel segmentation of bitemporal image pairs. A UNet-based Siamese neural network is then employed to mine the different information. The superpixels are then leveraged to reduce the latent noise in the pixel-level feature maps while preserving the edges, where a novel superpixelation module is used to serve this purpose. Furthermore, to compensate for the dependence on the number of superpixels, we propose an innovative adaptive superpixel merging (ASM) module, which has a concise form and is fully differentiable. A pixel-level refinement module making use of the multilevel decoded features is also appended to the end of the framework. Experiments on two public datasets confirmed the superiority of ESCNet compared to the traditional and state-of-the-art (SOTA) deep learning-based CD (DLCD) methods.

This paper studies the problem of multi-person pose estimation in a bottom-up fashion. With a new and strong observation that the localization issue of the center-offset formulation can be remedied in a local-window search scheme in an ideal situation, we propose a multi-person pose estimation approach, dubbed as LOGO-CAP, by learning the LOcal-GlObal Contextual Adaptation for human Pose. Specifically, our approach learns the keypoint attraction maps (KAMs) from the local keypoints expansion maps (KEMs) in small local windows in the first step, which are subsequently treated as dynamic convolutional kernels on the keypoints-focused global heatmaps for contextual adaptation, achieving accurate multi-person pose estimation. Our method is end-to-end trainable with near real-time inference speed in a single forward pass, obtaining state-of-the-art performance on the COCO keypoint benchmark for bottom-up human pose estimation. With the COCO trained model, our method also outperforms prior arts by a large margin on the challenging OCHuman dataset.

Sparse unmixing aims at finding the optimal subset of endmembers in a spectral library to approximate the observed data, and has received increasing attention as it can circumvent the estimation of the endmember. In this paper, a graph Laplacian regularized spectral-spatial-sparse unmixing algorithm is proposed, namely, gLapS3U, incorporating the graph Laplacian regularization to consider the similarity between pixels of the whole image, and enforcing the spectral-spatial-sparse constraints to enhance the local spatial information as well as the sparsity of the abundance solution jointly. Experimental results on simulated and real data show the superiority of the proposed algorithm compared with state-of-the-art existing methods.

In recent years, fully convolutional neural networks (FCNs) have been widely used in the field of remote sensing image semantic segmentation. However, these networks have huge amount of parameters and cost much computational efficiency. In this paper, an ultra-lightweight network (ULN) is proposed to overcome this problem. The proposed ULN model uses the encoder-decoder architecture to acquire the pixelwise result. In ULN, the efficient spatial pyramid network (ESPNet) is used to extract deep semantic features with fewer parameters. Considering the dilated convolutions will lose some semantic information in the encoding process, the feature enhancement block (FEB) is proposed. The recurrent criss-cross attention module is added at the end of skip connection to acquire the global contextual information. The proposed ULN is tested on the ISPRS Vaihingen dataset, the results show that our network achieves competitive results with fewer parameters(1.5M).

Due to the limitations of the imaging sensor, the spatial resolution of satellite imagery is often insufficient, namely, low resolution (LR). Therefore, super-resolution (SR) is proposed, which strives to improve image resolution, perfectly to compensate for the shortcomings of satellite sensor imaging. In this study, we develop a unique dilated convolution network with gradient prior (DCNG) for remote sensing SR, aiming to extract powerful low-level features with gradient prior and efficitive network and then reconstruct the high-level feature details. The DCNG is built of two components: the Multi-Scale Feature Extraction Network and the Feature Reconstruction Network. In the Multi-Scale Feature Extraction Network, the Double-Path Dilated Residual Block (DPDRB) is designed with the dilation convolution operation to obtain the multi-scale features and increase the receptive field, the Global Self-attention Module (GSA) to catch the long-range dependency among picture patches, and a Gradient Propagation Network (GPN) is proposed to extract high-level gradient information. In the Feature Reconstruction Network, the Pixel Shuffle is introduced to reconstruct the feature by combining characteristics of different frequency bands. Experiments using Massachusetts_Roads and 3K VEHICLE_SR data sets indicate that our DCNG surpasses state-of-the-art algorithms in terms of quantitative and qualitative evaluations.

As an important semantic understanding method of remote sensing images, scene classification has received much attention in recent years. Convolutional neural network (CNN) is the representative deep learning method for scene classification which has powerful ability in feature extraction. However, the multilevel features in CNN are acquired by hierarchical convolutional layers which have difficulty in considering the interaction of different objects in the scene. Vision transformer (ViT) model provides a new way to understand the image by directly modeling the contextual information of local patches. This paper makes a review of recent progress of ViT models in the field of computer vision and remote sensing. The major contributions are as follows: 1) A brief review of the traditional scene classification methods is made; 2) ViT based models for scene classification are introduced and compared with CNN models; 3) Experiments of recent ViT models are performed and analyzed on UCM and NWPU datasets.

Deep learning based remote sensing scene classification methods have become a research hotspot, but they can not fully mine the image information due to the architecture comes directly from natural image. The automatic search method-based network architecture has then attracted a lot of attention benefits by its ability to independently learn the network structure suitable for remote sensing data. However, in the process of search and sorting, slightly larger models with better performance after full training are often eliminated due to insufficient training. Moreover, the methods often spend a lot of time searching. In this paper, a micro network architecture evolutionary search method is proposed (MAE-Net), the contributions are reflected in the slightly larger model retention mechanism by two-layer multi-objective functions and the super network mechanism used to reduce search time through weight sharing. The effectiveness is proved by comparison with human expert and search based networks on NWPU45 dataset.

Sea ice concentration (SIC) is an indispensable parameter for the study of polar sea ice. The existing methods can obtain accurate SICs for most situations, but they usually perform poorly in low SIC regions because of the spatial differences in the neighboring pixels induced by the discontinuity of the sea ice cover. In this letter, to cope with the difficulty of this problem, an improved SIC estimation method is proposed to retrieve SIC, focusing on low SIC regions. The proposed method introduces the spatial relationships into SIC estimation by employing a total variation (TV) regularizer. Moreover, nonnegative constrained least squares (NCLS) is used to derive the optimal solutions from the SIC estimation equation. Verification was conducted in low SIC regions (0%–50%) of the Antarctic utilizing ship-based in situ data and the Moderate Resolution Imaging Spectroradiometer (MODIS), and the results were compared with those of some of the mature methods. The results indicated that the proposed method can obtain a superior accuracy with a smaller root-mean-square error (RMSE) (6.0%–14.61%) than the other algorithms in low SIC regions. Furthermore, the proposed method can accurately estimate the SIC of both first-year ice and multiyear ice. The findings of this study confirm the need to consider the spatial relationships in the processing of SIC estimation.

In he past decade, object detection has achieved significant progress in natural images but not in aerial images, due to the massive variations in the scale and orientation of objects caused by the bird's-eye view of aerial images. More importantly, the lack of large-scale benchmarks has become a major obstacle to the development of object detection in aerial images (ODAI). In this paper, we present a large-scale Dataset of Object deTection in Aerial images (DOTA) and comprehensive baselines for ODAI. The proposed DOTA dataset contains 1,793,658 object instances of 18 categories of oriented-bounding-box annotations collected from 11,268 aerial images. Based on this large-scale and well-annotated dataset, we build baselines covering 10 state-of-the-art algorithms with over 70 configurations, where the speed and accuracy performances of each model have been evaluated. Furthermore, we provide a code library for ODAI and build a website for evaluating different algorithms. Previous challenges run on DOTA have attracted more than 1300 teams worldwide. We believe that the expanded large-scale DOTA dataset, the extensive baselines, the code library and the challenges can facilitate the designs of robust algorithms and reproducible research on the problem of object detection in aerial images.

Non-local low-rank tensor approximation has been developed as a state-of-the-art method for hyperspectral image (HSI) restoration, which includes the tasks of denoising, compressed HSI reconstruction and inpainting. Unfortunately, while its restoration performance benefits from more spectral bands, its runtime also substantially increases. In this paper, we claim that the HSI lies in a global spectral low-rank subspace, and the spectral subspaces of each full band patch group should lie in this global low-rank subspace. This motivates us to propose a unified paradigm combining the spatial and spectral properties for HSI restoration. The proposed paradigm enjoys performance superiority from the non-local spatial denoising and light computation complexity from the low-rank orthogonal basis exploration. An efficient alternating minimization algorithm with rank adaptation is developed. It is done by first solving a fidelity term-related problem for the update of a latent input image, and then learning a low-dimensional orthogonal basis and the related reduced image from the latent input image. Subsequently, non-local low-rank denoising is developed to refine the reduced image and orthogonal basis iteratively. Finally, the experiments on HSI denoising, compressed reconstruction, and inpainting tasks, with both simulated and real datasets, demonstrate its superiority with respect to state-of-the-art HSI restoration methods.

Traditional target detection methods assume that the background spectrum is subject to the Gaussian distribution, which may only perform well under certain conditions. In addition, traditional target detection methods suffer from the problem of the unbalanced number of target and background samples. To solve these problems, this study presents a novel target detection method based on asymmetric weighted logistic metric learning (AWLML). We first construct a logistic metric-learning approach as an objective function with a positive semidefinite constraint to learn the metric matrix from a set of labeled samples. Then, an asymmetric weighted strategy is provided to emphasize the unbalance between the number of target and background samples. Finally, an accelerated proximal gradient method is applied to identify the global minimum value. Extensive experiments on three challenging hyperspectral datasets demonstrate that the proposed AWLML algorithm improves the state-of-the-art target detection performance.

Remote sensing image data clustering is a tough task, which involves classifying the image without any prior information. Remote sensing image clustering, in essence, belongs to a complex optimization problem, due to the high dimensionality and complexity of remote sensing imagery. Therefore, it can be easily affected by the initial values and trapped in locally optimal solutions. Meanwhile, remote sensing images contain complex and diverse spatial-spectral information, which makes them difficult to model with only a single objective function. Although evolutionary multiobjective optimization methods have been presented for the clustering task, the tradeoff between the global and local search abilities is not well adjusted in the evolutionary process. In this article, in order to address these problems, a multiobjective sine cosine algorithm for remote sensing image data spatial-spectral clustering (MOSCA_SSC) is proposed. In the proposed method, the clustering task is converted into a multiobjective optimization problem, and the Xie-Beni (XB) index and Jeffries-Matusita (Jm) distance combined with the spatial information term (SI_Jm measure) are utilized as the objective functions. In addition, for the first time, the sine cosine algorithm (SCA), which can effectively adjust the local and global search capabilities, is introduced into the framework of multiobjective clustering for continuous optimization. Furthermore, the destination solution in the SCA is automatically selected and updated from the current Pareto front through employing the knee-point-based selection approach. The benefits of the proposed method were demonstrated by clustering experiments with ten UCI datasets and four real remote sensing image datasets.

With the development of Earth observation technology, a very-high-resolution (VHR) image has become an important data source of change detection (CD). These days, deep learning (DL) methods have achieved conspicuous performance in the CD of VHR images. Nonetheless, most of the existing CD models based on DL require annotated training samples. In this article, a novel unsupervised model, called kernel principal component analysis (KPCA) convolution, is proposed for extracting representative features from multitemporal VHR images. Based on the KPCA convolution, an unsupervised deep siamese KPCA convolutional mapping network (KPCA-MNet) is designed for binary and multiclass CD. In the KPCA-MNet, the high-level spatial-spectral feature maps are extracted by a deep siamese network consisting of weight-shared KPCA convolutional layers. Then, the change information in the feature difference map is mapped into a 2-D polar domain. Finally, the CD results are generated by threshold segmentation and clustering algorithms. All procedures of KPCA-MNet do not require labeled data. The theoretical analysis and experimental results in two binary CD datasets and one multiclass CD datasets demonstrate the validity, robustness, and potential of the proposed method.

Deep learning techniques have been widely applied to hyperspectral image (HSI) classification and have achieved great success. However, the deep neural network model has a large parameter space and requires a large number of labeled data. Deep learning methods for HSI classification usually follow a patchwise learning framework. Recently, a fast patch-free global learning (FPGA) architecture was proposed for HSI classification according to global spatial context information. However, FPGA has difficulty in extracting the most discriminative features when the sample data are imbalanced. In this article, a spectral-spatial-dependent global learning (SSDGL) framework based on the global convolutional long short-term memory (GCL) and global joint attention mechanism (GJAM) is proposed for insufficient and imbalanced HSI classification. In SSDGL, the hierarchically balanced (H-B) sampling strategy and the weighted softmax loss are proposed to address the imbalanced sample problem. To effectively distinguish similar spectral characteristics of land cover types, the GCL module is introduced to extract the long short-term dependency of spectral features. To learn the most discriminative feature representations, the GJAM module is proposed to extract attention areas. The experimental results obtained with three public HSI datasets show that the SSDGL has powerful performance in insufficient and imbalanced sample problems and is superior to other state-of-the-art methods.

Feature selection is an effective way to handle the strong correlation of hyperspectral image data by screening the significant features, and is generally accepted to be a multiobjective optimization problem. Nevertheless, due to the randomness of the strategies and the ambiguity of the optimization directions, the existing multiobjective evolutionary optimization based feature selection methods can suffer from inefficient search and loss of search space with promising solutions when faced with the high-dimensional and multi-peak search space. The multiobjective evolutionary algorithm based on decomposition (MOEA/D) employs a decomposition framework to provide exact guidance for the optimization directions. Unfortunately, random operators are still used, leading to inadequate local optimization. Thus, evolutionary strategies with search preference such as clonal selection may be necessary for local search. In this paper, a novel decomposition-based multiobjective clonal selection algorithm for feature selection (MOCSA/D_FS) is proposed to obtain a feature subset with a superior classification performance. In MOCSA/D_FS, the information entropy and the ratio of the relative scatter value and mutual information are utilized as two objective functions to evaluate the information amount and redundancy. A series of subproblems are then obtained by decomposing the multiobjective problem through weight vectors, with anl2-norm constraint used to balance the search space. Subsequently, a clonal selection method with search space preference performs a detailed local search on each subproblem, which can fully exploit the potential optimal space. The effectiveness and generalizability of the proposed method was confirmed by experiments on four hyperspectral remote sensing image datasets.

Although deep learning has made semantic segmentation of very-high-resolution (VHR) remote sensing (RS) images practical and efficient, its large-scale application is still limited. Given the diversity of imaging sensors, acquisition conditions, and regional styles, a deep learning network well-trained on one source domain dataset often suffers from drastic performance drops when applied to other target domain datasets. Thus, we propose a novel end-to-end mutual information domain adaptation network (MIDANet) that can shift between semantic segmentation domains by integrating multitask learning in the convolutional neural networks within an entropy adversarial learning (EAL) framework. Through the joint learning of semantic segmentation and elevation estimation, the features extracted by MIDANet can concentrate more on the elevation clues while dropping the domain-variant information (i.e., texture, spectral information). First, one encoder is applied to excavate general semantic features. Two decoders that share the same architecture are used to perform pixel-level classification and digital surface model (DSM) regression. Second, feature interaction modules (FIMs) and a mutual information attention unit (MIAU) are designed to mine the latent relationships between the two tasks and enhance their feature representations. Finally, a final MIDANet is obtained for semantic segmentation that does not require any semantic segmentation labels in the target domain after the adversarial learning of the classification entropy at the output level. Extensive comparative experiments and ablation studies were conducted on the International Society for Photogrammetry and Remote Sensing (ISPRS) Potsdam and Vaihingen test datasets. The results show that MIDANet outperforms other state-of-the-art domain adaptation (DA) methods in both evaluation metrics and visual assessment.

The complementarity of synthetic aperture radar (SAR) and optical images allows remote sensing observations to “see” unprecedented discoveries. Image matching plays a fundamental role in the fusion and application of SAR and optical images. However, both the geometric imaging pattern and the physical radiation mechanism of these two sensors are significantly different, so that the images show complex geometric distortion and nonlinear radiation differences. This phenomenon brings great challenges to image matching, which neither the handcrafted descriptors nor the deep learning-based methods have adequately addressed. In this article, a novel image-based matching method for SAR to optical images via an image-based convolutional network with spatial pyramid aggregated pooling (SPAP) and an attention mechanism is proposed, namely MAP-Net. The original image is embedded through the convolutional neural network to generate the feature map. Through the information extraction and abstraction of the original imagery, the embedded features containing the high-level semantic information are more robust to the geometric distortion and radiation variation among the different modal images, which is beneficial to the matching of cross-modal images. The adoption of the SPAP module makes the network more capable of integrating global and local contextual information. The attention block weights the dense features generated from the network to extract the key features that are invariant, distinguishable, repeatable, and suitable for the image matching task. In the experiments, five sets of multisource and multiresolution SAR and optical images with wide and varied ground coverage were used to evaluate the accuracy of MAP-Net, compared to both handcrafted and deep learning-based methods. The experimental results show that the MAP-Net method is superior to the current state-of-the-art image matching methods for SAR to optical images.

Hyperspectral anomaly detection, which is aimed at locating anomaly, has received widespread attention. In this article, a new anomaly detector, named local spatial constraint and total variation (LSC-TV), is proposed for hyperspectral imagery. In anomaly detection methods based on low-rank representation, background pixels are usually considered to have a global low-dimensional structure. However, the complex background distribution in hyperspectral images (HSIs) means that this global low-dimensional structure rarely occurs. In LSC-TV, the effective local spatial information is extracted by superpixel segmentation, and the regularization based on the F-norm is used to force the background within the same superpixel to show uniform spectral features. Moreover, each pixel is given a penalty based on the degree of anomaly determined during model iteration, while the anomaly is not considered by the background constraint. In addition, the background pixels in the neighborhood often show a high correlation, whereas the anomaly does not possess this feature. Nonisotropic TV is introduced into the proposed LSC model using the correlation of first-order neighborhoods to make it easier for anomalies to be separated. The proposed LSC-TV method and current state-of-the-art methods are tested on a set of simulated data and four sets of real data. The experimental results demonstrate that the proposed method is superior to the comparative method in terms of both color map detection and quantitative evaluation.

Multispectral images play a crucial role in environmental monitoring or ecological analysis for their large scope, quick acquisition, and big data. With the rapid development of technology and increasing demand, very high-resolution multispectral images have attracted a lot of attention these days. However, due to sensor equipment and the imaging environment, the spatial resolution of multispectral images is always restricted. With the help of panchromatic images, pan-sharpening is a very important technique to enhance the spatial details of multispectral images. In this study, we proposed a knowledge optimization-driven pan-sharpening network with normalizer-free group ResNet prior, called PNXnet, which is unfolded from a physical knowledge optimization-driven variational model. We solved the memory overhead brought by the traditional ResNet relying on batch normalization. Results on four sensors show that high quantitative indexes and natural visual effects have verified the reliability of PNXnet. Focusing on the NIR band where spatial details are hard to be injected, we compared the Normalized Difference Vegetation Index (NDVI) generated from the fused results, the estimated NDVI shows a high consistency to the ground truth with R2above 0.91. Besides, we also compared the model generation. Furthermore, low model complexity and quicker computational speed make the daily application of PNXnet possible.

The fast development of self-supervised learning lowers the bar learning feature representation from massive unlabeled data and has triggered a series of researches on change detection of remote sensing images. Challenges in adapting self-supervised learning from natural images classification to remote sensing images change detection arise from difference between the two tasks. The learned patch-level feature representations are not satisfying for the pixel-level precise change detection. In this paper, we proposed a novel pixel-level self-supervised hyperspectral spatial-spectral understanding network (HyperNet) to accomplish pixel-wise feature representation for effective hyperspectral change detection. Concretely, not patches but the whole images are fed into the network and the multi-temporal spatial-spectral features are compared pixel by pixel. Instead of processing the two-dimensional imaging space and spectral response dimension in hybrid style, a powerful spatial-spectral attention module is put forward to explore the spatial correlation and discriminative spectral features of multi-temporal hyperspectral images (HSIs), separately. Only the positive samples at the same location of bi-temporal HSIs are created and forced to be aligned, aiming at learning the spectral difference-invariant features. Moreover, a new similarity loss function named focal cosine is proposed to solve the problem of imbalanced easy and hard positive samples comparison, where the weights of those hard samples are enlarged and highlighted to promote the network training. Six hyperspectral datasets have been adopted to test the validity and generalization of proposed HyperNet. The extensive experiments demonstrate the superiority of HyperNet over the state-of-the-art algorithms on downstream hyperspectral change detection tasks.

In deep learning (DL)-based hyperspectral imagery classification, “spatial patching” is primarily used as a preprocessing for incorporating local spatial information. This operation can help to promote classification accuracy but it is facing new challenges in the unmanned aerial vehicle (UAV)-borne hyperspectral imagery with high spatial and spectral resolutions (H2imagery). The ground objects’ various spatial scales result in it being challenging to determine the optimal size for the spatial patches. In addition, due to the severe spectral variability and spatial heterogeneity of the H2imagery, “spatial patching” only exploits the local spatial information and results in serious salt-and-pepper (SP) noise and isolated areas in the classification maps. In this article, to address these issues, a novel spectral patching network (SPNet) with an end-to-end DL architecture is proposed for UAV-borne H2imagery classification. The “spectral patching” approach is proposed to preserve the global spatial information and almost all the spectral information of the original hyperspectral imagery. An end-to-end deep encoder–decoder network is then constructed based on the spectral patching mechanism, which introduces the deep residual network (ResNet) and atrous spatial pyramid pooling (ASPP) modules to extract multiscale high-level semantic information for the H2imagery classification. The experimental results obtained with the Wuhan UAV-borne H2imagery (WHU-Hi) UAV-borne hyperspectral data set demonstrate that SPNet can achieve state-of-the-art accuracy and visualization performance in the classification of H2imagery.

Fine particulate matter (PM2.5) pollution can cause serious public health problems worldwide. A novel geographically and temporally weighted neural network constrained by global training (GC-GTWNN) is proposed in this article for the remote sensing estimation of surface PM2.5. The global neural network (NN) is trained to learn the overall effect of the influencing variables on surface PM2.5, and the local geographically and temporally weighted NN (GTWNN) addresses the spatiotemporal heterogeneity of the relationship between PM2.5and the influencing variables. Specifically, a global NN is trained with all samples collected from the entire study domain and period. Then, initialized with the global NN, the GTWNN models are built for each location and time and fine-tuned via spatiotemporally localized samples. Meanwhile, the geographically weighted loss function is designed for GTWNN. The proposed GC-GTWNN modeling is tested with a case study across China, which integrates satellite aerosol optical depth, surface PM2.5measurements, and auxiliary variables. Cross-validation results indicate that a remarkable improvement is observed from the global NN to GC-GTWNN modeling ($R^{2}$value increasing from 0.49 to 0.80), and GC-GTWNN modeling also notably outperforms the conventionally popular PM2.5estimation models.

Change detection (CD) aims to distinguish surface changes based on bitemporal images. Since high-resolution (HR) images cannot be typically acquired continuously over time, bitemporal images with different resolutions are often adopted for CD in practical applications. Traditional subpixel-based methods for CD using images with different resolutions may lead to substantial error accumulation when the HR images are employed, which is because of intraclass heterogeneity and interclass similarity. Therefore, it is necessary to develop a novel method for CD using images with different resolutions that are more suitable for the HR images. To this end, we propose a super-resolution-based change detection network (SRCDNet) with a stacked attention module (SAM). The SRCDNet employs a super-resolution (SR) module containing a generator and a discriminator to directly learn the SR images through adversarial learning and overcome the resolution difference between the bitemporal images. To enhance the useful information in multiscale features, a SAM consisting of five convolutional block attention modules (CBAMs) is integrated to the feature extractor. The final change map is obtained through a metric learning-based change decision module, wherein a distance map between bitemporal features is calculated. Ablation study and comparative experiments on two large datasets, building change detection dataset (BCDD) and season-varying change detection dataset (CDD), and a real-image experiment on the Google dataset fully demonstrate the superiority of the proposed method. The source code of SRCDNet is available athttps://github.com/liumency/SRCDNet.

Target tracking has received increased attention in the past few decades. However, most of the target tracking algorithms are based on RGB video data, and few are based on hyperspectral video data. With the development of the new “snapshot” hyperspectral sensors, hyperspectral videos can now be easily obtained. However, hyperspectral video target tracking datasets are still rare. In this article, a high spectral-spatial-temporal resolution hyperspectral video target tracking algorithm framework (H3Net) based on deep learning is proposed. The proposed framework consists of two main parts: 1) an unsupervised deep learning-based target tracking training framework for hyperspectral video; and 2) a dual-branch network structure based on a Siamese network. Using the dual-branch network, the H3Net framework can utilize both the spatial and spectral information. The combination of deep learning and a discriminative correlation filter (DCF) makes the features extracted by deep learning more suitable for the DCF. Compared with hyperspectral images, hyperspectral video data require more manpower to annotate, so we propose an unsupervised approach to train H3Net, without any annotation. To solve the problem of the lack of hyperspectral video datasets, we built a 25-band hyperspectral video dataset (the high spectral-spatial-temporal resolution hyperspectral video dataset: the WHU-Hi-H3dataset) for target tracking. The experimental results obtained with the WHU-Hi-H3dataset confirm the potential of unsupervised deep learning in hyperspectral video target tracking.

Road extraction from very high-resolution (VHR) remote sensing imagery remains a huge challenge, due to the shadows and occlusions of trees and buildings. Such complex backgrounds result in deep networks often producing fragmented roads with poor connectivity. Road extraction has three typical tasks: road surface segmentation (SS), centerline extraction (CE), and edge detection (ED), which are conducted in a wide range of real applications. Also, the three tasks have a symbiotic relationship, i.e., the road SS determines the location of the centerline and edges, and the CE and ED can allow the generation of more continuous road surfaces. However, most of the previous works have completed these three tasks separately, without exploiting the symbiotic relationship between them to boost the road connectivity. In this article, in order to improve road connectivity, a cascaded multitask (CasMT) road extraction framework for simultaneously extracting the road surface, centerline, and edges is proposed. In the proposed framework, topology-aware learning is applied to capture the long-distance topological relationships, and hard example mining (HEM) loss is employed to focus more on hard samples, to further enhance the road completeness. Extensive experiments were conducted on the DeepGlobe road dataset and a large-scale road dataset (called the LSCC dataset) from the three Chinese cities of Beijing, Shanghai, and Wuhan. The experimental results obtained on the public DeepGlobe dataset demonstrate that the proposed CasMT framework can significantly outperform the current state-of-the-art method. Moreover, the generalization capability of the model was verified on the LSCC dataset, where the proposed CasMT framework achieved the best performance in the average path length similarity (APLS) road topology metric, which further confirms the superiority of the proposed framework.

Convolutional neural network (CNN)-based methods are widely used in remote sensing image scene classification and can obtain excellent performances. However, the stacked receptive fields in the CNN-based methods have limitations in modeling the long-range dependencies of local features. The vision transformer (ViT) model provides a good solution as it directly considers the global interactions of local patches by the self-attention mechanism. However, the vanilla ViT model, which simply splits images into fixed-size patches treated as tokens, mainly considers the global information in the spatial domain. In this article, a spatial-channel feature preserving ViT (SCViT) model is proposed, which considers both the detailed geometric information of the high-spatial-resolution (HSR) imagery and the contribution of the different channels contained in the classification token. First, in the proposed method, tokens are generated by progressively aggregating the neighboring overlapping patches to extract the local structural features of the imagery. Second, a multihead self-attention (MSA) mechanism is used to model the global interactions of the tokens in the encoder. A lightweight channel attention (LCA) module is then introduced to consider the importance of the different channels in the classification token. Finally, a multilayer perceptron (MLP) is used to acquire the final results. Compared with the state-of-the-art scene classification methods, the experimental results confirm the potential of using ViT models in remote sensing image scene classification.

Remote sensing image scene classification is a challenging task. With the development of deep learning, methods based on convolutional neural networks (CNNs) have made great achievements in remote sensing image scene classification. Since the training of a CNN requires a large number of labeled samples, a generative adversarial network (GAN) for sample generation represents a new opportunity to solve the problem of the limited samples. However, most of the existing GAN-based sample generation methods can only generate unlabeled samples, instead of samples labeled with the corresponding scene category. In this article, to solve the problem, a supervised progressive growing generative adversarial network (SPG-GAN) is proposed for remote sensing image scene classification. The proposed method can generate labeled samples for the remote sensing image scene classification, significantly improving the classification accuracy in the case of limited samples. The SPG-GAN method has two main improvements. First, a conditional generative framework for labeled samples is proposed, in which the label information is added in the channel dimension as the input. By considering the constraints of the label information in the loss function, the network can be trained in the direction of a specific category. As a result, the network can generate remote sensing image scene classification samples with label categories. Second, a progressive growing sample generation method is introduced. In order to ensure that the generated samples have more spatial details, they are generated by progressively adding modules to the generator and discriminator, thereby ensuring that the generated sample is of better quality. After testing on two benchmark datasets and carrying out a large-scale experiment in the central area of the city of Wuhan in China, it was found that the proposed method can obtain a superior scene classification accuracy in the case of limited samples.

Image super-resolution (SR) is an effective solution to the limitation of the spatial resolution of video satellite images, which is caused by the degradation and compression in the imaging phase. For the processing of satellite videos, the commonly employed deep-learning-based single-frame SR (SFSR) framework has limited performance without using complementary information between the video frames. On the other side, the multiframe SR (MFSR) can utilize temporal subpixel information to super-resolve the high-resolution (HR) imagery. However, although deeper and wider deep learning network provides powerful feature representations for SR methods, it has always been a challenge to accurately reconstruct the boundaries of ground objects in video satellite images. In this article, to address these issues, we propose an edge-guided video SR (EGVSR) framework for video satellite image SR, which couples the MFSR model and the edge-SFSR (E-SFSR) model in a unified network. The EGVSR framework is composed of an MFSR branch and an edge branch. The MFSR branch is used to extract the complementary features from the consecutive video frames. Concurrently, the edge branch acts as an SFSR model to translate the edge maps from the low-resolution modality to the HR one. At the final SR stage, the DBFM is built to focus on the promising inner representations of the features of the two branches and fuse them. Extensive experiments on video satellite imagery show that the proposed EGVSR method can achieve superior performance compared to the representative deep-learning-based SR methods.

Change detection (CD) aims to identify surface changes from bitemporal images. In recent years, deep learning (DL)-based methods have made substantial breakthroughs in the field of CD. However, CD results can be easily affected by external factors, including illumination, noise, and scale, which leads to pseudo-changes and noise in the detection map. To deal with these problems and achieve more accurate results, a deeply supervised (DS) attention metric-based network (DSAMNet) is proposed in this article. A metric module is employed in DSAMNet to learn change maps by means of deep metric learning, in which convolutional block attention modules (CBAM) are integrated to provide more discriminative features. As an auxiliary, a DS module is introduced to enhance the feature extractor’s learning ability and generate more useful features. Moreover, another challenge encountered by data-driven DL algorithms is posed by the limitations in change detection datasets (CDDs). Therefore, we create a CD dataset, Sun Yat-Sen University (SYSU)-CD, for bitemporal image CD, which contains a total of 20 000 aerial image pairs of size$256\times256$. Experiments are conducted on both the CDD and the SYSU-CD dataset. Compared to other state-of-the-art methods, our network achieves the highest accuracy on both datasets, with an F1 of 93.69% on the CDD dataset and 78.18% on the SYSU-CD dataset.

High spatial resolution (HSR) remote sensing images can reflect more subtle changes and more specific types of land use and land cover (LULC) due to the abundant spatial geometric information. In this article, a class-prior object-oriented conditional random field (COCRF) framework consisting of a binary change detection (CD) task and a multiclass CD task is proposed to fill the application gap. In the proposed framework, the class-prior knowledge is used to improve the construction of the unary potential in both the binary and multiclass CD tasks, to reduce the influence of spectral variability. The binary CD result provides a constraint to the multiclass CD result. As a result, both parts have effective interaction. The class posterior probability images of two dates can be obtained automatically with the class-prior knowledge by sample migration. Furthermore, an object constraint described by the class dispersion within the objects is added to improve the smoothness in local objects, while the pairwise potential improves the smoothness of the whole area by using the eight-neighborhood spectral information of the center pixel. By integrating the above approaches, the problems of error accumulation and the manual intervention required in the traditional multiclass CD methods can be relieved. An adaptive parameter estimation strategy is also adopted in the proposed framework, to save the time required for manual parameter setting. The proposed COCRF framework was validated on two HSR remote sensing image data sets, where it achieved a better performance than the other state-of-the-art CD methods.

Spatiotemporal fusion (STF) is an effective solution to promote the application of remote sensing images, given that the tradeoff between the temporal resolution and the spatial resolution is ubiquitous in the production of remote sensing images. However, cloud coverage makes it difficult to obtain dense cloud-free Landsat–Moderate Resolution Imaging Spectroradiometer (MODIS) image pairs on the timeline, which limits the application of existing STF methods. Considering the lack of prior image pairs and the huge spatial resolution gap between Landsat and MODIS images, this article presents a novel remote sensing image STF method based on a generative adversarial network to handle one Landsat–MODIS prior image pair case (OPGAN), which contains a generator and a discriminator simultaneously trained in a min–max game. OPGAN is built based on the STF observation model that learns the base information from the prior Landsat image and then captures temporal change (TC) information from a difference image constructed from MODIS images collected at times 1 and 2 and sensor difference information from the difference image between Landsat and MODIS images at time 1. They are combined together to reconstruct the Landsat image at time 2 at both high spatial and high temporal resolution. Moreover, a change loss is proposed to further improve the accuracy of TC prediction. Extensive experiments on the STF dataset illustrate that the proposed OPGAN method can obtain more accurate prediction of spatial information and TCs in the case of insufficient prior information.

Conventional subpixel mapping (SPM) is performed based on the abundance maps obtained by spectral unmixing (SU), to interpret the mixed pixels and improve the mapping resolution for hyperspectral remote-sensing imagery. However, the SU and SPM tasks are separately conducted, so that the unmixing error is propagated to SPM, and the mapping result is strongly reliant on the quality of the abundance maps. In this article, a novel joint SPM and SU framework (MO_SUSM) based on multiobjective optimization is proposed to simultaneously perform unmixing and mapping. Specifically, the multiobjective joint optimization model with a data fidelity term and a Laplacian prior term is constructed for SU and SPM. For the data fidelity term, since the unmixing result can be recovered by downsampling the mapping result, the unmixing model is joined with the mapping model by the downsampling matrix, so that the reconstruction errors of the unmixing and mapping results can be minimized together. Meanwhile, the Laplacian prior term is used to maximize the spatial dependence of the mapping result and provide the spatial constraint for SU. In addition, the multiobjective optimization algorithm with local search is designed to search for the optimal unmixing and mapping results that can balance the objective terms. Since the two objective terms are dynamically integrated during optimization, there is no need to set sensitive weights for the objectives combination. Four experiments were conducted on hyperspectral images of various data sources, including ground, airborne, and satellite images. The unmixing results show that MO_SUSM can reduce the unmixing error and can improve the quality of the abundance maps. The mapping results show that MO_SUSM can alleviate the dependence of SPM on the abundance maps and can improve the mapping accuracy.

With the acceleration of urbanization, it is essential to carry out change detection (CD) and obtain surface change information in urban areas. In the early stages, the spectral information of remote sensing images was used as a change index to capture the spectral and texture changes of ground objects in a two-dimensional plane. However, due to the dense buildings in urban areas, shadows and occlusions can be easily formed, and spectral information is sensitive to the imaging environment, such as the illumination, atmospheric conditions, and imaging angles, so that the detection results based on remote sensing images can often be incomplete. Most changes include not only 2-D plane changes but also 3-D elevation changes. Compared with spectral information, the elevation is more stable and more resistant to interference. Therefore, the fusion of remote sensing image and digital surface model (DSM) data has the potential to be used to detect the changes in urban areas. In this article, we propose a complementary evidence fusion 3-D CD framework based on the Dempster–Shafer theory (CDST). In this framework, DSM and normalized difference vegetation index (NDVI) data are combined using a complementary evidence combination rule. The DSM data can effectively overcome the impact of shadows, and the NDVI data can capture the relevant changes of height-insensitive ground objects, such as vegetation and water. When mapping the basic probability assignment (BPA) of the difference image (DI), prior knowledge is used to ensure that the BPA is not affected by the data distribution. Since DSM and remote sensing image data are heterogeneous data, there is a high degree of conflict when representing the change information characteristics of specific areas. For example, the change between grassland and road is small in elevation but significant in the spectral details, and the traditional Dempster’s combination rule no longer applies. The proposed CDST framework uses a complementary evidence combination rule, which can effectively alleviate the conflicts between the evidence sources and improve the integrity of the detected changes. The experimental results obtained on real datasets confirm that the proposed method does indeed perform well.

In this article, we propose fully contextual networks (FullyContNets) for hyperspectral scene parsing. Different from the previous approaches that leveraging the local information, the proposed methods can effectively capture the more generic nonlocal contexts. To this end, we first propose the scale attention module (SAM) that can adaptively aggregate the multiple features through obtaining the interfeature dependencies of multiscale with self-attention mechanism, where the weights are determined by measuring the similarity between features. What is more, two fully contextual modules (FCMs) called pyramid fully contextual module (Pyramid-FCM) and atrous spatial pyramid fully contextual module (ASP-FCM) are separately developed to obtain the contextual information that simultaneously lying across positions, channels, and features when combining the intrafeature information aggregation algorithms with SAM on the foundation of existing multiscale modules, such as pyramid pooling (PP) in PSPNet and atrous spatial pyramid pooling (ASPP) in DeeplabV3. We design four schemes for FCMs to obtain more effective contexts. The corresponding FullyContNet-Pyramid and FullyContNet-ASP are separately constructed based on the Pyramid-FCM or ASP-FCM. There are extensive quantitative and qualitative experiments are conducted, depicting the capability of SAM and FCMs and demonstrating the competitiveness of proposed networks on four public hyperspectral scenes when comparing with the current state-of-the-art approaches.

Hyperspectral images (HSIs) are inevitably corrupted with various types of noise, which seriously degrades the data quality and usability. Denoising is an essential preprocessing task of HSI processing. Recently, benefiting from the great learning ability of deep learning, convolutional neural network (CNN) denoisers have obtained state-of-the-art performances for Gaussian noise removal. However, one central problem remains largely unsolved: how to deal with the complicated noise in the real-world HSIs, especially when a paired training data set is unavailable. In this article, a self-supervised hyperspectral image denoising network (SHDN) is proposed, which consists of a noise estimator and a CNN denoiser. Rather than defining a complex noise model to generate training pairs on the clean HSIs, a self-supervised training scheme is first proposed by considering the noisy HSI itself as the training data. Through the noise estimator, the realistic noise samples can be extracted and combined with the clean bands to make up the training pairs. In addition, to jointly restore the target noisy band and to maintain the spectral consistency, a flexible multi-to-single band convolutional network is designed, where the noisy band and the neighboring bands are jointly aggregated via multiscale contextualized dilated blocks and the spectral–spatial convolutional unit. Experiments on HSIs from spaceborne, airborne, unmanned aerial vehicle (UAV)-borne, and ground-based data sets demonstrate the applicability and the generalization of SHDN in the real scenarios. Additionally, the usability of the noisy bands and the suitability of the SHDN framework in the subsequent applications are verified in the land-cover mapping experiments.

Hyperspectral images (HSIs) are generally distorted by various types of damage and degradation due to limited imaging conditions. Hence, noise reduction is an essential process before HSI interpretations and applications. In this paper, a novel local-global feature-aware transformer based residual network (FATR) is proposed for hyperspectral image denoising. First, a spatial-spectral feature extraction module is built to extract spatial and spectral shallow features simultaneously. Second, these spatial-spectral features are forwarded to the deep feature extraction module, which contains several local-global feature-aware transformer blocks, where contextual information as well as local and global information can be further aggregated by multiscale windows transformer layers. Finally, in the reconstruction module, different hierarchical features from branches of two modules are merged into the final restoration to recover clean HSIs. Extensive experiments on both synthetic and real-world data demonstrate that the model has a better ability to restore HSIs in terms of evaluation metrics and visual assessments.

Hyperspectral anomaly detection is aimed at detecting observations that differ from their surroundings, and is an active area of research in hyperspectral image processing. Recently, autoencoders (AEs) have been applied in hyperspectral anomaly detection; however, the existing AE-based methods are complicated and involve manual parameter setting and preprocessing and/or postprocessing procedures. In this article, an autonomous hyperspectral anomaly detection network (Auto-AD) is proposed, in which the background is reconstructed by the network and the anomalies appear as reconstruction errors. Specifically, through a fully convolutional AE with skip connections, the background can be reconstructed while the anomalies are difficult to reconstruct, since the anomalies are relatively small compared to the background and have a low probability of occurring in the image. To further suppress the anomaly reconstruction, an adaptive-weighted loss function is designed, where the weights of potential anomalous pixels with large reconstruction errors are reduced during training. As a result, the anomalies have a higher contrast with the background in the map of reconstruction errors. The experimental results obtained on a public airborne data set and two unmanned aerial vehicle-borne hyperspectral data sets confirm the effectiveness of the proposed Auto-AD method.

Hyperspectral anomaly detection is aimed at detecting observations that differ from their surroundings. To achieve this goal, low-rank models and autoencoders (AEs) have attracted a lot of attention. Although the low-rank model is self-explainable, a low-rank prior may not completely match real data. In contrast, AEs can automatically learn the discriminative features between anomalies and background, whereas AEs are not self-explainable. In this article, a deep low-rank prior-based method (DeepLR) is proposed, which combines a model-driven low-rank prior and a data-driven AE. To be specific, the low-rank prior and a fully convolutional AE architecture are incorporated through modeling an energy minimization problem solved by an iterative optimization framework, in which low-rank background estimation and network training serve as two subproblems. The low-rank background is input into the network to calculate a low-rank regularized loss, constraining the training of the network. Finally, the background can be approximately reconstructed, while the anomalies are reconstructed with significant reconstruction errors; thus, the reconstruction errors indicate the anomalous degree. The experimental results obtained on several public datasets and two large unmanned aerial vehicle (UAV)-borne datasets confirm the merit and viability of the proposed method.

With the development of video satellites, multimoving object tracking in satellite video is possible and has become a new challenging task. The difficulties are mainly caused by the characteristics of satellite videos: 1) small objects; 2) low contrast between objects and background; and 3) background in a state of continuous motion. These characteristics make it difficult for the advanced multiobject tracking algorithms in the natural video to give full play to their advantages, resulting in vast false alarms, missed objects, ID switches, and low-confidence bounding boxes. To tackle these problems, a novel multimoving object tracking method considering slow features (SFs) and motion features has been proposed in this research, named SF and motion feature-guided multiobject tracking (SFMFMOT), which realizes the continuous tracking of moving vehicles in satellite videos. A nonmaximum suppression (NMS) module guided by bounding box proposals based on SFs is designed to assist the object detection part by utilizing the sensitivity of SF analysis to the changed pixels. While removing a large number of static false alarms and supplementing missed objects, it improves the recall rate by increasing the confidence score of the correctly detected object bounding boxes. In order to improve the tracking performance, a set of optimization strategies based on motion features and time accumulation information are proposed to smooth the trajectory, remove static false alarms, and duplicate bounding boxes. The proposed method is evaluated in three satellite videos and its superiority is demonstrated.

Enhancement of hyperspectral image (HSI) resolution is significant for better application in practice. In this article, a dual U-Net (D-UNet) is proposed to improve the spatial resolution of HSI. The whole network contains two parts. One is the detail extraction network, whose network architecture is encoder–decoder and mainly extracts various spatial features from multispectral images (MSIs). Another is the spatio-spectral fusion network (SSFN), which aims at injecting the features from the detail extraction network into HSI for better reconstruction. Furthermore, in the primary stage of the whole network, a novel multiscale spatio-spectral attention module (MSSAM) is utilized to pay more attention to important features at different scales. Considering the complex ground scenes, the features of different scale and depth are continually extracted and fused in the whole network. The experimental results show that the proposed method is more effective compared with the state-of-the-art methods.

As a new earth observation tool, satellite video has been widely used in remote-sensing field for dynamic analysis. Video super-resolution (VSR) technique has thus attracted increasing attention due to its improvement to spatial resolution of satellite video. However, the difficulty of remote-sensing image alignment and the low efficiency of spatial–temporal information fusion make poor generalization of the conventional VSR methods applied to satellite videos. In this article, a novel fusion strategy of temporal grouping projection and an accurate alignment module are proposed for satellite VSR. First, we propose a deformable convolution alignment module with a multiscale residual block to alleviate the alignment difficulties caused by scarce motion and various scales of moving objects in remote-sensing images. Second, a temporal grouping projection fusion strategy is proposed, which can reduce the complexity of projection and make the spatial features of reference frames play a continuous guiding role in spatial–temporal information fusion. Finally, a temporal attention module is designed to adaptively learn the different contributions of temporal information extracted from each group. Extensive experiments on Jilin-1 satellite video demonstrate that our method is superior to current state-of-the-art VSR methods.

Can we automatically learn meaningful semantic feature representations when training labels are absent? Several recent unsupervised deep learning approaches have attempted to tackle this problem by solving the data reconstruction task. However, these methods can easily latch on low-level features. To solve this problem, we propose an end-to-end spectral–spatial semantic feature learning network (S3FN) for unsupervised deep semantic feature extraction (FE) from hyperspectral images (HSIs). Our main idea is to learn spectral-spatial features from high-level semantic perspective. First, we utilize the feature transformation to obtain two feature descriptions of the same source data from different views. Then, we propose the spectral–spatial feature learning network to project the two feature descriptions into the deep embedding space. Subsequently, a contrastive loss function is introduced to align the two projected features, which should have the same implied semantic meaning. The proposed S3FN learns the spectral and spatial features separately, and then merges them. Finally, the learned spectral–spatial features by S3FN are processed by a classifier to evaluate their effectiveness. Experimental results on three publicly available HSI datasets show that our proposed S3FN can produce promising classification results with a lower time cost than other state-of-the-art (SOTA) deep learning-based unsupervised FE methods.

Due to the lack of high-quality labeled data and poor generalization ability of supervised models in remote sensing scene classification, cross-domain scene classification is proposed to better utilize the existing knowledge to classify the unlabeled data. Since there is a data distribution difference between the training (source domain) and test (target domain) set, many deep domain adaptation methods have been proposed to reduce such distribution discrepancy. However, existing deep domain adaptation methods usually use the discrepancy metric function to align the marginal distribution and do not consider the effect of each sample in different domains on the network weights. In addition, the existing domain adaptation methods cannot adaptively balance the relative importance of marginal and conditional distributions well. To overcome the above shortcomings, we propose a novel Attention-based Dynamic Alignment and Dynamic Distribution Adaptation (ADA-DDA) method to better align the marginal distribution between different distributions by calculating the dynamic weights of each sample in different domains and dynamically balance the relative importance of marginal and conditional distributions. Moreover, the attention mechanism enables purposeful knowledge transfer, so that the extracted features can be highly discriminative. The experimental results demonstrate that our proposed method is superior to the other state-of-the-art deep domain adaptation methods in the comparison, and outperforms the second place in accuracy by 5.36%.

Targeting at depicting land covers with pixelwise semantic categories, semantic segmentation in remote sensing images needs to portray diverse distributions over vast geographical locations, which is difficult to be achieved by the homogeneous pixelwise forward paths in the architectures of existing deep models. Although specific algorithms have been designed to select pixelwise adaptive forward paths for natural image analysis, it still lacks theoretical supports on how to obtain optimal selections. In this article, we provide mathematical analyses in terms of the parameter optimization, which guides us to design a method called hidden path selection network (HPS-Net). With the help of hidden variables deriving from an extra mini-branch, HPS-Net is able to tackle the inherent problem about inaccessible global optimums by adjusting the direct relationships between feature maps and pixelwise path selections in existing algorithms, which we call hidden path selection. For the better training and evaluation, we further refine and expand the 5-class Gaofen image dataset (GID-5) to a new one with 15 land-cover categories, i.e., GID-15. The experimental results on both GID-5 and GID-15 demonstrate that the proposed modules can stably improve the performance of different deep structures, which validates the proposed mathematical analyses.

Given two multitemporal aerial images, semantic change detection (SCD) aims to locate the land-cover variations and identify their change types with pixelwise boundaries. This problem is vital in many earth vision-related tasks, such as precise urban planning and natural resource management. Existing state-of-the-art algorithms mainly identify the changed pixels by applying homogeneous operations on each input image and comparing the extracted features. However, in changed regions, totally different land-cover distributions often require heterogeneous feature extraction procedures for images acquired at different times. In this article, we present an asymmetric Siamese network (ASN) to locate and identify semantic changes through feature pairs obtained from modules of widely different structures, which involves areas of various sizes and applies different quantities of parameters to factor in the discrepancy across land-cover distributions during different times. To better train and evaluate our model, we create a large-scale well-annotated SEmantic Change detectiON Dataset (SECOND), while an adaptive threshold learning (ATL) module and a separated kappa (SeK) coefficient are proposed to alleviate the influences of label imbalance in model training and evaluation. The experimental results demonstrate that the proposed model can stably outperform the state-of-the-art algorithms with different encoder backbones.

Hyperspectral images (HSIs) have a wealth of applications in many areas, due to their fine spectral discrimination ability. However, in the practical imaging process, HSIs are often degraded by a mixture of various types of noise, for example, Gaussian noise, impulse noise, dead pixels, dead lines, and stripe noise. Low-rank matrix decomposition theory has been widely used in HSI denoising, and has achieved competitive results by modeling the impulse noise, dead pixels, dead lines, and stripe noise as sparse components. However, the existing low-rank-based methods for HSI denoising cannot completely remove stripe noise when the stripe noise is no longer sparse. In this article, we extend the HSI observation model and propose a double low-rank (DLR) matrix decomposition method for HSI denoising and destriping. By simultaneously exploring the low-rank characteristic of the lexicographically ordered noise-free HSI and the low-rank structure of the stripe noise on each band of the HSI, the two low-rank constraints are formulated into one unified framework, to achieve separation of the noise-free HSI, stripe noise, and other mixed noise. The proposed DLR model is then solved by the augmented Lagrange multiplier (ALM) algorithm efficiently. Both simulation and real HSI data experiments were carried out to verify the superiority of the proposed DLR method.

The lack of labels is one of the major challenges in hyperspectral image (HSI) classification. Widely used Deep Learning (DL) models such as convolutional neural networks (CNNs) experience serious performance degradation when training samples are limited. In contrast, graph convolutional networks (GCNs) can simultaneously exploit the insufficient labeled data and massive unlabeled data of HSI in a semisupervised learning fashion. However, in order to reduce computational cost and mitigate noise, existing GCN-based classification methods usually perform superpixel segmentation as a preprocessing step and implement feature extraction as well as node classification on the predefined superpixel graph, where one superpixel might incorporate pixels with different labels. Moreover, the local spectral–spatial information within superpixels is generally ignored. To alleviate these two issues, we propose an end-to-end mixhop superpixel-based GCN (EMS-GCN) framework for HSI classification. Specifically, we first introduce the differentiable superpixel segmentation algorithm to map the pixel representations into a superpixel feature space, which allows refining the superpixel boundary with the training of the network. After that, a superpixel graph is constructed and fed into a novel mixhop superpixel-based GCN, where both the local information within superpixels and long-range information among superpixels are extracted, while the structure of the superpixel graph is updated at the same time. Finally, the enhanced superpixel representations are mapped back into a pixel feature space to conduct pixel-wise classification. Extensive experiments demonstrate the effectiveness of the proposed EMS-GCN method compared with other state-of-the-art methods.

Hyperspectral imagery (HSI) is one of the emerging tools to explore the physical properties and chemical composition of the lunar surface. Moon mineralogy mapper (M3) is the most widely used lunar HSI data set with the widest coverage and the excellent resolution; however, dense and nonperiodic stripes distributed across all bands in M3images hinder visual interpretation as well as their use in subsequent applications. In this article, a fast destriping approach for M3is proposed using the Hodrick–Prescott decomposition embedded in the low-rank framework (LRHP) to overcome this limitation. The integration of a statistical filter and variational model tackles the problem stemming from a lack of the correct residual information when certain pixels are corrupted in every band, thereby restoring severely degraded hyperspectral images (HSIs). Simulated and real experiments conducted on typical regions on the Moon with various levels of corruption demonstrate that the proposed LRHP rapidly achieves favorable performance against state-of-the-art approaches. Also, expanding tests on interference imaging spectrometer (IIM) data of Chang’E-1 and commonly used terrestrial remote sensing images show that LRHP has good generalization capability. Moreover, the integrated band depth (IBD) maps further verify the necessity of destriping and the high spectral fidelity of LRHP that benefits further applications.

Hyperspectral image (HSI) anomaly detection is an important task in remote sensing domain. In recent years, many scholars have been addicted to constructing deep network-based methods for hyperspectral anomaly detection and have developed numerous related methods. Many of them are designed based on autoencoder, which aims to reconstruct a stable background to identify anomalies. However, these autoencoder-based methods suffer from some problems, such as ignoring the inter-band correlation in HSI. That is, the hyperspectral image presents spectral similarity as well as redundancy between the contiguous bands, which would affect the reconstruction of the HSI. Moreover, the current anomaly detectors lack the use of spatial contextual information that exists in the pixel neighbor region when constructing the detector. To tackle these problems, this study presents a spatial-spectral joint reconstruction with the inter-band correlation based anomaly detector (denoted as SSRICAD) for hyperspectral images. We first divide the original HSI into several sub-HSIs by a band cross-grouping strategy to reduce the redundancy and impose the inter-band correlation constraint into the reconstruction process. Then, an outlier removal constraint is added to alleviate anomaly contamination, which could help rebuild a more stable and pure background component. Finally, spatial information is extracted from the pixel neighbor region to contribute to the spatial-spectral joint reconstruction and further enhance detection performance. Extensive experiments on three benchmark hyperspectral datasets indicate that the proposed SSRICAD can achieve superior performance in anomaly detection.

In recent years, classification has obtained ever-rising attention and has been applied to many areas in the field of remote sensing, including land use, forest monitoring, urban planning, and vegetation management. Due to the lack of labeled data and the poor generalization ability of supervised models, cross-scene classification is proposed for better utilization of the existing knowledge. Existing adaptation methods for cross-scene classification only consider the marginal distribution, while the conditional distribution is equally important in real applications. In addition, approaches based on deep learning align the distribution of features extracted from a single-scale structure, leading to the loss of information. To overcome the above drawbacks, an Attention-based Multiscale Residual Adaptation Network (AMRAN) is proposed for cross-scene classification tasks. In the proposed AMRAN, both the marginal and conditional distributions are taken into consideration for more comprehensive alignment. Besides, the attention mechanism and the multiscale strategy are used to extract more robust features and more complete information, respectively. Experimental results between four existing scene classification data sets demonstrate that AMRAN has a significant improvement compared with the state-of-the-art deep adaptation methods.

Oil spills have caused serious harm to the marine environment. Remote sensing technology is one of the important tools for marine environment monitoring. Synthetic aperture radar (SAR) has become an important technology for detecting marine pollution. Identifying dark spots is essential for oil spill detection based on SAR images. Dark spots’ detection can be achieved using image segmentation techniques. However, natural phenomena, such as waves and currents, can also cause dark spots, resulting in consistently uneven intensity, high noise, and blurred boundaries in oil spill images. In addition, existing oil spill detection models often perform well for large targets but have poor detection accuracy for small targets. To solve the above problems, the oil spill contextual and boundary-supervised detection network (CBD-Net) is proposed to extract refined oil spill regions by fusing multiscale features. To improve the internal consistency of oil spill regions, the spatial and channel squeeze excitation (scSE) block is introduced. In CBD-Net, boundary details are enhanced with optimized edge supervision. In addition, a manually labeled dataset is proposed, Deep-SAR Oil Spill (SOS) dataset, aiming to solve the problem of insufficient existing oil spill detection dataset. Experimental results demonstrate that CBD-Net outperforms other comparative models and is able to extract robust and accurate oil spill regions from complex SAR images. The highest mIoU of 83.42% and the highest F1 score of 87.87% were achieved on the SOS dataset. The CBD-Net model proposed in this article can play a guiding role in the marine oil spill decision support system.

Convolutional Neural Networks (CNN) and Graph Neural Networks (GNN), such as Graph Attention Networks (GAT), are two classic neural network models, which are applied to the processing of grid data and graph data respectively. They have achieved outstanding performance in hyperspectral images (HSIs) classification field, which have attracted great interest. However, CNN has been facing the problem of small samples and GNN has to pay a huge computational cost, which restrict the performance of the two models. In this paper, we propose Weighted Feature Fusion of Convolutional Neural Network and Graph Attention Network (WFCG) for HSI classification, by using the characteristics of superpixel-based GAT and pixel-based CNN, which proved to be complementary. We first establish GAT with the help of superpixel-based encoder and decoder modules. Then we combined the attention mechanism to construct CNN. Finally, the features are weighted fusion with the characteristics of two neural network models. Rigorous experiments on three real-world HSI data sets show WFCG can fully explore the high-dimensional feature of HSI, and obtain competitive results compared to other state-of-the art methods.

High-precision road detection from very high resolution (VHR) remote sensing images has broad application value. However, the most advanced deep learning based methods often fail to identify roads when there is a distribution discrepancy between the training samples and test samples, due to their limited generalization ability. In this paper, to address this problem, an open-source data-driven domain-specific representation (OSM-DOER) framework is proposed for cross-domain road detection. On the one hand, as the spatial structure information of the source and target domains is similar, but the texture information is different, the domain-specific representation (DOER) framework is proposed, which not only aligns the distributions of the spatial structure information, but also learns the domain-specific texture information. Furthermore, in order to enhance the representation of the target domain data distribution, open-source and freely available OpenStreetMap (OSM) road centerline data are utilized to generate target domain samples, which are then used in the network training as the supervised information for the target domain. Finally, to verify the superiority of the proposed OSM-DOER framework, we conducted extensive experiments with the public SpaceNet and DeepGlobe road datasets, and large-scale road datasets from Birmingham in the UK and Shanghai in China. The experimental results demonstrate that the proposed OSM-DOER framework shows obvious advantages over the mainstream road detection methods, and the use of OSM road centerline data has great potential for the road detection task.

Model-driven methods and data-driven methods have been widely developed for hyperspectral image (HSI) denoising. However, there are pros and cons in both model-driven and data-driven methods. To address this issue, we develop a self-supervised HSI denoising method via integrating model-driven with data-driven strategy. The proposed framework simultaneously cooperates the spectral low-rankness prior and deep spatial prior (SLRP-DSP) for HSI self-supervised denoising. SLRP-DSP introduces the Tucker factorization via orthogonal basis and reduced factor, to capture the global spectral low-rankness prior in HSI. Besides, SLRP-DSP adopts a self-supervised way to learn the deep spatial prior. The proposed method doesn't need a large number of clean HSIs as the label samples. Through the self-supervised learning, SLRP-DSP can adaptively adjust the deep spatial prior from self-spatial information for reduced spatial factor denoising. An alternating iterative optimization framework is developed to exploit the internal low-rankness prior of third-order tensors and the spatial feature extraction capacity of convolutional neural network. Compared with both existing model-driven methods and data-driven methods, experimental results manifest that the proposed SLRP-DSP outperforms on mixed noise removal in different noisy HSIs.

Hyperspectral images (HSIs) are crucial for many research works. Spectral super-resolution (SSR) is a method used to obtain high-spatial-resolution (HR) HSIs from HR multispectral images. Traditional SSR methods include model-driven algorithms and deep learning. By unfolding a variational method, this article proposes an optimization-driven convolutional neural network (CNN) with a deep spatial-spectral prior, resulting in physically interpretable networks. Unlike the fully data-driven CNN, auxiliary spectral response function (SRF) is utilized to guide CNNs to group the bands with spectral relevance. In addition, the channel attention module (CAM) and the reformulated spectral angle mapper loss function are applied to achieve an effective reconstruction model. Finally, experiments on two types of data sets, including natural and remote sensing images, demonstrate the spectral enhancement effect of the proposed method, and also, the classification results on the remote sensing data set verified the validity of the information enhanced by the proposed method.

Buildings constitute one of the most important landscapes in remote sensing (RS) images and have been broadly analyzed in a wide range of applications from urban planning to other socioeconomic studies. As very-high-resolution (VHR) RS imagery becomes more accessible, the current building extraction methods are confronted with the challenges of the diverse appearances, various scales, and complicated structures of buildings in complex scenes. With the development of context-aware deep learning methods, it has been proven by numerous works that capturing contextual information can offer spatial relation cues for robust recognition and detection of the objects. In this article, we propose a novel local-global dual-stream network (DS-Net) that adaptively captures local and long-range information for the accurate mapping of building rooftops in VHR RS images. The local branch and the global branch of DS-Net work in a complementary manner to each other with different fields of view on the input image. Through a well-defined dual-stream architecture, DS-Net learns hierarchical representations for both the local and global branches, and a deep feature sharing strategy is further developed to enforce more collaborative integration of the two branches. Extensive experiments were carried out to verify the effectiveness of our model on three widely used VHR RS data sets: the Massachusetts buildings data set, the Inria Aerial Image Labeling data set, and the DeepGlobe Building Detection Challenge data set. Empirically, the proposed DS-Net achieves competitive or superior performance compared with the current state-of-the-art methods in terms of quantitative measures and visual evaluations.

For high spatial resolution (HSR) remote sensing images, bitemporal supervised learning always dominates change detection using many pairwise labeled bitemporal images. However, it is very expensive and time-consuming to pairwise label large-scale bitemporal HSR remote sensing images. In this paper, we propose single-temporal supervised learning (STAR) for change detection from a new perspective of exploiting object changes in unpaired images as supervisory signals. STAR enables us to train a high-accuracy change detector only using unpaired labeled images and generalize to real-world bitemporal images. To evaluate the effectiveness of STAR, we design a simple yet effective change detector called ChangeStar, which can reuse any deep semantic segmentation architecture by the ChangeMixin module. The comprehensive experimental results show that ChangeStar outperforms the baseline with a large margin under single-temporal super-vision and achieves superior performance under bitemporal supervision. Code is available at https://github.com/Z-Zheng/ChangeStar.

With the rapid advancement of remote sensing (RS) technology, RS image interpretation has made great progress and been widely used in broad applications, in which the constructed benchmark datasets for developing and testing intelligent interpretation algorithms have been playing an increasingly critical role. Motivated by the essential prerequisites of dataset in the development of RS image interpretation algorithms, this manuscript provides a discussion on dataset construction for RS image interpretation. Specifically, we first analyze the current challenges of developing algorithms for RS image interpretation and a review on the widespread RS image datasets is conducted through the bibliometric analysis. We then propose some principles and discuss the methodology on constructing benchmark datasets. An implementation on creating the RS scene classification dataset demonstrates the practicability of our proposed framework and the experimental results show that our constructed dataset can serve as a promising benchmark for RS image scene interpretation.

Recently, deep learning-based methods have shown the great potential in hyperspectral image (HSI) classification. Nevertheless, feature extraction by convolutional neural network (CNN) is often performed on only one scale, resulting in multi-scale information loss. To address this problem, in this paper, we propose an automatically adjustable multi-scale feature extraction framework (A2MFE-Framework) for hyperspectral classification, including a scale reference network and two scale transformation networks. With the well-designed architecture, A2MFE-Framework can not only extract multiscale features, but also automatically change the network structure to match input features of different scales. Experimental results on two benchmark HSI datasets demonstrate that the A2MFE-Framework can better capture multi-scale features of different objects via an automatically adjustable feature extraction framework with higher classification accuracy compared with previous methods.

Hyperspectral unmixing is an essential task in hyperspectral imagery applications. Because of the strong feature extract ability and satisfying performance, deep learning methods have been used for hyperspectral unmixing. However, there are still several problems in existing deep learning based spectral unmixing methods. Supervised learning methods can only accomplish a single task and lack a large amount of data for supervised learning. While the unsupervised learning unmixing methods are easily misled by the traditional way of initialization. In this paper, a weakly supervised deep convolutional neural network is proposed for hyperspectral unmixing. The experimental results show that competitive results can also be obtained by pretraining with a small number of samples, and weakly supervised learning still has potential for hyperspectral unmixing.

Recently, hyperspectral anomaly detection methods based on low-rank representation(LRR) have been widely studied. However, the assumption of global low dimension of background may ignore the local structure information of hyperspectral image. In this paper, a novel LRR incorporating local spatial constraint method is proposed for hyperspectral anomaly detection. Different from LRR detector, the proposed method considers the spatial information based on the supe pixel in the background part. The proposed method and current state-of-the-art methods are tested on two sets of real data. The experimental results demonstrate that the proposed method is superior to the comparative method in terms of both colour map detection and quantitative evaluation.

Change detection is a geospatial application for social good whose development is restricted by a slow-growing labeling technology and outdated classification labels for remotely sensed images. In this paper, a change cross-detection method based on label improvements and multi-model fusion is proposed for Multi-temporal Semantic change Detection (MSD) with weak, noisy, and low-resolution labels. For unmatched labels, a Siamese Skip_FCN network is proposed to generate preliminary labels at high-resolution. Subsequently, a multi-model fusion method is introduced to perform accurate and stable land cover classification. In addition, a cross-detection structure is used to generate high precision change maps and a post-processing step further improves the final results. In the track MSD of the 2021 Data Fusion Contest (DFC21-MSD), the proposed method achieved a mean intersection over union (mIoU) of 70.25% in phase 1 and 67.72% in phase 2, ranking first in both phases [1].

Drought, leading to water supply deficits and vegetation growth anomalies, is one of the most serious but least understood natural phenomenon. Different vegetation types vary in sensitivity to drought and exhibit diverse kinds of responses. However, few studies have focused on the sensitivity and the response of various types of vegetation. Thus, in this paper, a new Drought Index based on land cover Types Response Adjustment (TRADI) is proposed based on response adjustment for vegetation, to gauge the response factors among various vegetation types and provide a comprehensive assessment of drought conditions. Multisource data from the Moderate Resolution Imaging Spectroradiometer (MODIS), the Tropical Rainfall Measuring Mission (TRMM) and the Global Land Data Assimilation System (GLDAS) was used to derive drought condition indices (CIs) and construct the TRADI. A correlation analysis between TRADI and Palmer Drought Severity Index (PDSI) monitoring results for typical drought events in the CONtinental United States (CONUS) was carried out to evaluate the reliability and efficiency of the proposed method. These results showed that drought was underestimated or overestimated by varying degrees when using traditional condition-based indices (CIs), but the TRADI can correct these misestimates and improve the drought monitoring accuracy.

In this study, we combine both the deep prior with low-rank tensor completion (DP-LRTC) for thick cloud removal in Sentinel-2 time-series images. On the one hand, DP-LRTC utilizes the low-rank property of multitemporal images via 3-order tensor completion. On the other hand, DP-LRTC employs the 3D spatiotemporal feature expression ability by deep learning. Through integrating both model-driven with data-driven strategy, the proposed method can effectively removal thick cloud in Sentinel-2 time-series images.

The limited training sample has become a great challenge for hyperspectral target detection with deep learning-based methods. In this paper, a long short-term memory based hyperspectral target detector is proposed. To handle the insufficient background training samples, an endmember extraction based pixel selection strategy is proposed to select background pixels from the entire image. For the target training samples, we utilize a synthesis method to generate sufficient target samples using the given target spectrum and the extracted background samples. Then the obtained target and background samples are fed into the well-designed long short-term memory network to learn the discriminative ability. Finally, the detected pixels are classified by the well-trained LSTM network and the detection results are achieved. The experiments on the Muufl and Nuance data sets demonstrate the superiority of the proposed LSTMTD in target detection.

In this paper, an encoder-decoder long short-term memory network-based anomaly detector (denoted as EDLAD) is proposed for hyperspectral images. The proposed EDLAD aims to simultaneously alleviate anomaly contamination and build a stable background component for anomaly detection. To reduce anomaly contamination, the EDLAD first utilizes a well-designed encoder-decoder LSTM to reconstruct the hyperspectral image. Based on the concept that the anomaly pixels occupy an extremely small fraction of the image, the well-designed encoder-decoder LSTM network tends to maintain the background and alleviate anomaly during the reconstruction process since the whole image is employed for training the network. Then the dimension reduction is used to further alleviate the anomaly contamination and build a stable background component. Finally, the EDLAD applies the Mahalanobis distance differences to detect the probable anomalies. The experiments on two benchmark hyperspectral images demonstrate the superiority of the EDLAD in anomaly detection.

Domain adaptation is widely used in the field of remote sensing, which can transfer the existing knowledge to new tasks and promote performance. When applied in the field of scene classification, it can be called cross-scene classification. Previous cross-scene classification methods mainly consider the coarse-grained alignment in the global aspect, which may ignore the structures behind the data and lose the local information with respect to specific categories. To implement fine-grained alignment, we present an adversarial fine-grained adaptation network (AFGAN) which simultaneously captures the complex structures behind the data distributions to improve the discriminability and reduce the local discrepancy of different domains to align the relevant category distributions. Experimental results based on three existing scene classification datasets demonstrate the effectiveness of AFGAN.

This paper presents regional attraction of line segment maps, and hereby poses the problem of line segment detection (LSD) as a problem of region coloring. Given a line segment map, the proposed regional attraction first establishes the relationship between line segments and regions in the image lattice. Based on this, the line segment map is equivalently transformed to an attraction field map (AFM), which can be remapped to a set of line segments without loss of information. Accordingly, we develop an end-to-end framework to learn attraction field maps for raw input images, followed by a squeeze module to detect line segments. Apart from existing works, the proposed detector properly handles the local ambiguity and does not rely on the accurate identification of edge pixels. Comprehensive experiments on the Wireframe dataset and the YorkUrban dataset demonstrate the superiority of our method. In particular, we achieve an F-measure of 0.831 on the Wireframe dataset, advancing the state-of-the-art performance by 10.3 percent.

The application of convolutional neural networks has been shown to significantly improve the accuracy of building extraction from very high-resolution (VHR) remote sensing images. However, there exist so-called semantic gaps among different kinds of buildings due to the large intraclass variance of buildings, and most of the present-day methods are ineffective in extracting various buildings in large areas that cover different scenes, for example, urban villages and high-rise buildings, because existing building extraction strategies are the same for various scenes. With the improvement of the resolution of remote sensing images, it is feasible to improve the image interpretation based on the scene prior. However, this idea has not been fully utilized in building extraction from VHR remote sensing imagery. This study proposes a scene-driven multitask parallel attention convolutional network (MTPA-Net) to resolve these limitations. The proposed approach classifies the input image into multilabel scenes and further separately maps the buildings in pixel level under different scenes. In addition, a simple postprocessing method is applied to integrate the building extraction results and scene prior. Our proposed method does not require multimodel training and the network can learn in an end-to-end manner. The performance of our proposed method is evaluated on a data set that includes various urban and rural scenes with diverse landscapes. The experimental results show that the proposed MTPA-Net outperforms state-of-the-art algorithms by reducing misclassification areas and maintaining improved robustness.

Presents corrections to the above named paper.

Subpixel mapping (SPM) is an effective way to solve the mixed pixel problem, which is a ubiquitous phenomenon in remotely sensed imagery, by characterizing subpixel distribution within the mixed pixels. In fact, the majority of the classical and state-of-the-art SPM algorithms can be viewed as a convolution process, but these methods rely heavily on fixed and handcrafted kernels that are insufficient in characterizing a geographically realistic distribution image. In addition, the traditional SPM approach is based on the prerequisite of abundance images derived from spectral unmixing (SU), during which process uncertainty inherently exists and is propagated to the SPM. In this article, a kernel-learnable convolutional neural network (CNN) framework for subpixel mapping (SPMCNN-F) is proposed. In SPMCNN-F, the kernel is learnable during the training stage based on the given training sample pairs of low- and high-resolution patches for learning a geographically realistic prior, instead of fixed priors. The end-to-end mapping structure enables direct subpixel information extraction from the original coarse image, avoiding the uncertainty propagation from the SU. In the experiments undertaken in this study, two state-of-the-art super-resolution networks were selected as application demonstrations of the proposed SPMCNN-F method. In experiment part, three hyperspectral image data sets were adopted, two in a synthetic coarse image approach and one in a real coarse image approach, for the validation. Additionally, a new data set with pairs of Moderate-resolution Imaging Spectroradiometer (MODIS) and Landsat images were adopted in a real coarse image approach, for further validation of SPMCNN-F in large-scale area. The restored fine distribution images obtained in all the experiments showed a perceptually better reconstruction quality, both qualitatively and quantitatively, confirming the superiority of the proposed SPM framework.

Sparse unmixing, as a semisupervised unmixing method, has attracted extensive attention. The process of sparse unmixing involves treating the mixed pixels of hyperspectral imagery as a linear combination of a small number of spectral signatures (endmembers) in a standard spectral library, associated with fractional abundances. Over the past ten years, to achieve a better performance, sparse unmixing algorithms have begun to focus on the spatial information of hyperspectral images. However, less accurate spatial information greatly limits the performance of the spatial-regularization-based sparse unmixing algorithms. In this article, to overcome this limitation and obtain more reliable spatial information, a novel sparse unmixing algorithm named superpixel-based reweighted low-rank and total variation (SUSRLR-TV) is proposed to enhance the performance of the traditional spatial-regularization-based sparse unmixing approaches. In the proposed approach, superpixel segmentation is adopted to consider both the spatial proximity and the spectral similarity. In addition, a low-rank constraint is enforced on the objective function as pixels within each superpixel have the same endmembers and similar abundance values, and they naturally satisfy the low-rank constraint. Differing from the traditional nuclear norm, a reweighted nuclear norm is used to achieve a more efficient and accurate low-rank constraint. Meanwhile, low-rank consideration is also used to enhance the spatial continuity and suppress the effects of random noise. Furthermore, TV regularization is introduced to promote the smoothness of the abundance maps. Experiments on three simulated data sets, as well as a well-known real hyperspectral imagery data set, confirm the superior performance of the proposed method in both the qualitative assessment and the quantitative evaluation, compared with the state-of-the-art sparse unmixing methods.

Current hyperspectral image classification assumes that a predefined classification system is closed and complete, and there are no unknown or novel classes in the unseen data. However, this assumption may be too strict for the real world. Often, novel classes are overlooked when the classification system is constructed. The closed nature forces a model to assign a label given a new sample and may lead to overestimation of known land covers (e.g., crop area). To tackle this issue, we propose a multitask deep learning method that simultaneously conducts classification and reconstruction in the open world (named MDL4OW) where unknown classes may exist. The reconstructed data are compared with the original data; those failing to be reconstructed are considered unknown based on the assumption that they are not well represented in the latent features due to the lack of labels. A threshold needs to be defined to separate the unknown and known classes; we propose two strategies based on the extreme value theory for few- and many-shot scenarios. The proposed method was tested on real-world hyperspectral images; state-of-the-art results were achieved, e.g., improving the overall accuracy by 4.94% for the Salinas data. By considering the existence of unknown classes in the open world, our method achieved more accurate hyperspectral image classification, especially under the few-shot context.

Hyperspectral image (HSI) denoising plays an important role in image quality improvement and related applications. Convolutional neural network (CNN)-based image denoising methods have been predominant due to advances made in the field of deep learning in recent years. Spatial and spectral information are crucial to HIS denoising, along with their correlations. However, existing methods fail to consider the global dependence and correlation between spatial and spectral information. Accordingly, in this article, we propose a novel dual-attention denoising network to overcome these limitations. We design two parallel branches to process the spatial and spectral information separately. The position attention module is applied to the spatial branch to formulate the interdependencies on the feature map, while the channel attention module is applied to the spectral branch to simulate the spectral correlation before the two branches are combined. A multiscale structure is also employed to extract and fuse the multiscale features following the fusion of spatial and spectral information. Experimental results on simulated and real data substantiate the superiority of our method both visually and quantitatively when compared with state-of-the-art methods.

The Chinese Gaofen-3 satellite currently provides us with an open way to access fully polarimetric data in the C-band frequency. The noise equivalent sigma zero (NESZ) is a crucial factor when calibrating the additive noise in radar imagery. For most radar sensors, NESZ coefficients are stored in header files, but these are not provided for the Gaofen-3 products. The minimum eigenvalue estimator (MEE) and maximum likelihood estimator (MLE) are the two most common techniques used to derive the NESZ from polarimetric imagery. Nevertheless, the bias has been found to be higher than 5 dB compared with the noise measurement circuit (NMC) of the hardware. In this article, we propose a minimum noise envelope estimator (MNEE) for the robust estimation of the Gaofen-3 NESZ. In this article, we carried out an in-depth investigation to analyze the error sources of the MEE and MLE techniques. Based on our analysis, the MNEE framework requires the use of the ocean surface as a reference, and MNEE is combined with the minimum operation to suppress overestimation. In the experimental section, we describe how we validated the proposed algorithm with Radarsat-2 images, and the MNEE is treated as a tool to estimate the NESZ of Gaofen-3 polarimetric products. We found that the Gaofen-3 NESZ is generally less than -20 dB, which satisfies the design specification. The range-dependent NESZ coefficients are provided here to allow convenient noise correction for Gaofen-3 data users.

Endmember extraction (EE) plays a crucial part in the hyperspectral unmixing (HU) process. To obtain satisfactory EE results, the EE can be considered as the multiobjective optimization problem to optimize the volume maximization (VM) and root-mean-square error (RMSE) simultaneously. However, it is often quite challenging to balance the conflict of these objectives. In order to tackle the challenges of multiobjective EE, we present a (μ + λ) multiobjective differential evolution algorithm ((μ + λ)-MODE) based on ranking multiple mutations. In the (μ + λ)-MODE algorithm, ranking multiple mutations are adopted to create the mutant vectors via the scaling factor pool to enhance the population diversity. Moreover, mutant vectors employ the binary crossover operator to generate the trial vectors through a crossover control parameter pool in (μ + λ)-MODE to take advantage of the good information of the population. In addition, (μ + λ)-MODE utilizes the fast nondominated sorting approach to sort the parent and trial vectors, and then selects the elitism offspring as the next population via the (μ + λ) selection strategy. Eventually, experimental comparative results in three real HSIs reveal that our proposed (μ + λ)-MODE is superior to other EE methods.

In this article, we propose an end-to-end adaptive spectral-spatial multiscale network to extract multiscale contextual information for hyperspectral image (HSI) classification, which contains spectral feature extraction (FE) and spatial FE subnetworks. In spectral FE aspect, different from previous methods where features are obtained in a single scale, which limits the accuracy improvement, we propose two schemes based on band grouping strategy, and the long short-time memory (LSTM) model is used for perceiving spectral multiscale information. In spatial subnetwork, on the foundation of existing multiscale architecture, the spatial contextual features which are usually ignored by previous literature are successfully obtained under the aid of convolutional LSTM (ConvLSTM) model. Besides, a new spatial grouping strategy is proposed for convenience of ConvLSTM to extract the more discriminative features. Then, a novel adaptive feature combining way is proposed considering the different importance of spectral and spatial parts. Experiments on three public data sets in HSI community demonstrate that our methods achieve competitive results compared with other state-of-the-art methods.

Determining the optimal number of endmember sources, which is also called “virtual dimensionality” (VD), is a priority for hyperspectral unmixing (HU). Although the VD estimation directly affects the HU results, it is usually solved independently of the HU process. In this article, a saliency-based autonomous endmember detection (SAED) algorithm is proposed to jointly estimate the VD in the process of endmember extraction (EE). In SAED, we first demonstrate that the abundance anomaly (AA) value is an important feature of undetected endmembers since pure pixels have larger AA values than “distractors” (i.e., mixed pixels and pure pixels of detected endmembers). Then, motivated by the fact that endmembers usually gather in certain local regions (superpixels) in the scene, due to spatial correlation, a superpixel prior is introduced in SAED to distinguish endmembers from noise. Specifically, the undetected endmembers are defined as visual stimuli in the AA subspace, the EE is formulated as a salient region detection problem, and the VD is automatically determined when there are no salient objects in the AA subspace. Since the spatial-contextual information of the endmembers is exploited during the saliency analysis, the proposed method is more robust than the spectral-only methods, which was verified using both real and synthetic hyperspectral images.

Deep learning algorithms, especially convolutional neural networks (CNNs), have recently emerged as a dominant paradigm for high spatial resolution remote sensing (HRS) image recognition. A large amount of CNNs have already been successfully applied to various HRS recognition tasks, such as land-cover classification and scene classification. However, they are often modifications of the existing CNNs derived from natural image processing, in which the network architecture is inherited without consideration of the complexity and specificity of HRS images. In this article, the remote sensing deep neural network (RSNet) framework is proposed using an automatically search strategy to find the appropriate network architecture for HRS image recognition tasks. In RSNet, the hierarchical search space is first designed to include module- and transition-level spaces. The module-level space defines the basic structure block, where a series of lightweight operations as candidates, including depthwise separable convolutions, is proposed to ensure the efficiency. The transition-level space controls the spatial resolution transformations of the features. In the hierarchical search space, a gradient-based search strategy is used to find the appropriate architecture. In RSNet, the task-driven architecture training process can acquire the optimal model parameters of the switchable recognition module for HRS image recognition tasks. The experimental results obtained using four benchmark data sets for land-cover classification and scene classification tasks demonstrate that the searched RSNet can achieve a satisfactory accuracy with a high computational efficiency and, hence, provides an effective option for the processing of HRS imagery.

Deep neural networks, which can learn the representative and discriminative features from data in a hierarchical manner, have achieved state-of-the-art performance in the remote sensing scene classification task. Despite the great success that deep learning algorithms have obtained, their vulnerability toward adversarial examples deserves our special attention. In this article, we systematically analyze the threat of adversarial examples on deep neural networks for remote sensing scene classification. Both targeted and untargeted attacks are performed to generate subtle adversarial perturbations, which are imperceptible to a human observer but may easily fool the deep learning models. Simply adding these perturbations to the original high-resolution remote sensing (HRRS) images, adversarial examples can be generated, and there are only slight differences between the adversarial examples and the original ones. An intriguing discovery in our study shows that most of these adversarial examples may be misclassified into the wrong category by the state-of-the-art deep neural networks with very high confidence. This phenomenon, undoubtedly, may limit the practical deployment of these deep learning models in the safety-critical remote sensing field. To address this problem, the adversarial training strategy is further investigated in this article, which significantly increases the resistibility of deep models toward adversarial examples. Extensive experiments on three benchmark HRRS image data sets demonstrate that while most of the well-known deep neural networks are sensitive to adversarial perturbations, the adversarial training strategy helps to alleviate their vulnerability toward adversarial examples.

Supervised hyperspectral image (HSI) classification has been widely studied and used in many different applications. However, the performance of the supervised classifiers, including the traditional machine learning methods and the deep neural networks, is significantly affected by the inaccurate labeling of training samples, which is a common problem in HSI supervised classification. In this article, we propose a superpixel guided sample selection neural network (S3Net) framework with end-to-end training for handling noisy labels in HSI classification. It includes two stages: sample selection and sample correction. In sample selection, a sample with a small training loss has a higher probability of being the correct label and hence selected from the noisy labels for model training. In order to avoid the error propagation caused by the noisy labels, we utilize a cross-selection update strategy that exchanges selected samples between two neural networks during conventional loss backpropagation. Sample selection is a pruning process, which may cause insufficient training sample problem in HSI classification. To solve this problem, we propose the sample correction strategy to correct the noisy labels by propagating clean label information in the homogeneous regions obtained by superpixel. Experimental results on three public HSI data sets demonstrate the effectiveness of the proposed S3Net framework when handling noisy labels.

Deep learning-based hyperspectral image (HSI) classification methods have recently attracted significant attention. However, features captured by convolutional neural network (CNN) are always partial due to the restrictions of the respective fields and the loss of multiscale information, which lead to features being discontinuous when extracted. In a departure from existing approaches, in this article, we propose a novel Enhanced Multiscale Feature Fusion Network (EMFFN). As a deeper and wider network, EMFFN can extract sufficiently multiscale features from the parallel multipath of three stages for HSI classification purposes. There are two subnetworks for multiscale spectral and spatial information in EMFFN, respectively. First, we propose a spectral Cascaded Dilated Convolutional Network (CDCN) designed to obtain a larger respective field for long-ranged information and extract multiscale features. Subsequently, a Parallel Multipath Network (PMN) is proposed to capture large-scale, middle-scale, and small-scale spatial features in parallel during all three stages. In the next step, hierarchical features are fused successively, and shallower feature maps can achieve better learning performance when guided by deeper semantic information. As PMN deepens in different stages, more multiscale information flows into the network, enabling finer classification results. To incorporate abundant spectral and spatial features, moreover, we combine features collected from two subnetworks into EMFFN using the designed consolidated loss function. As a result, the network facilitates the learning of not only localization-preserved features, but also high-level semantic features. In our experiments, three benchmark HSIs are utilized to evaluate the performance of the proposed method. Our results demonstrate that the proposed EMFFN can outperform state-of-the-art methods.

Clustering is a common method for hyperspectral image (HSI) interpretation in the case of no labeled samples. Many subspace clustering methods have now been proposed for HSIs and have obtained remarkable success. However, because of the prohibitively large computational complexity induced by the self-dictionary representation, these methods suffer from the scalability issue and are ineffective for large HSIs. In this article, to address this issue, we focus on a scalable subspace clustering scheme and introduce the recently developed sketched subspace clustering (sketched-SC) model to HSI. The sketched-SC model is computationally inexpensive and is suitable for the large HSI clustering task as it constructs a compact yet expressive dictionary. However, several problems degrade the performance of sketched-SC, i.e., the inadequate mining of the structural information and no consideration of spatial information. In view of this, a novel scalable nonlocal means regularized sketched reweighted sparse and low-rank (NL-SSLR) SC algorithm is proposed for use with large HSIs. On the one hand, the SSLR representation model is constructed to explore the underlying local and global structural information of the HSIs at the same time. On the other hand, the nonlocal means regularization is used to fully explore the spatial correlation information and better account for the self-similarity of HSIs, to further boost the clustering performance. The experimental results obtained on two well-known hyperspectral data sets corroborate the superiority of the proposed algorithm over the other state-of-the-art HSI clustering methods.

Hyperspectral image (HSI) clustering is extremely challenging because of the complexity of the image structure. Recently, the subspace clustering algorithms have achieved competitive performance for HSIs. However, these methods generally are computationally complex and time-and-memory-consuming, given their reliance on large-scale adjacency matrix learning and graph segmentation, which limits their application to large HSIs and reduces their attractiveness in real applications. In this article, in view of this, two novel sparsity-based clustering algorithms are proposed for large HSIs, named sparse coding-based clustering (SCC) and joint SCC (JSCC). To the best of our knowledge, we are the first to use the sparse representation recovery residual to cluster HSIs. Based on a structured dictionary constructed by$k$-means and$k$-nearest neighbor (KNN), an SCC model is constructed to cluster HSIs according to the recovery residual minimization criterion. By dealing with a pixel-wise sparse recovery problem instead of the large-scale graph optimization problem of the whole image, the computational complexity and the time-and-memory cost are reduced to a large degree, which makes sense for practical applications. Then, by introducing the super-pixel neighborhood, a JSCC model is constructed to better explore the interpixel correlation of HSIs and further improve the clustering performance. The proposed algorithms were verified on three widely used HSIs. All the three experiments confirm the effectiveness of the proposed algorithms, which can be considered as competitive tools for use with large HSIs.

Due to technological limitations and budget constraints, spatiotemporal fusion is considered a promising way to deal with the tradeoff between the temporal and spatial resolutions of remote sensing images. Furthermore, the generative adversarial network (GAN) has shown its capability in a variety of applications. This article presents a remote sensing image spatiotemporal fusion method using a GAN (STFGAN), which adopts a two-stage framework with an end-to-end image fusion GAN (IFGAN) for each stage. The IFGAN contains a generator and a discriminator in competition with each other under the guidance of the optimization function. Considering the huge spatial resolution gap between the high-spatial, low-temporal (HSLT) resolution Landsat imagery and the corresponding low-spatial, high-temporal (LSHT) resolution MODIS imagery, a feature-level fusion strategy is adopted. Specifically, for the generator, we first super-resolve the MODIS images while also extracting the high-frequency features of the Landsat images. Finally, we integrate the features from the MODIS and Landsat images. STFGAN is able to learn an end-to-end mapping between the Landsat-MODIS image pairs and predicts the Landsat-like image for a prediction date by considering all the bands. STFGAN significantly improves the accuracy of phenological change and land-cover-type change prediction with the help of residual blocks and two prior Landsat-MODIS image pairs. To examine the performance of the proposed STFGAN method, experiments were conducted on three representative Landsat-MODIS data sets. The results clearly illustrate the effectiveness of the proposed method.

In this article, a single-spectrum-driven binary-class sparse representation target detector (SSBSTD) via target and background dictionary construction (BDC) is proposed. The SSBSTD leans upon the binary-class sparse representation (BSR) model. Due to the fact that a background spectrum usually consists in background samples composed low-dimensional subspace and a target spectrum also consists in target samples composed low-dimensional subspace, only background samples should be used for sparsely representing the test pixel under the target absent hypothesis and the samples from target-only dictionary for target present hypothesis. To alleviate the problem that there are insufficient available target samples in the sparse representation model, this article proposed a predetection method to construct the target dictionary utilizing the given target spectrum. With regard to the BDC, we proposed an approach based on the classification to generate a global over-complete background dictionary. The detection output is composed of the residual difference between the BSR. Extensive experiments were made on four benchmark hyperspectral images and the experimental results indicate that our SSBSTD algorithm demonstrates superior detection performances.

In this article, a two-stream convolutional network-based target detector (denoted as TSCNTD) for hyperspectral images is proposed. The TSCNTD utilizes the two-stream convolutional networks to extract abundant spectral information in hyperspectral images. For the background samples, the TSCNTD finds enough typical background pixels via a hybrid sparse representation and classification-based pixel selection strategy in the entire image. To tackle the problem under limited target samples, a novel synthesis method is proposed to generate sufficient target samples with a target priori and some typical background pixels. Once the target and background samples are obtained, then the designed two-stream convolutional networks were trained with a target priori, target samples, and background samples. During training, a target priori and a target sample, which construct a positive training sample, are considered as two inputs of the two-stream convolutional networks, while a target priori and a background sample construct a negative training sample. During testing, the test samples, which are constructed by a target priori and the detected pixels, are classified by the well-trained network. The outputs of the network constitute the final detection result of the TSCNTD. Extensive experiments were made on four benchmark hyperspectral images. The experimental results indicate that the TSCNTD can achieve superior performances in target detection.

Deep learning models have shown their great capability for the hyperspectral image (HSI) classification task in recent years. Nevertheless, their vulnerability towards adversarial attacks could not be neglected. In this study, we systematically analyze the influence of adversarial attacks on the HSI classification task for the first time. While existing research of adversarial attacks focuses on the generation of adversarial examples in the RGB domain, the experiments in this study show such adversarial examples could also exist in the hyperspectral domain. Although the difference between the generated adversarial image and the original hyperspectral data is imperceptible to the human visual system, most of the existing state-of-the-art deep learning models could be fooled by the adversarial image to make wrong predictions. To address this challenge, a novel self-attention context network (SACNet) is further proposed. We discover that the global context information contained in HSI can significantly improve the robustness of deep neural networks when confronted with adversarial attacks. Extensive experiments on three benchmark HSI datasets demonstrate that the proposed SACNet possesses stronger resistibility towards adversarial examples compared with the existing state-of-the-art deep learning models.

Due to the physical limitations of the imaging devices, hyperspectral images (HSIs) are commonly distorted by a mixture of Gaussian noise, impulse noise, stripes, and dead lines, leading to the decline in the performance of unmixing, classification, and other subsequent applications. In this paper, we propose a novel end-to-end low-rank spatial-spectral network (LR-Net) for the removal of the hybrid noise in HSIs. By integrating the low-rank physical property into a deep convolutional neural network (DCNN), the proposed LR-Net simultaneously enjoys the strong feature representation ability from DCNN and the implicit physical constraint of clean HSIs. Firstly, spatial-spectral atrous blocks (SSABs) are built to exploit spatial-spectral features of HSIs. Secondly, these spatial-spectral features are forwarded to a multi-atrous block (MAB) to aggregate the context in different receptive fields. Thirdly, the contextual features and spatial-spectral features from different levels are concatenated before being fed into a plug-and-play low-rank module (LRM) for feature reconstruction. With the help of the LRM, the workflow of low-rank matrix reconstruction can be streamlined in a differentiable manner. Finally, the low-rank features are utilized to capture the latent semantic relationships of the HSIs to recover clean HSIs. Extensive experiments on both simulated and real-world datasets were conducted. The experimental results show that the LR-Net outperforms other state-of-the-art denoising methods in terms of evaluation metrics and visual assessments. Particularly, through the collaborative integration of DCNNs and the low-rank property, the LR-Net shows strong stability and capacity for generalization.

This paper presents a fast and parsimonious parsing method to accurately and robustly detect a vectorized wireframe in an input image with a single forward pass. The proposed method is end-to-end trainable, consisting of three components: (i) line segment and junction proposal generation, (ii) line segment and junction matching, and (iii) line segment and junction verification. For computing line segment proposals, a novel exact dual representation is proposed which exploits a parsimonious geometric reparameterization for line segments and forms a holistic 4-dimensional attraction field map for an input image. Junctions can be treated as the “basins” in the attraction field. The proposed method is thus called Holistically-Attracted Wireframe Parser (HAWP). In experiments, the proposed method is tested on two benchmarks, the Wireframe dataset [14] and the YorkUrban dataset [8]. On both benchmarks, it obtains state-of-the-art performance in terms of accuracy and efficiency. For example, on the Wireframe dataset, compared to the previous state-of-the-art method L-CNN [36], it improves the challenging mean structural average precision (msAP) by a large margin (2.8% absolute improvements), and achieves 29.5 FPS on a single GPU (89% relative improvement). A systematic ablation study is performed to further justify the proposed method.

Hyperspectral unmixing is an essential task in hyperspectral imagery applications. Deep learning methods have been taken into hyperspectral unmixing because of its great feature extraction ability and better performance. However, there are several problems in existing deep learning based spectral unmixing methods. The networks are not deep enough to exploit their feature extraction capabilities in these unsupervised autoencoders based methods, and their effects are not stable. The main reason may be the limited prior information limited the ability of conducting the supervised method. In this manuscript, a semi-supervised deep learning based unmixing method is proposed. Unlike the existing methods, our model uses deeper neural networks without pooling layers, and the endmember spectrum are selected supervised from the original data, which uses nature and nurture cooperatively. The experimental results show that the proposed method achieves better performance and produces more accurate abundance maps, as well as higher quantitative results, compared with the current state-of-the-art deep learning unmixing algorithms.

Remote sensing scene change detection (SCD) is to detect whether and what changes have occurred in the semantic category of corresponding scenes for a long time at the semantic level. This can provide detailed land use/land cover change information for Urban planning and environmental monitoring. Previous studies take regular patches divided by uniform grid sampling as scene units. This may lead to mosaic phenomenon, and use fixed Window to extract features, ignoring the multi-scale features of ground objects, while extracting scene features. To solve the problems, the multi-scale irregular bag of visual features (MIBVF) framework is proposed for high spatial resolution (HSR) imagery SCD. In this paper, we integrate image classification of the physical characteristics from remote sensing data with the socio-economic attributes from open source geographic data. Road network data is used to preserve the geological significance and semantic integrity of urban scenes, and multi-scale window sampling is used to solve the problem of different object sizes. To confirm the feasibility of the proposed method, experiments with multi-temporal images of the Pudong area in Shanghai indicate that the proposed method achieves a clearly higher change detection accuracy than current state-of-the-art methods.

Buildings constitute one of the most important landscapes in remote sensing images. Automatic building extraction methods towards the very high resolution remote sensing imagery feature both the local refinement of segmentation results and the context-aware reasoning for segmentation. In this paper, we propose a novel dual-stream convolutional neural network (DS-Net) to collaboratively incorporate local and global features for accurately segmenting buildings in very high resolution aerial images. We develop a hierachical representation and a deep feature sharing strategy for both the local branch and global branch in DS-Net to effectively exploit the complementarity between the two branches. Through extensive experiments on the large-scale building detection datasets, we show that the proposed DS-Net can benefit from both the local and global features, which significantly improves the accuracy of building extraction over diversified remote sensing scenes.

Spatio-temporal fusion is a promising way to deal with the tradeoff between the temporal resolution and spatial resolution of the remote sensing images. This paper presents a novel remote sensing image spatio-temporal fusion model to expand the application of spatio-temporal fusion with insufficient data, based on a generative adversarial network to handle one prior image pair cases (OPGAN). Considering the huge spatial resolution gap between the high-spatial, low-temporal (HSLT) resolution Landsat imagery and the corresponding low-spatial, high-temporal (LSHT) resolution MODIS imagery, the proposed OPGAN simultaneously trains a generator and a discriminator in a min-max game to reconstruct the high-spatial-high-temporal (HSHT) resolution Landsat images, significantly improving the accuracy of change prediction with the help of the temporal changes and sensor differences. Experimental results on three representative Landsat-MODIS datasets illustrate the effectiveness of the proposed OPGAN method.

Tropospheric Monitoring Instrument (TROPOMI) carried onboard the Sentinel-5 Precursor (S5P) satellite could provide a daily global product of carbon monoxide (CO) total column at a high spatial resolution of 7*7 km2. However, data gaps exist in the TROPOMI CO product, which likely blocks the researches on the atmospheric CO. Therefore, a fusing method based on XGBoost is purposed in this study to recover the missing data in the TROPOMI CO product. The results show that the R and RMSE for the sample-based cross validation are 0.952 and 0.00303, respectively. It's believed that the proposed method could accurately predict the TROPOMI CO total column. Meanwhile, the spatial patterns of the daily predicted CO total column are similar to those of the TROPOMI. For temporal variations, the proposed method is able to mitigate the rapid fluctuations of the region-averaged TROPOMI CO total column in time series.

In order to improve the accuracy of surface classification and meet the needs of sub-pixel-level target detection, spectral unmixing has been one of the hot spots in hyperspectral remote sensing research. The employment of the probabilistic topic model to acquire latent topics of hyperspectral image has been an effective way for spectral unmixing. However, this approach fails to consider the sparsity of the semantic representation and high computational complexity. In addition, the number of endmembers cannot be determined automatically. To solve the problem, in this paper, the novel spectral unmixing method based on semi-automatic fully sparse semantic modeling framework (SFSSM) is proposed. In SFSSM, modestly few arithmetic operations are required to identify the pure spectral signatures (endmembers) and the fractional abundances of the endmembers. Meanwhile, the sparsity and representativeness of the topics generated by SFSSM guarantee that the endmembers can be obtained automatically in low time consumption. The experimental results obtained with two real image confirm that the proposed method significantly improves the performance when compared with the other methods.

In this paper, we present a novel hyperspectral image (HSI) denoising method especially for mixed noise removal. The proposed method combines both data-driven with model-driven strategy via a deep spatio-spectral variational structure. The mixed noise estimation and removal are collaboratively derived through fusing the Bayesian spatio-spectral posterior and deep learning model. The framework can both utilize the logicality of traditional model-driven methods, and the high efficiency of data-driven methods for parameters optimizing. Simulated and actual experiments demonstrate that the presented method outperforms other existing methods for HSI mixed noise removal, on both reconstructing effects and time-consuming.

Road extraction, which aims to label remote images with a specific road detection, is a fundamental task for understanding remote sensing imagery. Deep learning has strong characteristic learning ability, for example, the state-of-the-art D-Linknet is an effective way to capture the road information. However, existing regularization methods either do not match the performance for large batches, or still exhibit degradation in performance for smaller batches. Besides, the roads in different areas have various characteristics and lack a good transfer. To remedy these issues, we proposed a novel road extraction network which integrated the filter response normalization (FRN) layer with D-Linknet (FND-Linknet). The FRN layer is effective and robust for road extraction task, and can eliminate the dependency on other batch samples. In addition, the multisource road dataset is collected and annotated to improve features transfer. Experimental results on three datasets verify that the proposed FND-Linknet framework outperforms the state-of-the-art methods both in accuracy and connectivity.

The lunar hyperspectral remote sensing is one of the most important means to understand the physical properties and chemical constituents of lunar surface materials. Moon Mineral Mapper (M3) is currently the only hyperspectral image (HSI) data of Moon. However, due to the limitations of sensor manufacture and the impact of complex extraterrestrial environment, there are serious stripes on the M3 images, which do harm to subsequent identifications and analysis. In this paper, an effective destriping algorithm for lunar HSIs based on the intrinsic characteristics of the stripes on M3 is proposed. Experimental results demonstrate that our method shows an improvement in terms of visual perception and spectral fidelity.

High spatial resolution remote sensing image (RSI) scene classification, aimed at automatically labelling images with the given semantic categories, has been a hot issue. As it's difficult for RSI to quickly obtain a large number of training samples from a specific area. Traditional scene classification researches were mainly using deep learning models to transfer natural images to RSI. Considering the differences between natural images and RSI, we trained several Super Resolution GAN models by using different resolution RSI data from Google earth image. This paper proposed a novel SRGAN-CNN framework. Through transferring the data with scene classification dataset to obtain high resolution fake RSI. The experimental results demonstrate that the proposed framework can enhance transfer effect and help improve the accuracy of scene classification using low resolution RSI.

From text analysis to image interpretation, the topic model (TM) always plays an important role. With its powerful semantic mining capabilities, it is able to capture the latent spectral and spatial information from remote sensing (RS) images. Recent years have witnessed widespread use of TM to solve the problems in RS image interpretation, i.e., semantic segmentation, target detection, and scene classification. However, there has not yet been a study expatiating and summarizing the current situation of RS applications with TM. This paper intends to systematically summarize the application of TM in RS images and to conduct several typical experiments for comparison. Specifically, the architecture of our work can be explained as follows: 1) the theory of TM; 2) the applications of RS based on TM; 3) experimental analysis of typical TM methods to provide reference for further understanding, and 4) summary and prospects for guiding further research into TM for RS data.

The positive unlabeled (PU) learning aims to train a binary classifier from a set of positive labeled samples and other unlabeled samples. Much research has been done on this special branch of weakly supervised classification problems. Since only part of the positive class is labeled, the classical PU model trains the classifier assuming the class-prior is known. However, the true class prior is usually difficult to obtain and must be learned from the given data, and the traditional methods may not work. In this paper, we formulate a convex formulation to jointly solve the class-prior unknown problem and train an accurate classifier with no need of any class-prior assumptions or additional negative samples. The class prior is estimated by pursuing the optimal solution of gradient thresholding and the classifier is simultaneously trained by performing empirical unbiased risk. The detailed derivation and theoretical analysis of the proposed model are outlined, and a comparison of our experiments with other representative methods prove the superiority of our method.

Hyperspectral image (HSI) contains complex multiple structures. Therefore, the key problem analyzing the intrinsic properties of an HSI is how to represent the structure relationships of the HSI effectively. Hypergraph is very effective to describe the intrinsic relationships of the HSI. In general, Euclidean distance is adopted to construct the hypergraph. However, this method cannot effectively represent the structure properties of high-dimensional data. To address this problem, we propose a sparse-adaptive hypergraph discriminant analysis (SAHDA) method to obtain the embedding features of the HSI in this letter. SAHDA uses the sparse representation to reveal the structure relationships of the HSI adaptively. Then, an adaptive hypergraph is constructed by using the intraclass sparse coefficients. Finally, we develop an adaptive dimensionality reduction mode to calculate the weights of the hyperedges and the projection matrix. SAHDA can adaptively reveal the intrinsic properties of the HSI and enhance the performance of the embedding features. Some experiments on the Washington DC Mall hyperspectral data set demonstrate the effectiveness of the proposed SAHDA method, and SAHDA achieves better classification accuracies than the traditional graph learning methods.

As one of the important applications of hyperspectral imagery (HSI) processing, the Mahalanobis distance-based detector in anomaly detection used to extract knowledge from the background and then calculate the Mahalanobis distance to obtain the detection result. Different from the previous work, a novel low-rank and sparse matrix decomposition (LRaSMD)-based dictionary reconstruction and anomaly extraction framework constructs the detector by a comprehensive combination of the low-rank and sparse matrix for hyperspectral anomaly detection. We use the LRaSMD to fully exploit the background and sparse information to construct the detector. For the low-rank part, we use atom usage probability to reconstruct the dictionary for follow-up collaborative representation (CR). For the sparse part, we calculate the Euclidean distance and get the result by using a ratio to add these two parts. The proposed algorithm was tested on three real-world HSI data sets and demonstrated outstanding detection performance when compared with other state-of-the-art detectors.

Scene classification is one of the most fundamental task in interpretation of high-resolution remote sensing (HRRS) images. Many recent works show that the probabilistic topic models which are capable of mining latent semantics of images can be effectively applied to HRRS scene classification. However, the existing approaches based on topic models simply utilize low-level hand-crafted features to form semantic features, which severely limit the representative capability of the semantic features derived from topic models. To alleviate this problem, this paper propose to build powerful semantic features using the probabilistic latent semantic analysis (pLSA) model, by employing the pre-trained deep convolutional neural networks (CNNs) as feature extractors rather than relying on the hand-crafted features. Specifically, we develop two methods to generate semantic features, called multi-scale deep semantic representation (MSDS) and multi-level deep semantic representation (MLDS), by extracting CNN features from different layers: (1) in MSDS, the final semantic features are learned by the pLSA with multi-scale features extracted from the convolutional layer of a pre-trained CNN; (2) in MLDS, we extract CNN features for densely sampled image patches at different size level from the fully-connected layer of a pre-trained CNN, and concatenate the sematic features learned by the pLSA at each level. We comprehensively evaluate the two methods on two public HRRS scene datasets, and achieve significant performance improvement over the state-of-the-art. The outstanding results demonstrate that the pLSA model is capable of discovering considerably discriminative semantic features from the deep CNN features.

Remote sensing (RS) image retrieval is of great significant for geological information mining. Over the past two decades, a large amount of research on this task has been carried out, which mainly focuses on the following three core issues: feature extraction, similarity metric, and relevance feedback. Due to the complexity and multiformity of ground objects in high-resolution remote sensing (HRRS) images, there is still room for improvement in the current retrieval approaches. In this article, we analyze the three core issues of RS image retrieval and provide a comprehensive review on existing methods. Furthermore, for the goal to advance the state-of-the-art in HRRS image retrieval, we focus on the feature extraction issue and delve how to use powerful deep representations to address this task. We conduct systematic investigation on evaluating correlative factors that may affect the performance of deep features. By optimizing each factor, we acquire remarkable retrieval results on publicly available HRRS datasets. Finally, we explain the experimental phenomenon in detail and draw conclusions according to our analysis. Our work can serve as a guiding role for the research of content-based RS image retrieval.

In recent years, patchwise classification methods are commonly adopted when dealing with the hyperspectral image (HSI) classification. Despite their promising results from the perspective of accuracy, the efficiency of these methods can hardly be ensured since there are redundant computations between adjacent patches. In this paper, we propose a spectral-spatial fully convolutional network for HSI classification with an end-to-end, pixel-to-pixel architecture. Compared with patchwise methods, the proposed framework can avoid the patch extraction and is more efficient. Since the training samples in HSIs are highly sparse, the training strategy in original fully convolutional networks is no longer feasible for HSIs. To solve this problem, we propose a novel mask matrix to assist the back-propagation in the training stage. Considering the importance of spectral and spatial features may vary for different objects and scenes, we combine both features with two weighting factors which can be adaptively learned during the network training. Besides, the dense conditional random field (CRF) is introduced into the framework to further balance the local and global information. Experiments on three benchmark HSI data sets demonstrate that the proposed method can yield competitive results with less time costs compared with patchwise methods.

In this article, a new subspace selection-based discriminative forest (SSDF) method is proposed for the anomaly detection of hyperspectral remote sensing imagery. Most of the existing anomaly detection approaches construct a profile of background instances and then identify instances that do not conform to the background materials as anomalies. However, this type of method generally fails to avoid the background contamination caused by abnormal targets. In this case, we borrow from the concept of isolation and propose an isolation-based discriminative forest model which exploits subsampling rather than modeling the background instances. Furthermore, considering that the data volume of a hyperspectral image is usually huge, the proposed discriminative forest model explores a subspace selection process while splitting the leaf nodes of the binary trees to preserve those bands containing crucial abnormal target information and improve the reliability of the tree-splitting criterion. The proposed detector successfully integrates dimensionality reduction and the data-splitting technique to define pixels as anomaly or background. The extensive experimental results obtained with four hyperspectral data sets demonstrate that the proposed SSDF algorithm outperforms the other state-of-the-art algorithms and hence provides a new perspective for the anomaly detection of hyperspectral remote sensing imagery.

With the rapid development of Earth observation technology, very-high-resolution (VHR) images from various satellite sensors are more available, which greatly enrich the data source of change detection (CD). Multisource multitemporal images can provide abundant information on observed landscapes with various physical and material views, and it is exigent to develop efficient techniques to utilize these multisource data for CD. In this article, we propose a novel and general deep siamese convolutional multiple-layers recurrent neural network (RNN) (SiamCRNN) for CD in multitemporal VHR images. Superior to most VHR image CD methods, SiamCRNN can be used for both homogeneous and heterogeneous images. Integrating the merits of both convolutional neural network (CNN) and RNN, SiamCRNN consists of three subnetworks: deep siamese convolutional neural network (DSCNN), multiple-layers RNN (MRNN), and fully connected (FC) layers. The DSCNN has a flexible structure for multisource image and is able to extract spatial-spectral features from homogeneous or heterogeneous VHR image patches. The MRNN stacked by long-short term memory (LSTM) units is responsible for mapping the spatial-spectral features extracted by DSCNN into a new latent feature space and mining the change information between them. In addition, FC, the last part of SiamCRNN, is adopted to predict change probability. The experimental results in two homogeneous data sets and one challenging heterogeneous VHR images data set demonstrate that the promising performances of the proposed network outperform several state-of-the-art approaches.

The maximum a posteriori (MAP) estimation model-based sub-pixel mapping (SPM) method is an alternative way to solve the ill-posed SPM problem. The MAP estimation model has been proven to be an effective SPM approach and has been extensively developed over the past few years, as a result of its effective regularization capability that comes from the spatial regularization model. However, various spatial regularization models do not always truly reflect the detailed spatial distribution in a real situation, and the over-smoothing effect of the spatial regularization model always tends to efface the detailed structural information. In this article, under the scenario of time-series observation by remote sensing imagery, the joint spectral-spatial-temporal MAP-based (SST_MAP) model for SPM is proposed. In SST_MAP, a newly developed temporal regularization model is added to the MAP model, based on the prerequisite for a temporally close fine image covering the same study region. This available fine image can provide the specific spatial structures most closely conforming to the ground truth for a more precise constraint, thereby reducing the over-smoothing effect. Furthermore, the three dimensions are mutually balanced and mutually constrained, to reach an equilibrium point and achieve restoration of both smooth areas for the homogeneous land-cover classes and a detailed structure for the heterogeneous land-cover classes. Four experiments were designed to validate the proposed SST_MAP: three synthetic-image experiments and one real-image experiment. The restoration results confirm the superiority of the proposed SST_MAP model. Notably, under the background of time-series observation, SST_MAP provides an alternative way of land-cover change detection (LCCD), achieving both detailed spatial-scale and high-frequency temporal LCCD observation for the study case of urbanization analysis within the city of Wuhan in China.

Dimensionality reduction (DR) is an important way of improving the classification accuracy of a hyperspectral image (HSI). Graph learning, which can effectively reveal the intrinsic relationships of data, has been widely used in the case of HSIs. However, most of them are based on a simple graph to represent the binary relationships of data. An HSI contains complex high-order relationships among different samples. Therefore, in this article, we propose a hybrid-graph learning method to reveal the complex high-order relationships of the HSI, termed enhanced hybrid-graph discriminant learning (EHGDL). In EHGDL, an intraclass hypergraph and an interclass hypergraph are constructed to analyze the complex multiple relationships of a HSI. Then, a supervised locality graph is applied to reveal the binary relationships of a HSI which can form the complementarity of a hypergraph. Simultaneously, we also construct a weighted neighborhood margin model to boost the difference of samples from different classes. Finally, we design a DR model based on the intraclass hypergraph, the interclass hypergraph, the supervised locality graph, and the weighted neighborhood margin to improve the compactness of the intraclass samples and the separability of the interclass samples, and an optimal projection matrix can be achieved to extract the low-dimensional embedding features of the HSI. To demonstrate the effectiveness of the proposed method, experiments have been conducted on the Indian Pines, PaviaU, and HoustonU data sets. The experimental results show that EHGDL can generate better classification performance compared with some related DR methods. As a result, EHGDL can better reveal the complex intrinsic relationships of a HSI by the complementarity of different characteristics and enhance the discriminant performance of land-cover types.

Visible remotely sensed images usually suffer from the haze, which contaminates the surface radiation and degrades the data quality in both spatial and spectral dimensions. This study proposes a spatial-spectral adaptive haze removal method for visible remote sensing images to resolve spatial and spectral problems. Spatial adaptation is considered from global and local aspects. A globally nonuniform atmospheric light model is constructed to depict spatially varied atmospheric light. Moreover, a bright pixel index is built to extract local bright surfaces for transmission correction. Spectral adaptation is performed by exploring the relationships between image gradients and transmissions among bands to estimate spectrally varied transmission. Visible remote sensing images featuring different land covers and haze distributions were collected for synthetic and real experiments. Accordingly, four haze removal methods were selected for comparison. Visually, the results of the proposed method are completely free from haze and colored naturally in all experiments. These outcomes are nearly the same as the ground truth in the synthetic experiments. Quantitatively, the mean-absolute-error, root-mean-square-error, and spectral angle are the smallest, and the coefficient-of-determination (R2) is the largest among the five methods in the synthetic experiments. R2, structural similarity index measure, and the correlation coefficient between the result of the proposed method and the reference image are closest to 1 in the real data experiments. All experimental analyses demonstrate that the proposed method is effective in removing haze and recovering ground information faithfully under different scenes.

In this article, we propose a polarimetric calibration (PolCal) algorithm to estimate the system crosstalk, cross-polarization (x-pol), and co-polarization (co-pol) channel imbalance (CI) when ground corner reflectors (CRs) are unavailable. The current PolCal process requires at least one trihedral CR to determine the co-pol CI. However, the deployment of ground CRs is costly and may even be impossible in some areas. To calibrate a polarimetric image without CRs, our proposed method automatically extracts the volume-dominated and Bragg-like pixels as a reference to estimate the crosstalk, x-pol, and co-pol CI values. Then, a first-order polynomial model is exploited to fit the co-pol CI to further improve calibration accuracy. In the experimental section, we demonstrate the effectiveness of our proposed method with data from two of China's newly developed very high-resolution systems. The experiments confirmed that the proposed workflow can be considered as a feasible calibration scheme when the ground deployment of CRs is impossible, and it is also an effective analysis tool for the assessment of calibrated products.

Subpixel mapping (SPM) is a useful technique that can interpret the spatial distribution inside mixed pixels and produce a finer-resolution classification map for hyperspectral remote-sensing imagery. However, SPM is essentially an ill-posed problem that requires additional information to produce the unique solution. The limited information of a single image is insufficient to make the mapping problem well posed, whereas the complementary spatial information of multiple shifted images is able to reduce the uncertainty and generate an accurate map. The maximum a posteriori model is a feasible way to incorporate auxiliary information for SPM with multiple shifted images, but it introduces a sensitive regularization parameter, which is difficult to preset. Furthermore, the fixed parameter in the iterations influences the incorporation of the multiple images and the spatial prior. In this article, to address these issues, a multiobjective SPM framework for use with multiple shifted hyperspectral images (MOMSM) is proposed. In the proposed algorithm, a multiobjective model consisting of two objective functions, i.e., data fidelity and spatial prior terms, is constructed to transform the SPM into a multiobjective optimization problem, to get rid of the sensitive regularization parameter. To simultaneously optimize the two objective functions, a multiobjective memetic algorithm with a local search operator and an adaptive global replacement strategy is proposed. The multiple images and spatial information can be dynamically fused and the optimal mapping solution with a good balance between the two objectives can be finally obtained. Experiments conducted on both synthetic and real data sets confirm that the proposed method outperforms the other tested SPM algorithms.

Feature selection is an effective way to reduce the data dimensionality of hyperspectral imagery and obtain a better performance in the subsequent applications, such as classification. The ideal approach is to obtain the optimal tradeoff between two criteria for hyperspectral image feature selection: 1) information preservation and 2) redundancy reduction. However, constructing a hyperspectral feature selection model for the above two criteria is difficult due to the complexity of hyperspectral imagery. Although evolutionary multiobjective optimization methods have been recently presented to simultaneously optimize the above criteria, they cannot control the global exploration versus local exploitation capabilities in the search space for the hyperspectral feature selection problem. Thus, in this article, a novel discrete sine cosine algorithm (SCA)-based multiobjective feature selection (MOSCA_FS) approach is proposed for hyperspectral imagery. In the proposed method, a novel and effective framework of multiobjective hyperspectral feature selection is designed. In the framework, the ratio between the Jeffries-Matusita (JM) distance and mutual information (MI) is modeled to minimize the redundancy and maximize the relevance of the selected feature subset. In addition, another measurement - the variance (Var) - is applied for maximizing the information amount. Furthermore, to resolve the discrete hyperspectral feature selection problem, a novel discrete SCA is first proposed, which enhances the selection of the ideal feature subset. The effectiveness and universality of the proposed method was verified by experiments with ten University of California at Irvine (UCI) data sets, five hyperspectral image data sets, and one spectral data set of typical surface features.

Hyperspectral images (HSIs) are typical high-dimensional and complex data. As such, the clustering of HSIs is a challenging task. Out of the motivation to find the low-dimensional structure representation of the high-dimensional data, sparse subspace clustering (SSC) methods have been proposed in recent studies. Sparse representation is an important technique in SSC, which is aimed at obtaining the sparse coefficient matrix of the HSI data. Generally speaking, the acquisition of the sparse coefficient matrix is an ill-posed problem, and the existing methods introduce an extra condition as a regularization term to resolve it. However, the regularization parameter is determined manually, which is difficult and lacks self-adaptability. Hence, in this article, a multi-objective SSC method for hyperspectral imagery is proposed, which simultaneously optimizes the sparse term and the data fidelity term. In addition, the spatial structure information of the HSIs is often neglected in the processing model, and thus, a spatial prior term, as the third optimization objective function, is also tested in this article. As a result, there is no need to manually set a regularization parameter. Furthermore, by using the l0norm as the sparse term, this reduces the error caused by the convex relaxation of the other norms. In the proposed method, a multi-objective optimization model is first used to acquire the sparse coefficient matrix, in which a strategy for constructing the dictionary is proposed for more precise and efficient multi-objective optimization. In addition, a knee point-based selection method is utilized to automatically select the optimal sparse representation solution from the Pareto front. The adjacency matrix is then constructed according to the sparse coefficient matrix. Finally, a spectral clustering method is used to obtain clustering results. Experiments undertaken with four HSI data sets confirm the effectiveness of the proposed method.

Anomaly detection is an active area of research in hyperspectral information processing. Recently, low-rank representation has been applied in hyperspectral anomaly detection. However, the existing low-rank-based methods either involve a complicated dictionary construction process or the anomaly-background separation which is not sufficient. In this article, to solve these problems, a novel hyperspectral anomaly detection method based on a locally enhanced low-rank prior (LELRP-AD) is proposed. This article is inspired by the observation that, in local homogeneous regions, the background signals hold an enhanced low-rank property while the anomalies exhibit spatial sparsity. Based on this observation, the background pixels can be low-rank reconstructed by a set of basis background signals, whereas anomalies can be represented as sparse residuals. First, image segmentation is performed to enhance the homogeneity of the background, in which a Potts-based image segmentation algorithm is adopted with postprocessing, thus avoiding the need for a complicated spectral dictionary for the representation of the background. Furthermore, the original hyperspectral data matrix is augmented with extracted background endmembers for the low-rank and sparse matrix decomposition, to further achieve anomaly-background separation. The experimental results obtained on four real hyperspectral data sets demonstrate the merit and viability of the proposed method compared with the current state-of-the-art methods.

Hyperspectral images (HSIs) are normally corrupted by a mixture of various noise types, which degrades the quality of the acquired image and limits the subsequent application. In this article, we propose a novel denoising method for the HSI restoration task by combining nonlocal low-rank tensor decomposition and total variation regularization, which we refer to as TV-NLRTD. To simultaneously capture the nonlocal similarity and high spectral correlation, the HSI is first segmented into overlapping 3-D cubes that are grouped into several clusters by the k-means++ algorithm and exploited by low-rank tensor approximation. Spatial-spectral total variation (SSTV) regularization is then investigated to restore the clean HSI from the denoised overlapping cubes. Meanwhile, the ℓ1-norm facilitates the separation of the clean nonlocal low-rank tensor groups and the sparse noise. The proposed TV-NLRTD method is optimized by employing the efficient alternating direction method of multipliers (ADMM) algorithm. The experimental results obtained with both simulated and real hyperspectral data sets confirm the validity and superiority of the proposed method compared with the current state-of-the-art HSI denoising algorithms.

Deep learning techniques have provided significant improvements in hyperspectral image (HSI) classification. The current deep learning-based HSI classifiers follow a patch-based learning framework by dividing the image into overlapping patches. As such, these methods are local learning methods, which have a high computational cost. In this article, a fast patch-free global learning (FPGA) framework is proposed for HSI classification. The proposed framework consists of three main parts: 1) a designed sampling strategy; 2) an encoder-decoder-based fully convolutional network (FCN); and 3) lateral connections between the encoder and decoder. In FPGA, an encoder-decoder-based FCN is utilized to consider the global spatial information by processing the whole image, which results in fast inference. However, it is difficult to directly utilize the encoder-decoder-based FCN for HSI classification as it always fails to converge due to the insufficiently diverse gradients caused by the limited training samples. To solve the divergence problem and maintain the FCNs abilities of fast inference and global spatial information mining, a global stochastic stratified (GS2) sampling strategy is first proposed by transforming all the training samples into a stochastic sequence of stratified samples. This strategy can obtain diverse gradients to guarantee the convergence of the FCN in the FPGA framework. For a better design of FCN architecture, FreeNet, which is a fully end-to-end network for HSI classification, is proposed to maximize the exploitation of the global spatial information and boost the performance via a spectral attention-based encoder and a lightweight decoder. A lateral connection module is also designed to connect the encoder and decoder, fusing the spatial details in the encoder and the semantic features in the decoder. The experimental results obtained using three public benchmark data sets suggest that the FPGA framework is superior to the patch-based framework in both speed and accuracy for HSI classification.

Monitoring airports using remote sensing imagery require us to first detect the airports and then perform airplane detection. Detecting airports and airplanes with large-scale remote sensing imagery are significant and challenging tasks in the field of remote sensing. Although many detection algorithms have been developed for detecting airports and airplanes in remote sensing imagery, the efficiency of the processing does not meet the needs of real applications in large-scale remote sensing imagery. In recent years, deep learning techniques, such as deep convolutional neural networks (DCNNs), have achieved great progress in image recognition. However, training a DCNN needs a large number of training examples to accurately fit the data distribution. Annotating training examples in large-scale remote sensing imagery is time-consuming, which makes the pipeline inefficient. In this article, to overcome the above two weaknesses, we propose a novel cycling data-driven framework for efficient and robust airport localization and airplane detection. The proposed method consists of three modules: cycling by example refinement (C), offline learning (OL), and online representation (OR), namely cycling, offline learning, and online representation (COLOR). The OR module is a coarse-to-fine cascaded convolutional neural network, which is used to detect airports and airplanes. The example refinement (ER) module implements the cycling and makes use of the unlabeled remote sensing images and the corresponding predictions obtained by the OR module, to generate training examples. The OL module aims to use the training examples from the ER module to update the OR module, to further improve the performance. The whole workflow involves COLOR. The COLOR framework was used to detect airplanes and airports in 512 large-scale Gaofen-2 (GF-2) remote sensing images with 29$200\times27$ 620 pixels. The results showed that the proposed method obtained a mean average precision (mAP) of 88.32% for the airplane detection. In addition due to the proposed coarse-to-fine cascaded OR module the proposed method is much faster than the traditional approaches in real-world applications.

Recent years have witnessed the great success of deep learning models in semantic segmentation. Nevertheless, these models may not generalize well to unseen image domains due to the phenomenon of domain shift. Since pixel-level annotations are laborious to collect, developing algorithms which can adapt labeled data from source domain to target domain is of great significance. To this end, we propose self-ensembling attention networks to reduce the domain gap between different datasets. To the best of our knowledge, the proposed method is the first attempt to introduce selfensembling model to domain adaptation for semantic segmentation, which provides a different view on how to learn domain-invariant features. Besides, since different regions in the image usually correspond to different levels of domain gap, we introduce the attention mechanism into the proposed framework to generate attention-aware features, which are further utilized to guide the calculation of consistency loss in the target domain. Experiments on two benchmark datasets demonstrate that the proposed framework can yield competitive performance compared with the state of the art methods.

This paper presents a region-partition based attraction field dual representation for line segment maps, and thus poses the problem of line segment detection (LSD) as the region coloring problem. The latter is then addressed by learning deep convolutional neural networks (ConvNets) for accuracy, robustness and efficiency. For a 2D line segment map, our dual representation consists of three components: (i) A region-partition map in which every pixel is assigned to one and only one line segment; (ii) An attraction field map in which every pixel in a partition region is encoded by its 2D projection vector w.r.t. the associated line segment; and (iii) A squeeze module which squashes the attraction field to a line segment map that almost perfectly recovers the input one. By leveraging the duality, we learn ConvNets to compute the attraction field maps for raw in-put images, followed by the squeeze module for LSD, in an end-to-end manner. Our method rigorously addresses several challenges in LSD such as local ambiguity and class imbalance. Our method also harnesses the best practices developed in ConvNets based semantic segmentation methods such as the encoder-decoder architecture and the a-trous convolution. In experiments, our method is tested on the WireFrame dataset and the YorkUrban dataset with state-of-the-art performance obtained. Especially, we advance the performance by 4.5 percents on the WireFramedataset. Our method is also fast with 6.6∼10.4 FPS, outperforming most of existing line segment detectors.

In this work, a progressively spatio-temporal patch group learning framework for cloud and shadow removal in Sentinel-2 data is proposed. Through sorting the spatial and corresponding multi-temporal patches with masks as the patch group fashion, a spatiotemporal patch group recovering model is developed using a global-local deep CNN. Finally, all the ergodic patches are weighted aggregated with integrity measure, then updated spatial data and its mask are regenerated through progressive iteration. Two experiments have been performed to demonstrate the effectiveness of the proposed method on Sentienl-2 MSI data, with single/multiple temporal imageries in small and largescale scenarios.

Crop classification and mapping are important to socioeconomic, food safety, and policy-making. Accurate and timely spatial distribution of crop types based on remote sensing is important for both scientific and practical purposes. However, the existing studies mostly focused on main crops, such as winter wheat, paddy rice and some others. For the garlic crop, there is few relevant research available. So in this paper, in order to explore the feasibility of garlic extraction, a preliminary experiment of garlic mapping based on random forest (GMRF) is carried out with multi-temporal 10-m resolution Sentinel-2 images, by taking Jinxiang County, Shandong Province, China as the experimental area. The experimental results suggest that the proposed GMRF method can achieve a good performance, with an overall accuracy (OA) of 98.56% and a kappa coefficient of 0.967.

Considering the reconstruction error of autoencoder can reflect the characteristic of anomalies, this paper presents a novel hyperpsectral anomaly detection algorithm utilizes the residual of reconstruction error to estimate the anomalous information of the image. We firstly employ an adaptive dual concentric window to collect the training sets, and input pixels in the whole window and the outer window into the sparse autoencoders to learn the representations of the input, respectively. Then we use the representation parameters to reconstruct the test pixel and calculate the detection result by a deviation of reconstruction errors under two cases. The proposed sparse autoencoder-based anomaly detector experimental results have been conducted into the San Diego airport dataset and the Urban area dataset, the detection performances verified by the receiver operating characteristic curve and the area under curve show that the proposed method outperforms other representative detection methods.

In this paper, we propose a multi-level fusion framework to address the pairwise semantic stereo issue. For disparity estimation, we adopt the pyramid stereo matching network. For semantic segmentation, the single segmentation network is proposed with respect to the left image, along with the disparity fusion segmentation network for the combination of semantic features and disparity features. Specifically, the multi-receptive fusion block is designed and employed to fully extract and fuse the contextual information. Finally, the refined segmentation result is obtained via yet another fusion of the multi-model results. The proposed method achieved a mean intersection over union (mIoU) of 79.05%, an average endpoint error (EPE) of 1.3966, and an mIoU-3 of 77.75%, ranking first in the Pairwise Semantic Stereo Challenge of the 2019 IEEE GRSS Data Fusion Contest [1],[2].

Non-linear radiation difference (NRD) will lead to the corresponding features cannot be mapped one by one, so the traditional image feature matching methods based on intensity or gradient fail to be directly applied to the multisource remote sensing image registration. In this paper, a new robust feature descriptor is proposed, which has the invariance of radiation, scale and rotation. The nonlinear diffusion function which is insensitive to the radiation difference is used to construct the scale space so that the descriptors can be used in images with different resolutions. A pixel-by-pixel local phase congruency algorithm is used to extract the corresponding points, and then the features are described by means of rotation invariance description. Feature matching is completed based on feature vector constructed by the descriptor, thus to realize image registration. In the experimental part, three kinds of multisource remote sensing images with large radiation differences were used to test the descriptors. The results showed that the proposed method effectively extracted the corresponding features, and achieved the best effect in the quantitative evaluation of image registration.

Building extraction using additional information such as height information has been paid more attention, due to its promising performance. However, additional information is not always easy to collect and may be not available at test stage in practice. Existing building extraction methods using additional information will fail, due to the lack of clear strategies to deal with this situation. This paper proposes a novel multiple kernel SVM+ (MK-SVM+) method to fully exploit additional information (referred as privileged information) which is only available at the training stage. MK-SVM+ simultaneously learns optimal adaptive combined kernels using multiple different base kernels, and builds a new SVM+ model using privileged information. As a result, the derived MK-SVM+ method has more discriminative ability for building extraction. Performance evaluations on a real-world dataset show that our method outperforms compared methods and demonstrate the effectiveness of the proposed method.

Vegetation water content (VWC) is recognized as an important parameter in vegetation growth study. Recently, the ground-based GNSS-R method is emerging in monitoring VWC owing to its high accuracy. However, the small footprint and sparse distribution hinder its application. Therefore, we propose a method to improve the spatial resolution of GNSS-R VWC products by downscaling with other products highly correlated with VWC, using random forest (RF). Satisfactory downscaling results with cross-validation R values of 0.83 and RMSE of 0.025 were obtained. VWC images with a 500-m spatial resolution were then acquired, which is consistent with the distribution of NDVI and GPP, further indicating the accuracy of the downscaling results.

Sub-pixel mapping (SPM) can interpret the sub-pixel spatial distribution of land-cover classes in hyperspectral image, which is an ill-posed problem due to the inadequate information of a single image. Auxiliary information provided by multiple shifted (MS) images can make SPM problem well-posed and improve mapping accuracy. The maximum a posteriori (MAP) technique can incorporate the auxiliary information of MS images, but it introduces a fixed weight parameter to fuse the auxiliary information and spatial prior information, heavily influencing the mapping result. This paper proposed a novel SPM method to model the auxiliary information and spatial prior information into two objective functions, which can be simultaneously optimized by the devised multiobjective evolutionary algorithm. Therefore, there is no need of weight parameter, and the two objective functions can be intelligently integrated during the evolution. Experimental results and parameter analysis have indicated the superiority of the proposed method.

The tailings reservoir is an inevitable part of the production of metal mines, and due to it is usually the accumulation of waste residue and waste water, the risk source of artificial debris flow with high potential energy has been formed and the environmental risk cannot be underestimated. Thus, it is an important disaster and environmental protection project for the mining enterprises. However, the existing methods cannot conduct a comprehensive disaster and environmental monitoring, considering the remote sensing technology is an effective method for the large-scale monitoring, thus, this global monitoring will be carried out through a novel UAV-ground hyper-spectral joint observation and processing, where the UAV hyper-spectral image, the ground hyper-spectral data of the water and waste residue, and water quality testing report will be used. In addition, the study area is in Xinjiang, the Belt and Road.

For hyper-spectral image band selection, there are two main key concerns, which are the curial information preservation and the redundancy information reduction. Since the two objectives are contradictory, the single-objective based band selection methods are usually unsatisfactory, thus, a superior approach which can obtain a trade-off between them is needed. In order to address this problem, an evolutionary computation method called sine cosine algorithm which has the capabilities of the global exploration and the local exploitation is applied, and its multi-objective discrete version is designed for multi-objective band selection. In addition, in this paper, two novel measures are utilized for meeting the requirements. The effectiveness of the proposed method is confirmed by the experimental results obtained with two real hyper-spectral images.

Deriving accurate snow depth is of great importance since snow cover is an informative indicator of climate change. The objective of this study is to develop a snow-depth retrieval algorithm based on a deep learning approach by fusing passive microwave remote sensing brightness temperature, station observations and GNSS-R snow-depth product to improve the accuracy of snow-depth retrieval. The results show that DBN performs the best compared with three alternative algorithms.

In this study, the performance of the MODIS C6 multiangle implementation of atmospheric correction (MAIAC) 1-km aerosol optical depth (AOD) products in China during 2008-2016 are validated using 17 AERONET sites. The results show that the correlation coefficient (R) for MAIAC in C6 is 0.942 and the RMSE is 0.2. Meanwhile, the fraction above the EE is shown as 49.85%, indicating that the MAIAC retrievals will integrally overestimate AOD values in China. Especially, significant overestimation of MAIAC is found at vegetation sites with high elevation and in areas round water. The scale of high-resolution AOD distribution is fine while an obvious boundary caused by aerosol models is also observed.

Optical remote sensing images are inevitably corrupted by clouds during the acquisition process. To reconstruct the missing information contaminated by clouds, this paper introduces a new cloud removal method based on X-fork generative adversarial network with multitemporal data, which can be named X-MTGAN. By utilizing the auxiliary differential image between two imaging times, X-MTGAN can be well trained with multitemporal SAR-optical data. Then, the target optical image is synthesized with an end-to-end generator of the X-MTGAN, which has advantages in capturing change information between two temporal images. Finally, the cloud-free image can be subsequently acquired by replacing cloud-contaminated regions with the simulated image. By utilizing Setinel-1 and Sentinel-2 data, experiments are conducted to validate the feasibility of the proposed approach. Compared with the state-of-the-art methods, the results illustrate that X-MTGAN is visually and quantitatively effective in the removal of clouds, which has favorable applicability and competitive performance.

Recent research has shown the great potential of convolutional neural networks (CNNs) in hyperspectral image (HSI) classification. Nevertheless, CNN based approaches may lead to over-smoothing effect due to the spatial information loss during the convolution and pooling operations. To address this problem, in this paper, we propose a deep supervised and boundary-constrained network (DSBC-Net) which takes the boundary information into consideration. With the well-designed architecture, DSBC-Net can achieve segmentation and edge detection for HSI simultaneously. Besides, a novel deep supervision strategy is proposed to improve the training of the deep neural network. Experimental results on two benchmark HSI datasets demonstrate that the DSBC-Net can better maintain the boundaries of different objects with higher classification accuracy compared with previous methods.

This study attempts to develop a novel solution for the estimation of regional surface soil moisture (SSM) using a machine learning model trained on in situ measurement target data. Specifically, the generalized regression neural network (GRNN) is employed to establish the relationship between in-situ measurements from Sparse Network Stations (SNSs) in the continental U.S. and passive microwave observations from the Soil Moisture Active Passive (SMAP) satellite for April 2015 to March 2018. More importantly, to address the scale mismatch issue resulting from the small spatial support of in situ measurements, we turn to the extended triple collocation technique whereby individual SNSs' reliability at the SMAP coarse footprint is determined before fed into GRNN. The cross-validation results show that the GRNN model trained on reliable SNSs obtains a fairly good performance, with out-of-sample cross-validated R and unbiased RMSE values of 0.92 and 0.043 cm3cm-3, respectively. Moreover, the comparison in space shows that the spatial patterns of GRNN retrievals is the most consistent with in situ measurements than both the SMAPL3SMP and the ERA-Interim SSM data. Furthermore, the GRNN-estimated SSM time series over stations agrees much better with in-situ measurements than the official SMAP passive SSM product. All these results indicate that the statistical GRNN modeling has shown great potential in estimating reliable regional SSM climate records using in-situ measurements as training references.

High-resolution remote sensing (HRS) image analysis is a fundamental but challenging problem. To bridge the semantic gap, scene understanding has been proposed to achieve higher-level interpretation, through classifying the HRS scene through spatial relationship cognition and semantic induction between the land-cover objects. As a new research field, however, there has not yet been a study expatiating and summarizing the current situation of scene understanding. This paper first defines the concept of scene understanding for HRS imagery, which is different from natural image scene classification. The theory of scene understanding for HRS imagery is investigated, and is classified into four main categories: 1) scene classification based on semantic objects; 2) scene classification based on mid-level features; 3) scene classification based on deep learning; and 4) scene understanding applications based on geographic data mining.

This letter presents two main contributions to the data quality assessment of China's new GaoFen-3 radar satellite. First, we observed a half-pixel misregistration between the horizontal (H) and vertical (V) transmitting channels in the azimuth direction. This was determined by investigating the corner reflector (CR) response of some stripmap products of GaoFen-3 quad-pol stripmap (QPSI) mode. Second, to check whether the azimuth misregistration exists in different beams of QPSI mode, we improved the RADARSAT-2 channel checking method as a tool and evaluated more than 300 stripmap scenes. This letter confirms that the half-pixel misregistration problem, which can cause about 10% decoherence in the co-pol and cross-pol channel correlation coefficients, is common in the GaoFen-3 stripmap products of QPSI mode. Furthermore, the improved method can be considered as an effective way to fix the misregistration problem.

Hyperspectral target detection refers to an approach that tries to locate targets in a hyperspectral image (HSI) on the condition of given targets spectrum, which plays an important role in hyperspectral remote sensing image processing. In this letter, we propose a binary-class collaborative representation-based detector. The proposed algorithm uses the concept that each background pixel can be approximately represented by its adjacent pixels within a sliding dual-window, and each target pixel can also be approximately represented by some pixels of the image; we use the given target pixels to represent it. Before estimating each background pixel, a background dictionary purification process is proposed to further improve the detector performance. The proposed algorithm was tested on three benchmark HSI data sets, and the experimental results show that the proposed algorithm demonstrates outstanding detection performances when compared with other state-of-the-art detectors.

Hyperspectral image (HSI) contains a large number of spatial-spectral information, which will make the traditional classification methods face an enormous challenge to discriminate the types of land-cover. Feature learning is very effective to improve the classification performances. However, the current feature learning approaches are mostly based on a simple intrinsic structure. To represent the complex intrinsic spatial-spectral of HSI, a novel feature learning algorithm, termed spatial-spectral hypergraph discriminant analysis (SSHGDA), has been proposed on the basis of spatial-spectral information, discriminant information, and hypergraph learning. SSHGDA constructs a reconstruction between-class scatter matrix, a weighted within-class scatter matrix, an intraclass spatial-spectral hypergraph, and an interclass spatial-spectral hypergraph to represent the intrinsic properties of HSI. Then, in low-dimensional space, a feature learning model is designed to compact the intraclass information and separate the interclass information. With this model, an optimal projection matrix can be obtained to extract the spatial-spectral features of HSI. SSHGDA can effectively reveal the complex spatial-spectral structures of HSI and enhance the discriminating power of features for land-cover classification. Experimental results on the Indian Pines and PaviaU HSI data sets show that SSHGDA can achieve better classification accuracies in comparison with some state-of-the-art methods.

Change detection has been a hotspot in the remote sensing technology for a long time. With the increasing availability of multi-temporal remote sensing images, numerous change detection algorithms have been proposed. Among these methods, image transformation methods with feature extraction and mapping could effectively highlight the changed information and thus has a better change detection performance. However, the changes of multi-temporal images are usually complex, and the existing methods are not effective enough. In recent years, the deep network has shown its brilliant performance in many fields, including feature extraction and projection. Therefore, in this paper, based on the deep network and slow feature analysis (SFA) theory, we proposed a new change detection algorithm for multi-temporal remotes sensing images called deep SFA (DSFA). In the DSFA model, two symmetric deep networks are utilized for projecting the input data of bi-temporal imagery. Then, the SFA module is deployed to suppress the unchanged components and highlight the changed components of the transformed features. The change vector analysis pre-detection is employed to find unchanged pixels with high confidence as training samples. Finally, the change intensity is calculated with chi-square distance and the changes are determined by threshold algorithms. The experiments are performed on two real-world data sets and a public hyperspectral data set. The visual comparison and the quantitative evaluation have shown that DSFA could outperform the other state-of-the-art algorithms, including other SFA-based and deep learning methods.

In recent decades, spatiotemporal subpixel mapping (SSM) approaches have been extensively developed to deal with the mixed-pixel problem by incorporating fine spatial resolution images with the same field of view from different acquisition times. This is an alternative to the conventional subpixel mapping (SPM) method, which is based on only monotemporal images. SSM has become one of the state-of-the-art SPM approaches, and has been widely applied in urban management and ecological monitoring. However, in the traditional SSM methods, the spatial correlation within the multitemporal images is insufficiently exploited and is ignored in the spatiotemporal model construction. In addition, the contribution of the land covers' spatial distribution in the multitemporal images is incompletely considered, and the geographic variation during the time interval is ignored, which underutilizes the spatiotemporal information. In this paper, an SSM algorithm based on a geographically weighted regression (GWR) model and evolutionary algorithm theory, called spatiotemporal subpixel geographical evolution mapping (STGEM), is proposed for multitemporal remote sensing images. The proposed algorithm considers the spatiotemporal dependence not only between the current subpixel and the corresponding fine pixel, but also with the neighboring fine distribution patterns within the fine image. Moreover, the potential temporal information of the geospatial variation is fully realized by considering not only the time interval between the bitemporal images, but also the ratio of changed area between them, based on the GWR model. Two synthetic-image experiments with bitemporal Landsat 8 images and bitemporal QuickBird images were carried out to validate the proposed algorithm. Furthermore, a real-image experiment using a bitemporal pair of Gaofen-2 images and a Landsat 8 image was also undertaken. A comparison was made with several traditional SPM methods, as well as the state-of-the-art SSM approaches, and the experimental results confirmed the superiority of the proposed STGEM algorithm. The proposed STGEM achieves a fine spatial and temporal resolution thematic map, both qualitatively and quantitatively, and has great potential for fine-scale and frequent time-series observation and monitoring.

Ice surface temperature (IST) is a key parameter for the study of polar ice sheets and ice shelves. In this study, an improved single-channel (ISC) algorithm based on the radiative transfer equation is proposed for IST retrieval from Landsat-8 band 10 data. The main steps in the proposed ISC algorithm include: 1) simulation of atmospheric radiative parameters by regression against the atmospheric water vapor content and the effective mean atmospheric temperature; 2) calculation of IST using Planck's equation, instead of using Taylor's approximation; and 3) implementation of an iterative scheme for IST calculation. The errors from using Taylor's approximation and the atmospheric radiative parameter simulation were quantitatively estimated. A sensitivity analysis of ISC to possible errors in atmospheric water vapor content, brightness temperature, and satellite observations was also conducted. The results of the sensitivity analysis showed that the proposed algorithm is robust to the atmospheric water vapor content, but is sensitive to the calibration precision of the thermal infrared sensor. Verification using a simulated approach showed better IST variability from ISC than the original SC algorithm [the root-mean-square errors (RMSEs) were 0.3252 and 0.7176 K, respectively]. When compared with near-surface air temperatures from 68 automatic weather stations data in Greenland and 25 data in the Antarctic, the bias and RMSE from the ISC algorithm were again better than those from the SC algorithm. The IST from Moderate Resolution Imaging Spectroradiometer (MODIS) was found to be underestimated with respect to the results of both the SC and ISC algorithms. Maps of the spatial distributions of IST derived from samples of Landsat-8 images are presented. The rationale of each step in the proposed ISC algorithm is also presented so that this can provide further support to the authenticity of the results.

Enhancing the spatial resolution of hyperspectral (HS) images by fusing with higher spatial resolution multispectral (MS) data is of significance for applications. However, due to the narrow bandwidth, HS images (HSIs) are vulnerable to various types of noise, such as Gaussian noise and stripes, which can severely affect the fusion performance. This paper focuses on antinoise HS and MS image fusion to enhance the spatial details and suppress the noise. By analysis of the intrinsic structure and noise properties, we formulate this problem as the minimization of an objective function. Under the optimization framework, small multilinear ranks in tensor are first used to identify the intrinsic structures of the clean HSI part. Then, considering the high spectral correlation, it is assumed that any bands can be represented by the combination of certain adjacent bands. The difference between one band and its corresponding combination can be used to preserve the spatio-spectral consistency and characterize the distribution of sparse noise (such as stripe noise), based on the variational properties along two directions. The alternating direction method of multipliers (ADMM) is applied to solve and accelerate the model optimization. Experiments with both simulated- and real-data demonstrate the effectiveness of the proposed model and its robustness to the noise, in terms of both qualitative and quantitative perspectives.

Despite the availability of an increasing amount of remote sensing images, problems still arise in that the knowledge from existing images is underutilized and the collection of reference knowledge for each newly obtained image is expensive. Recently, an attractive solution called “transfer learning” has received increasing attention in the remote sensing field, by transferring knowledge from source domains to help improve the learning procedure in the target domain. In this paper, we propose a sparse subspace correlation analysis-based supervised classification (SSCA-SC) method for transfer learning in hyperspectral remote sensing image classification, which is not restricted by the data dimensionality or the data acquisition sensors. Specifically, we first propose a sparse subspace correlation analysis (SSCA) method to simultaneously learn the optimal projection matrices for heterogeneous domains into a common subspace and obtain sparse reconstruction coefficients over a shared self-expressive dictionary in the derived subspace. In order to fully utilize the label information to improve the class separability, the SSCA-SC framework learns more discriminative representations for the input data by training a corresponding SSCA model for each class. As a result, the projected data belonging to the same class are maximally correlated and represented well, while those from different classes will have a low correlation. Another advantage of the SSCA-SC framework lies in the fact that it not only learns new representations for the data from different domains but it also designs a discriminative and robust classifier that properly adapts to the new representation. The proposed method was tested with three hyperspectral remote sensing data sets, and the experimental results confirm the effectiveness and reliability of the proposed SSCA-SC method.

The optical remote sensing images not only have to make a fundamental tradeoff between the spatial and spectral resolutions, but also are inevitable to be polluted by the clouds; however, the existing pansharpening methods mainly focus on the resolution enhancement of the optical remote sensing images without cloud contamination. How to fuse the cloud-contaminated images to achieve the joint resolution enhancement and cloud removal is a promising and challenging work. In this paper, a pansharpening method for the challenging cloud-contaminated very high-resolution remote sensing images is proposed. Furthermore, the cloud-contaminated conditions for the practical observations with all the thick clouds, the thin clouds, the haze, and the cloud shadows are comprehensively considered. In the proposed methods, a two-step fusion framework based on multisource and multitemporal observations is presented: 1) the thin clouds, the haze, and the light cloud shadows are proposed to be first jointly removed and 2) a variational-based integrated fusion model is then proposed to achieve the joint resolution enhancement and missing information reconstruction for the thick clouds and dark cloud shadows. Through the proposed fusion method, a promising cloud-free fused image with both high spatial and high spectral resolutions can be obtained. To comprehensively test and verify the proposed method, the experiments were implemented based on both the cloud-free and cloud-contaminated images, and a number of different remote sensing satellites including the IKONOS, the QuickBird, the Jilin (JL)-1, and the Deimos-2 images were utilized. The experimental results confirm the effectiveness of the proposed method.

In the field of spatial–spectral fusion, the variational model-based methods and the deep learning (DL)-based methods are state-of-the-art approaches. This paper presents a fusion method that combines the deep neural network with a variational model for the most common case of spatial–spectral fusion: panchromatic (PAN)/multispectral (MS) fusion. Specifically, a deep residual convolutional neural network (CNN) is first trained to learn the gradient features of the high spatial resolution multispectral image (HR-MS). The image observation variational models are then formulated to describe the relationships of the ideal fused image, the observed low spatial resolution multispectral image (LR-MS) image, and the gradient priors learned before. Then, fusion result can then be obtained by solving the fusion variational model. Both quantitative and visual assessments on high-quality images from various sources demonstrate that the proposed fusion method is superior to all the mainstream algorithms included in the comparison, in terms of overall fusion accuracy.

Endmember extraction (EE) is a significant task in hyperspectral unmixing. From a multiobjective optimization perspective, this task is extremely challenging because objectives often conflict with each other. Currently, a multiobjective discrete particle swarm optimization algorithm (MODPSO) is applied to handle the multiobjective optimization EE problem such as the root-mean-square error (RMSE) and the volume maximization (VM). However, in MODPSO, the minimization of RMSE by unconstrained least squares (Ucls) may lack accuracy, the update of velocity by the predefined random selection probability p can also affect the exploration and exploitation, and it may lose good solution in the process of the update of particles when the particles are randomly chosen in the nondominated relationship. To address these issues, we present an improved MODPSO (IMODPSO) for hyperspectral EE. IMODPSO employs nonnegative constrained least squares (Ncls) to enhance the accuracy of RMSE. Moreover, IMODPSO eliminates the effects of probability p and combines the restart mechanism to achieve a balance of the exploration and exploitation. In addition, IMODPSO utilizes the archive strategy to reserve good nondominated particles to strengthen the population diversity. The experiments have been conducted on three real hyperspectral images and the results have demonstrated that IMODPSO obtains best performances for EE.

This paper focuses on the blind unmixing technique for analyzing hyperspectral images (HSIs). A joint deconvolution and blind hyperspectral unmixing (DBHU) algorithm is proposed, which is aimed at eliminating the impact of the adjacency effect (AE) on unmixing. In remote sensing imagery, the AE occurs in the presence of atmospheric scattering over a heterogeneous surface. The AE leads to blurring and additional mixing of HSIs and makes it difficult to estimate endmembers and abundances accurately. In this paper, we first model the blurred HSIs by the use of a bilinear mixing model (BMM), where a blurring kernel is used to model the mixing caused by the AE. Based on the BMM, the DBHU problem is formulated as a constrained and biconvex optimization problem. Specifically, the minimum-volume simplex (MVS) is incorporated to deal with the additional mixing caused by the AE, and 3-D total variation (TV) priors are adopted to model the spectral-spatial correlation of the data. In DBHU, the biconvex problem is efficiently solved by a nonstandard application of the alternating direction method of multipliers (ADMM) algorithm, where a block coordinate descent scheme is applied by splitting the original problem into two saddle point subproblems, and then minimizing the subproblems alternately via the ADMM until convergence. The experimental results obtained with both simulated and real data confirm the viability of the proposed algorithm, and DBHU works well, even where both blurring and noise are present in the scene.

Hyperspectral image (HSI) denoising is a crucial preprocessing procedure to improve the performance of the subsequent HSI interpretation and applications. In this paper, a novel deep learning-based method for this task is proposed, by learning a nonlinear end-to-end mapping between the noisy and clean HSIs with a combined spatial-spectral deep convolutional neural network (HSID-CNN). Both the spatial and spectral information are simultaneously assigned to the proposed network. In addition, multiscale feature extraction and multilevel feature representation are, respectively, employed to capture both the multiscale spatial-spectral feature and fuse different feature representations for the final restoration. The simulated and real-data experiments demonstrate that the proposed HSID-CNN outperforms many of the mainstream methods in both the quantitative evaluation indexes, visual effects, and HSI classification accuracy.

Clustering is a very challenging task for hyperspectral imagery (HSI) because of the complex spectral-spatial structures found in such data. Recently, the sparse recovery-based approaches have been introduced to deal with hyperspectral clustering, and have achieved state-of-the-art performances. Several recent works have shown that it is the collaborative representation mechanism over all the dictionary atoms, rather than the sparse constraint that determines the recognition performance. Based on this fact, in this paper, we focus on the working mechanism of collaborative representation to explore its potential in HSI clustering. However, directly introducing collaborative representation clustering (CRC) to HSIs results in several problems, i.e., the high redundancy of the global dictionary atoms and the absence of spatial information, which greatly limit the clustering performance. In view of this, we propose a novel total variation regularized CRC with a locally adaptive dictionary (TV-CRC-LAD) algorithm for HSI. First, the LAD construction strategy is introduced instead of the global dictionary to relieve the high redundancy and the interference of unrelated atoms in the representation process, to more precisely represent each pixel only with the highly correlated atoms. Second, TV regularization is integrated to better account for the rich spatial-contextual information and promotes the piecewise smoothness of the HSI clustering result. The proposed algorithm was tested on three widely used hyperspectral data sets, and the experimental results clearly illustrate that the proposed algorithm outperforms the corresponding sparsity-based clustering methods and the other state-of-the-art methods.

Band selection is an effective approach to mitigate the “Hughes phenomenon” of hyperspectral image (HSI) classification. Recently, sparse representation (SR) theory has been successfully introduced to HSI band selection, and many SR-based methods have been developed and shown great potential and superiority. However, due to the inherent limitations of the SR scheme, i.e., individually representing each band with only a few other bands from the same subspace, the SR-based methods cannot effectively capture the global structures of the data, which limit the band selection performance. In this paper, to overcome this obstacle, the novel Laplacian-regularized low-rank subspace clustering (LLRSC) algorithm is proposed for HSI band selection. On the one hand, the low-rank subspace clustering model is introduced to capture the global structure information for the learned representation coefficient matrix and deal with the HSI band selection task in the clustering framework. On the other hand, considering the high correlation between adjacent bands, 1-D Laplacian regularization is utilized to incorporate the neighboring band information and further reduce the representation bias. Lastly, an eigenvalue analysis algorithm based on band mutation information is utilized to estimate the appropriate size of the band subset. The experimental results indicate that the proposed LLRSC algorithm outperforms the other state-of-the-art methods and achieves a very competitive band selection performance for HSIs.

The existence of hybrid noise in hyperspectral images (HSIs) severely degrades the data quality, reduces the interpretation accuracy of HSIs, and restricts the subsequent HSI applications. In this paper, the spatial-spectral gradient network (SSGN) is presented for mixed noise removal in HSIs. The proposed method employs a spatial-spectral gradient learning strategy, in consideration of the unique spatial structure directionality of sparse noise and spectral differences with additional complementary information for effectively extracting intrinsic and deep features of HSIs. Based on a fully cascaded multiscale convolutional network, SSGN can simultaneously deal with different types of noise in different HSIs or spectra by the use of the same model. The simulated and real-data experiments undertaken in this study confirmed that the proposed SSGN outperforms at mixed noise removal compared with the other state-of-the-art HSI denoising algorithms, in evaluation indices, visual assessments, and time consumption.

Exemplar-based dynamic texture synthesis (EDTS) is targeted to generate new samples of high quality that are perceptually similar to a given input dynamic texture exemplar. This paper addresses the issue of learning the synthesizability of dynamic texture samples. Given a dynamic texture sample, how is its possibility of being synthesized by EDTS methods estimated, and what is the most suitable EDTS algorithm to complete the task? To this end, we propose associating dynamic texture samples with synthesizability scores by learning regression models on a compiled dynamic texture dataset annotated in terms of synthesizability. More precisely, we first define the synthesizability of DT samples and characterize them by a set of spatiotemporal features. We then train regression models on the annotated dataset with feature representation to predict the synthesizability scores of the DT samples and learn classifiers to select the most suitable EDTS algorithm. We further complete the selection, partition and synthesizability prediction of the DT samples in a hierarchical scheme. The learned synthesizability is finally applied to detecting synthesizable regions in videos. Both quantitative and qualitative experiments demonstrate that our method can efficiently learn and predict the synthesizability of DT samples.

In this paper, we propose a novel image inpainting framework consisting of an interpolation step and a low-rank tensor completion step. More specifically, we first initial the image with triangulation-based linear interpolation, and then we find similar patches for each missing-entry centered patch. Treating a group of patch matrices as a tensor, we employ the recently proposed effective t-SVD tensor completion algorithm with a warm start strategy to inpaint it. We observe that the interpolation step is such a rough initialization that the similar patch we found may not exactly match with the reference, so we name the problem as Patch Mismatch and analyse the error caused by it thoroughly. Our theoretical analysis shows that the error caused by Patch Mismatch can be decomposed into two components, one of which can be bounded by a reasonable assumption named local patch similarity, and another part is lower than that using matrix. Experiments on real images verify our method's superiority to the state-of-the-art inpainting methods.

Object detection is an important and challenging problem in computer vision. Although the past decade has witnessed major advances in object detection in natural scenes, such successes have been slow to aerial imagery, not only because of the huge variation in the scale, orientation and shape of the object instances on the earth's surface, but also due to the scarcity of well-annotated datasets of objects in aerial scenes. To advance object detection research in Earth Vision, also known as Earth Observation and Remote Sensing, we introduce a large-scale Dataset for Object deTection in Aerial images (DOTA). To this end, we collect 2806 aerial images from different sensors and platforms. Each image is of the size about 4000 × 4000 pixels and contains objects exhibiting a wide variety of scales, orientations, and shapes. These DOTA images are then annotated by experts in aerial image interpretation using 15 common object categories. The fully annotated DOTA images contains 188, 282 instances, each of which is labeled by an arbitrary (8 d.o.f.) quadrilateral. To build a baseline for object detection in Earth Vision, we evaluate state-of-the-art object detection algorithms on DOTA. Experiments demonstrate that DOTA well represents real Earth Vision applications and are quite challenging.

Object detection in aerial images plays a significant role in intelligent interpretation of aerial images. Hence many effective methods, especially the new-generation data-driven methods, have been developed for this task. Here, we hold the ODAI, a new contest that focused on object detection in aerial images, based on a new large-scale aerial image dataset called DOTA [1]. This contest contains over 3000 large-size images ( 4k×4k pixels), which cover 211,581 instances divided into 15 categories. Each instance is labeled by an arbitrary (8 d.o.f.) quadrilateral. Besides, we propose two tasks for this contest, named object detection with the horizontal bounding box (OD-HBB) and object detection with the oriented bounding box (OD-OBB). The contest was opened on February 7, 2018, and ended on April 30, 2018. A website is open to the public, which provides links to download data and evaluation server. We have totally received 60 registrations. There are 8 teams that have successfully submitted results on the OD-HBB task with the top mAP as 0.719, and 9 teams that have successfully submitted results on the OD-OBB task with the top mAP as 0.705. Through the contest, we hope to draw extensive attention from a wide range of communities and call for more future research and efforts for the task of object detection in aerial images.

The example-based dynamic texture synthesis (EDTS) methods have emerged in multitude, dedicated to generating new dynamic textures (DTs) of high quality from an input exemplar. The problem of EDTS has been studied for several decades, but none of the existing synthesis methods are able to tackle all kinds of dynamic textures equally well. Rather than focus on new synthesis methods, we turn to another way to help EDTS by investigating dynamic texture synthesizability - how synthesizable a specific dynamic texture sample is by EDTS. We propose to predict synthesizability score of a given dynamic texture sample, and suggest which EDTS method is best suited to synthesize it. To this end, we compiled a dynamic texture dataset and annotated each DT in terms of synthesizability. We address the problem of learning dynamic texture synthesizability by using regression model to train a predictor on the data collection. More precisely, we first characterize DT samples by a set of spatiotemporal features. Then, based on dynamic texture descriptors, we train regression models to estimate synthesizability scores and use an additional classifier to choose the optimal EDTS methods. The experiments demonstrate that our method can predict the synthesizability of DT samples effectively.

Blind hyperspectral unmixing (HU) technique aims at identifying pure materials in a hyperspectral image (HSI), called endmembers, and quantifying the corresponding proportions, called abundances, with little prior knowledge. In this paper, the degradation mechanism during data collection - adjacent effect (AE), is considered in the process of blind HU. Since the AE leads to blurring (the loss of sharpness, contrast and apparent resolution) in scene, it blocks the quantitative analysis of HSI in sub-pixel level and makes the estimated endmembers and abundances inaccurate. To solve this problem, a bilinear mixing model is developed to simulate the AE, and a novel algorithm, termed joint deconvolution and blind HU (DBHU) is proposed. In DBHU, the bi-convex optimization problem is efficiently solved by a nonstandard application of the alternating direction method of multipliers (ADMM) algorithm, where a block coordinate descent scheme is applied by splitting the original problem into two saddle-point subproblems and then minimizing the subproblems alternatively via ADMM until convergence. The experimental results on both simulated and real HSI illustrate the viability of the proposed algorithm.

Hyperspectral images have been widely used in classification because of the abundant spectral information. But it can't distinguish the objective with similar spectral character but different elevation. However, LiDAR data can obtain elevation information. Therefore, it will obtain better classification maps if fusing the two data. In recent years, CNN has attracted much attention due to its powerful ability to excavate the potential representation and features of the raw data. However, it's difficult to distinguish the objects with different spectral information but similar surface character. Unlike CNN features, the traditional manual features, such as the normalized vegetation index (NDVI), have a certain characteristic expression significance. In order to consider both the semantic information of traditional manual features and the advanced features of CNN features, this paper proposes a fusion algorithm of hyperspectral and LiDAR fusion based on feature fusion. The proposed algorithm has achieved a good fusion classification effect on the MUUFL Gulfport Hyperspectral and LiDAR Data set.

In this paper, a novel spatiotemporal fusion model based on deep learning is proposed, which handles the huge spatial resolution gap and the nonlinear mapping between the high spatial resolution (HSR) image and the corresponding high temporal resolution (HTR) image at the same imaging time. Considering the huge spatial resolution gap, a two-layer fusion strategy is adopted. In each layer, the convolutional neural network (CNN) model is employed to exploit the non-linear mapping between the HSR and HTR image and reconstruct the high-spatial and high-temporal (HSHT) resolution images. In the experiment, Landsat data is the representation of the high spatial resolution images, MODIS data is used as the corresponding low spatial resolution images. The experimental results on two different datasets clearly illustrate the superiority of the proposed model.

In the past decades, land cover change detection (LCCD) has been dramatically developed, since it provides corroborative support for policy decision, regulatory actions, and subsequent urban-rural activities. Satellite remote sensing image is the major source of LCCD since it is able to revisit the Earth's surface regularly and provide time series images for monitoring and space-time analysis. However, there is always a trade-off between spatial scale and temporal scale, i.e., finer spatial resolution image generally has a lower revisit frequency, leading to an observation omission; while higher revisit frequency image usually has a lower spatial resolution, resulting in a deficiency in detecting finer scale change information. In this paper, a spatial-temporal sub-pixel mapping (SSM) algorithm is proposed on the premise that one pair of fine spatial resolution image with low frequency revisit period and coarse spatial resolution with high frequently revisit period are available, and SSM is taken to restore the coarse image to a finer scale thematic map which can be then compared to the fine image, realizing a frequency and detailed LCCD. SSM is an extension of traditional mono-temporal sub-pixel mapping (SPM) algorithm, and is improved by incorporating temporally fine distribution patterns for a more appropriate restoration of coarse image. A study case for urban expansion LCCD were carried out to verify the ability of the proposed algorithm to handle change detection based on one pair of china-made Gaofen-2 image (GF-2) and Landsat-8 image, the result demonstrate that the proposed SSM algorithm outperform the other traditional SPM, achieving both fine temporal resolution and spatial resolution LCCD for further applications.

Scene classification is a fundamental task in interpretation of remote sensing images, and has become an active research topic in remote sensing community due to its important role in a wide range of applications. Over the past years, tremendous efforts have been made for developing powerful approaches for scene classification of remote sensing images, evolving from the traditional bag-of-visual-words model to the new generation deep convolutional neural networks (CNNs). The deep CNN based methods have exhibited remarkable breakthrough on performance, dramatically outperforming previous methods which strongly rely on hand-crafted features. However, performance with deep CNNs has gradually plateaued on existing public scene datasets, due to the notable drawbacks of these datasets, such as the small scale and low-diversity of training samples. Therefore, to promote the development of new methods and move the scene classification task a step further, we deeply discuss the existing problems in scene classification task, and accordingly present three open directions. We believe these potential directions will be instructive for the researchers in this field.

This paper aims to address the problem of detecting buildings from remote sensing images with very high resolution (VHR). Inspired by the observation that buildings are always more distinguishable in geometries than in texture or spectral, we propose a new geometric building index (GBI) for accurate building detection, which relies on the geometric saliency of building structures. The geometric saliency of buildings is derived from a mid-level geometric representations based on meaningful junctions that can locally describe anisotropic geometrical structures of images. The resulting GBI is measured by integrating the derived geometric saliency of buildings. Experiments on three public datasets demonstrate that the proposed GBI achieves very promising performance, and meanwhile shows impressive generalization capability.

Aerial image scene classification is a fundamental problem for understanding high-resolution remote sensing images and has become an active research task in the field of remote sensing due to its important role in a wide range of applications. However, the limitations of existing datasets for scene classification, such as the small scale and low-diversity, severely hamper the potential usage of the new generation deep convolutional neural networks (CNNs). Although huge efforts have been made in building large-scale datasets very recently, e.g., the Aerial Image Dataset (AID) which contains 10,000 image samples, they are still far from sufficient to fully train a high-capacity deep CNN model. To this end, we present a larger-scale dataset in this paper, named as AID++, for aerial scene classification based on the AID dataset. The proposed AID++ consists of more than 400,000 image samples that are semi-automatically annotated by using the existing the geographical data. We evaluate several prevalent CNN models on the proposed dataset, and the results show that our dataset can be used as a promising benchmark for scene classification.

Satellite remote sensing is a promising approach for the estimation of ground-level PM2.5. In this paper, a deep learning framework for satellite-based PM2.5 estimation is presented. Taking advantage of multi-layer learning and layer-by-layer pre-training, deep learning has the great potential to mine nonlinear relationship between PM2.5 and satellite observations. Firstly, the presented deep learning framework can be employed to estimate ground PM2.5 using satellite-derived aerosol optical depth (AOD). Secondly, the AOD products are retrieved from satellite top-of-atmosphere (TOA) reflectance. The deep learning framework can further be adopted to estimate ground PM2.5 directly from satellite TOA reflectance. The presented framework was tested for AOD-based PM2.5 estimation in China and reflectance-based PM2.5 estimation across Wuhan Metropolitan Area, respectively. The results show that the deep learning framework achieves an outstanding performance for both AOD-based and reflectance-based PM2.5 estimation. This study provides an effective way for the satellite-based estimation of PM2.5.

Hyperspectral image (HSI) contains a large number of spatial-spectral information, which will make the traditional classification methods face an enormous challenge to discriminate the types of land-cover. Feature learning is very effective to improve the classification performances. However, the current feature learning approaches are most based on a simple intrinsic structure. To represent the complex intrinsic spatial-spectral of HSI, a novel feature learning algorithm, termed discriminant spatial-spectral hypergraph learning (DSSHL), has been proposed on the basis of spatial-spectral information and hypergraph learning. DSSHL constructs an intraclass spatial-spectral hypergraph and an interclass spatial-spectral hypergraph to represent the intrinsic properties of HSI. Then, a feature learning model is designed to compact the intraclass information and separate the interclass information. DSSHL can effectively reveal the complex spatial-spectral structures of HSI for land-cover classification. Experimental results on the Salinas HSI data set shows that DSSHL can achieve better classification accuracies in comparison with some state-of-the-art methods.

Many significant applications need land cover information of remote sensing images that are acquired from different areas and times, such as change detection and disaster monitoring. However, it is difficult to find a generic land cover classification scheme for different remote sensing images due to the spectral shift caused by diverse acquisition condition. In this paper, we develop a novel land cover classification method that can deal with large-scale data captured from widely distributed areas and different times. Additionally, we establish a large-scale land cover classification dataset consisting of 150 Gaofen-2 imageries as data support for model training and performance evaluation. Our experiments achieve outstanding classification accuracy compared with traditional methods.

This paper focuses on dimension reduction (DR) technique for hyperspectral image (HSI). In this paper, we proposed a superpixel-based linear discriminant analysis (SP-LDA) dimension reduction method for HSI classification. Pixels within a local spatial neighborhood are expected to have similar spectral curves and share the same class label. To fully exploit the spatial structure, superpixel segmentation is firstly introduced to generate the superpixel map, which can adaptively explore the neighborhood structure information. Moreover, we extend the SP-LDA algorithm by combining the extracted feature from spectral and spatial dimensions, which can fully exploit complementary and consistent information from both dimensions. The experimental results on two standard hyperspectral datasets confirm the superiority of the proposed algorithms.

This paper presents a new data fusion methodology named Fusion-FCN for the classification of multi-source remote sensing data using fully convolutional networks (FCNs). Three different types of data including LiDAR data, hyperspectral images and very high resolution images are utilized in the proposed framework. Considering the confusions between similar categories (e.g., road and highway), we further implement post-classification processing with the topological relationship among different objects based on the result yielded by the proposed Fusion-FCN. The proposed method achieved an overall accuracy of 80.78% and a kappa coefficient of 0.80, which ranked first in the 2018 IEEE GRSS Data Fusion Contest.

The insufficiency of training samples is really a great challenge for hyperspectral image (HSI) classification. Samples generation is a commonly used technique in deep learning based remote sensing field which can extend the training set. However, previous methods ignore the real distribution of the training samples in the feature space and thus can hardly ensure that the generated samples possess the same patterns with the real ones. In this paper, we propose a generative adversarial network based method (SpecGAN) to handle this problem. Different from traditional GAN framework where the generated samples have no categories, for the first time we take the label information into consideration for hyperspectral images. Feeding a random noise z and a class label vector y into the generator, we can get a spectral sample of the corresponding category. The experiments on the Pavia University data set demonstrate the potential of the proposed SpecGAN in spectral samples generation.

In this paper, a unified spatial-temporal-spectral framework of missing information reconstruction in remote sensing images is proposed. Based on an end-to-end non-linear mapping structure, the proposed method employs a unified deep convolutional neural network combined with joint spatial-temporal-spectral supplementary information. It should be noted that the proposed model can use multi-source data (spatial, spectral, and temporal) as the input of the unified framework. The results of real-data experiments demonstrate that the proposed model exhibits high effectiveness in missing information reconstruction tasks like dead lines in Aqua MODIS band 6, Landsat ETM+ SLC-off and thick cloud removal.

In this letter, permutation preference is used to represent the data points for the linkage clustering to segment the tracking points belonging to different motions. In order to exclude the impact of outliers, an alternate sampling and clustering strategy is performed, that iteratively alternates between sampling the hypotheses within the clusters and clustering the points with the permutation preference. As a result, points with similar permutation preferences are sampled as the hypotheses, and outliers are effectively excluded, thus, making the preferences more distinguishable and improving the clustering. The iterative interaction between sampling and clustering results in a good convergent result. The proposed method obtains robust segmentation results with the Hopkins 155 dataset, which are better than the results obtained by the state-of-the-art methods.

Many hyperspectral anomaly detectors are designed based on the traditional Mahalanobis distance-based RX algorithm, which is usually considered as an inverse operation of the principal components analysis. Such detectors include the uniform target detector (UTD) algorithm, RX-UTD algorithm, and so on. However, the possibility of background statistical contamination caused by anomalies still exists. In order to alleviate this problem, in this paper, we propose a spectral matched filter (background-anomaly component projection and separation optimized filter) to minimize the average output energy of separate image components and the output values of the weighted background regular term for hyperspectral image anomaly detection, which could strengthen the separation between anomalies and backgrounds. By calculating the optimal solution to the background-anomaly component projection and separation function, we obtain the optimal projection, where we can effectively suppress the background while highlighting the anomalies. Proposed algorithm has the following research advantages: 1) it creates a novel collaborative component projection and robust background optimization function to separate the background and anomalies and 2) it analyzes the intrinsic statistical distribution of pixels and applies appropriate iterative shrinkage-thresholding algorithm to solve the ℓ1-min problem. Experiments were conducted on three real hyperspectral data sets. The detection results demonstrate that the proposed algorithm is superior to other state-of-the-art anomaly detection algorithms.

High spatial resolution (HSR) remote sensing images provide detailed geometric information about land cover. As a result, it is possible to detect more subtle changes with the help of HSR images. However, due to the increased spatial resolution and the limited spectral information, it is difficult to identify the real changes only through the spectral feature of the image. To fully explore the spectral–spatial information and improve the change detection performance for HSR images, this paper proposes the hybrid conditional random field (HCRF) model, which combines the traditional random field method with an object-based technique. In the proposed method, the spectral discriminative information of a single pixel is extracted by the unary potential, which is modeled using a soft clustering method to make an initial separation of changed and unchanged pixels. The pairwise potential then considers the contextual information of adjacent pixels to favor spatial smoothing. An object term is also introduced in the HCRF model to keep the homogeneity of changed objects. By the use of these approaches, the oversmoothing problem of the random field-based methods and the detection error caused by the segmentation strategy in the object-based methods can be relieved. The proposed method was tested on three HSR image data sets and outperformed the compared state-of-the-art techniques.

The hyperspectral subpixel mapping (SPM) technique can generate a land-cover map at the subpixel scale by modeling the relationship between the abundance map and the spatial distribution image of the subpixels. However, this is an inverse ill-posed problem. The most widely used way to resolve the problem is to introduce additional information as a regularization term and acquire the unique optimal solution. However, the regularization parameter either needs to be determined manually or it cannot be determined in a fully adaptive manner. Thus, in this paper, the multiobjective subpixel land-cover mapping (MOSM) framework for hyperspectral remote sensing imagery is proposed, in which the two function terms [the fidelity term and the prior term (i.e., the regularization term)] can be optimized simultaneously, and there is no need to determine the regularization parameter explicitly. In order to achieve this goal, two strategies are designed in MOSM: 1) a high-resolution distribution image-based individual encoding strategy is designed in order to calculate the prior term accurately and 2) a subfitness-based individual comparison strategy is designed in order to generate subpixel land-cover mapping solutions with a high quality to update the population. Four data sets (one simulated, two synthetic, and one real hyperspectral image) were used to test the proposed method. The experimental results show that MOSM can perform better than the other subpixel land-cover mapping methods, demonstrating the effectiveness of MOSM in balancing the fidelity term and prior term in the SPM model.

This paper focuses on the endmember extraction (EE) technique for analyzing hyperspectral images. We first prove that the reconstruction errors (REs) and abundance anomalies (AAs) (abundances that fail to satisfy the abundance constraints) are effective in extracting undetected endmembers. Then, according to the spatial continuity of the endmember objects and differing from noise or outliers with a sparse distribution, the endmembers are assumed to be located at some salient areas in the RE and AA maps. A novel EE algorithm termed saliency-based endmember detection (SED) is proposed, where the visual saliency model is introduced to explore and analyze the spatial information that is contained in the AA and RE maps. Specifically, the AA and RE maps are regarded as the visual inputs, whereas the endmembers are treated as the visual stimuli. In SED, we assume that the pure pixel assumption holds. Based on the characteristics of the human visual system, the proposed method can not only extract endmembers in homogenous areas, but it can also highlight the small targets whose abundances may be spatially varied. In addition, since the spatial information is exploited in the reconstruction, the capability of the endmembers to represent the hyperspectral scene is automatically considered in the process of EE, and the detected endmembers are both accurate and reliable. The experimental results obtained on both simulated and real hyperspectral data confirm the merits and viability of the proposed algorithm.

In this paper, a new joint spectral-spatial subpixel mapping model is proposed for hyperspectral remotely sensed imagery. Conventional approaches generally use an intermediate step based on the derivation of fractional abundance maps obtained after a spectral unmixing process, and thus the rich spectral information contained in the original hyperspectral data set may not be utilized fully. In this paper, a concept of subpixel abundance map, which calculates the abundance fraction of each subpixel to belong to a given class, was introduced. This allows us to directly connect the original (coarser) hyperspectral image with the final subpixel result. Furthermore, the proposed approach incorporates the spectral information contained in the original hyperspectral imagery and the concept of spatial dependence to generate a final subpixel mapping result. The proposed approach has been experimentally evaluated using both synthetic and real hyperspectral images, and the obtained results demonstrate that the method achieves better results when compared to other seven subpixel mapping methods. The numerical comparisons are based on different indexes such as the overall accuracy and the CPU time. Moreover, the obtained results are statistically significant at 95% confidence.

In this paper, we propose a spectral–spatial unified network (SSUN) with an end-to-end architecture for the hyperspectral image (HSI) classification. Different from traditional spectral–spatial classification frameworks where the spectral feature extraction (FE), spatial FE, and classifier training are separated, these processes are integrated into a unified network in our model. In this way, both FE and classifier training will share a uniform objective function and all the parameters in the network can be optimized at the same time. In the implementation of the SSUN, we propose a band grouping-based long short-term memory model and a multiscale convolutional neural network as the spectral and spatial feature extractors, respectively. In the experiments, three benchmark HSIs are utilized to evaluate the performance of the proposed method. The experimental results demonstrate that the SSUN can yield a competitive performance compared with existing methods.

High spatial resolution (HSR) imagery scene classification has been the subject of increased interest in recent years, and has great potential for many applications, such as urban functional analysis. Rooted in natural information processing, the use of the probabilistic topic model (PTM) to capture latent topics to represent HSR images has been an effective way to bridge the semantic gap. However, how to effectively discover discriminative information to recognize the HSR scenes is a challenging task. In this paper, the sparse homogeneous-heterogeneous topic feature model (SHHTFM) is proposed for HSR image scene classification. Differing from the conventional PTM-based scene classification methods, which utilize only heterogeneous features, SHHTFM explores the effect of the homogeneous information. Based on the union of uniform grid sampling and simple linear iterative clustering superpixel sampling, SHHTFM exploits both the heterogeneous and homogeneous information. After separately mining different types of low-level features and latent topics, the sparse topic inference procedure of SHHTFM further improves the fusion of the sparse heterogeneous and homogeneous topics. In addition, multisource geographical data are effectively integrated, where the water and vegetation boundaries define a more accurate way to restrict the boundaries of different scenes, and are then combined with the road network data to further improve the scene annotation performance. This provides more reliable and applicable results for us to better understand the complex scenes. The experimental results obtained with two HSR image classification data sets and an HSR image annotation data set demonstrate that the proposed SHHTFM framework can solve the scene classification problem, with a high classification accuracy as well as a high time efficiency.

High spatial resolution (HSR) imagery scene classification, which involves labeling an HSR image with a specific semantic class according to the geographical properties, has received increased attention, and many algorithms have been proposed for this task. The employment of the probabilistic topic model to acquire latent topics and the convolutional neural networks (CNNs) to capture deep features for representing HSR images has been an effective ways to bridge the semantic gap. However, the midlevel topic features are usually local and significant, whereas the high-level deep features convey more global and detailed information. In this paper, to discover more discriminative semantics for HSR images, the adaptive deep sparse semantic modeling (ADSSM) framework combining sparse topics and deep features is proposed for HSR image scene classification. In ADSSM, the fully sparse topic model and a CNN are integrated. To exploit the multilevel semantics for HSR scenes, the sparse topic features and deep features are effectively fused at the semantic level. Based on the difference between the sparse topic features and the deep features, an adaptive feature normalization strategy is proposed to improve the fusion of the different features. The experimental results obtained with four HSR image classification data sets confirm that the proposed method significantly improves the performance when compared with the other state-of-the-art methods.

Blind source separation (BSS) aims to discover the underlying source signals from a set of linear mixture signals without any prior information of the mixing system, which is a fundamental problem in signal and image processing field. Most of the state-of-the-art algorithms have independently handled the decompositions of mixture signals. In this paper, we propose a new algorithm named multi-task sparse model to solve the BSS problem. Source signals are characterized via sparse techniques. Meanwhile, we regard the decomposition of each mixture signal as a task and employ the idea of multi-task learning to discover connections between tasks for the accuracy improvement of the source signal separation. Theoretical analyses on the optimization convergence and sample complexity of the proposed algorithm are provided. Experimental results based on extensive synthetic and real-world data demonstrate the necessity of exploiting connections between mixture signals and the effectiveness of the proposed algorithm.

Junctions play an important role in characterizing local geometrical structures of images, and the detection of which is a longstanding but challenging task. Existing junction detectors usually focus on identifying the location and orientations of junction branches while ignoring their scales, which, however, contain rich geometries of images. This paper presents a novel approach for junction detection and characterization, which especially exploits the locally anisotropic geometries of a junction and estimates its scales by relying on an a-contrario model. The output junctions are with anisotropic scales, saying that a scale parameter is associated with each branch of a junction and are thus named as anisotropic-scale junctions (ASJs). We then apply the new detected ASJs for matching indoor images, where there are dramatic changes of viewpoints and the detected local visual features, e.g., key-points, are usually insufficient and lack distinctive ability. We propose to use the anisotropic geometries of our junctions to improve the matching precision of indoor images. The matching results on sets of indoor images demonstrate that our approach achieves the state-of-the-art performance on indoor image matching.

Sparse unmixing has been successfully applied to hyperspectral remote sensing imagery based on the assumption that the observed image signatures can be expressed in a linear sparse regression with a large standard spectral library. Prior work for sparse unmixing usually utilizes L1norm or Laplacian distribution to promote sparsity. Unfortunately, the L1norm is not differentiable, which may lead to unstable results. In this paper, we adopt Bregman divergence for sparse unmixing, which is a differentiable, smoother prior. Based on the Maximum A Posterior (MAP) estimation, the proposed method has achieved sparse, stable and precise fractional abundances. The experimental results both simulated dataset and the real hyperspectral image demonstrate the effectiveness of the proposed differentiable sparse unmixing algorithm.

Geospatial object detection from high spatial resolution (HSR) imagery is significant and challenging for further analyzing the object-related information in various civil and military applications. Traditional object detection methods based on the handcrafted features are limited by their efficiency in describing the multi-class objects from large-swath and complex-context HSR imagery. Although convolutional neural network (CNN) can extract the features automatically, the feature extraction and detection stages are still separate and time-consuming. In addition, manual labelling information is limited and an efficient real-time one-stage detection framework for HSR imagery is scare. In this paper, a robust pre-trained efficient multi-class geospatial object detection framework - pre-trained Faster R-CNN sharing the convolutional features between region proposal stage and detection stage is proposed for HSR imagery. Extensive experiments and evaluations on a ten-class object detection dataset are conducted for the proposed method.

Mixed pixel is a ubiquitous phenomenon in remotely sensed imagery, especially in moderate and low spatial resolution imagery, which compromise the hard land cover classification since the dominant class will shadow the information of other vulnerable classes, bringing trouble to imagery interpretation. Since the past decades, sub-pixel mapping (SPM) approaches were developed to deal with the mixture problem, on the basis of soft classification, to retrieval the pure components and its geospatial distribution within mixed pixels. Recently, SPM integrated with auxiliary information is gradually been a state-of-the-art method for mixed pixel problem, and has been proved effectively. However, few works has been dedicated to explore the geostatistic inter-correlation between spatial and temporal among the time sequences imageries. In this paper, a novel SPM algorithm based on swarm intelligence theory, considering spatiotemporal geographical attraction among multi-temporal imageries, called spatiotemporal attraction based sub-pixel evolution mapping (SASEM), is proposed for remote sensing imagery, Experiments were carried out to verify the proposed algorithm, and the result illustrate that the proposed algorithm outperform the traditional SPM, achieving a fine spatial resolution thematic map for further applications.

Soil Moisture Active Passive (SMAP) satellite mission, launched on Jan. 31, 2015, can provide a 9 km soil moisture product globally by merging passive and active observations. However, the radar sensor of SMAP was failed since Jul. 7, 2015 and SMAP 9 km SM product (SMAP_AP) is only available for 85 days. To ameliorate the vacancy of SMAP_AP, a spatio-temporal fusion model STNLFFM combined with the SMAP 36 km soil moisture product (SMAP_P) is utilized to generate 9 km soil moisture SM product (SMAP_F). Generation of SMAP_F was implemented over one year from Apr. 13, 2015 to Apr. 12, 2016 in the paper. Then SMAP_F was evaluated by SMAP_AP and in-situ soil moisture from international soil moisture network. It is revealed that the STNLFFM can be taken as an effective method for SMAP 9 km soil moisture generation.

Endmember extraction (EE) is one of the most important issues in hyperspectral mixture analysis, and it is also one of the most challenging tasks due to the intrinsic complexity of remote sensing images and the lack of priori knowledge. In recent years, a number of EE methods have been developed, and several different optimization objectives have been proposed from different perspectives. In all of these methods, there is only one objective function to be optimized, which represents a specific characteristic of the remote sensing image. However, one single-objective function may not provide satisfactory results because of the complexity of remote sensing images. In this paper, a multiobjective discrete particle swarm optimization algorithm (MODPSO) is utilized to tackle the problem of EE, where two objective functions, namely, volume maximization and root-mean-square error (RMSE) minimization are simultaneously optimized. The result set of Pareto-optimal solutions contains a number of non-dominated solutions, from which the user can judge relatively and pick up the most promising one according to the problem requirements. Experiments on two real hyperspectral images were conducted to evaluate the proposed MODPSO algorithm, which confirmed the effectiveness of the proposed algorithm.

In this paper, a structural conditional random field framework (SCRF) is proposed to detect the detailed change information from high spatial resolution (HSR) remote sensing imagery. Traditional random field based methods encounter the over-smoothing problem when deal with HSR images and the boundary of changed objects cannot be preserved well. To solve this problem, in SCRF, fuzzy c means (FCM) is used to model the unary potential while avoiding the independent assumption. Pairwise potentials with different shapes are selected as the structural set to model the spatial features of land cover such as buildings and roads. Based on SCRF, a set of change belief maps are generated to describe the observed image from different aspects. An object based fusion strategy is then followed to combine the belief maps to get the refined result. The results of the proposed method on two HSR data sets outperform some state-of-art algorithms.

This paper focuses on the spectral unmixing technique for analyzing hyperspectral image (HSI). In this paper, we first prove that the reconstruction errors and the abundance anomalies (AAs, abundances that are negative or greater than one) are effective in measuring the purity of pixels. Then, due to the continuity of the objects in the space, the endmembers are assumed to be located at some noticeable areas in residual and AA maps. A saliency-based endmember detection (SED) algorithm which aims at iteratively extracting endmembers from the residual and AA maps is proposed, where the visual attention mechanism is developed to understand and analyze the spatial pattern of endmembers. In addition, when searching for new endmembers, the spectral properties are also utilized to promote the robustness of the proposed method. The experimental results on both simulated data and real hyperspectral data illustrate the merits and viability of the proposed algorithm.

Pan-sharpening is a fundamental and significant task in the field of remote sensed imagery fusion, which demands fusion of panchromatic and multi-spectral images with the rich information accurately preserved in both spatial and spectral domains. In this paper, to overcome the drawbacks of traditional pan-sharpening methodologies, we employed the advanced concept of deep learning to propose a Multi-Scale-and-Depth Convolutional Neural Network (MSDCNN) as an end-to-end pan-sharpening model. By the results of a large number of quantitative and visual assessments, the qualities of images fused by the proposed network have been confirmed superior to compared state-of-the-art methods.

Missing data reconstruction (e.g., dead pixel repair and cloud removing) in remote sensing images is a very important problem for the subsequent image analysis. It is well-known that missing data reconstruction is an ill-posed problem. In remote sensing images, there is a strong correlation in spectral frequencies or in temporal frames, and also there are a lot of self-similarity patterns in spatial domain. We can make use of these properties to derive low rank matrices according to their spectral, temporal and spatial dimensions. In this paper, we propose a tensor completion model based on these low rank matrices to deal with missing data reconstruction problem. We also present a weighting method for spectral, temporal and spatial dimensions and for their distribution of singular values. Our experimental results demonstrate that the weighting method can recover remote images very well. In particular, we show the effectiveness of the proposed method for both simulated and real data sets, and the performance of the proposed in terms of visual and quantitative measures is better than those of the other testing methods.

In this paper, we propose total variation regularized collaborative representation clustering with a locally adaptive dictionary for hyperspectral remote sensing imagery. With regard to the high redundancy of the global dictionary and the interference of unrelated dictionary atoms in the representation process, the collaborative representation clustering model with a locally adaptive dictionary is introduced to more precisely represent each pixel only with highly correlated atoms. In addition, total variation regularization is integrated to better account for the rich spatial contextual information. The extensive experimental results clearly illustrate the superiority of the proposed algorithm.

In recent years, the science of hyperspectral remote sensing has huge development in virtue of the integration of low-cost lightweight hyperspectral sensors and unmanned aerial vehicles (UAVs). As an alternative of manned aircraft, UAV has some unique advantages enabling the researchers acquire the hyperspectral images of their interest area flexibly and promptly. This review focuses on the recent developments of UAV borne hyperspectral remote sensing system, and gives an overview of the corresponding platforms, sensors, data acquisition, processing and current applications. Future challenges and research directions for UAV borne hyperspectral data are also addressed.

How can a doctor diagnose new diseases with little historical knowledge, which are emerging over time? Active learning is a promising way to address the problem by querying the most informative samples. Since the diagnosed cases for new disease are very limited, gleaning knowledge from other domains (classical prescriptions) to prevent the bias of active leaning would be vital for accurate diagnosis. In this paper, a framework that attempts to glean knowledge from multiple domains for active learning by querying the most uncertain and representative samples from the target domain and calculating the importance weights for re-weighting the source data in a single unified formulation is proposed. The weights are optimized by both a supervised classifier and distribution matching between the source domain and target domain with maximum mean discrepancy. Besides, a multiple domains active learning method is designed based on the proposed framework as an example. The proposed method is verified with newsgroups and handwritten digits data recognition tasks, where it outperforms the state-of-the-art methods.

Manifold learning algorithms such as the isometric mapping (ISOMAP) algorithm have been widely used in the analysis of hyperspectral images (HSIs), for both visualization and dimension reduction. As advanced versions of the traditional linear projection techniques, the manifold learning algorithms find the low-dimensional feature representation by nonlinear mapping, which can better preserve the local structure of the original data and thus benefit the data analysis. However, the high computational complexity of the manifold learning algorithms hinders their application in HSI processing. Although there are a few parallel implementations of manifold learning approaches that are available in the remote sensing community, they have not been designed to accelerate the eigen-decomposition process, which is actually the most time-consuming part of the manifold learning algorithms. In this letter, as a case study, we discuss the graphics processing unit parallel implementation of the ISOMAP algorithm. In particular, we focus on the eigen-decomposition process and verify the applicability of the proposed method by validating the embedding vectors and the subsequent classification accuracies. The experimental results obtained on different HSI data sets show an excellent speedup performance and consistent classification accuracy compared with the serial implementation.

High-level feature extraction and hierarchical feature representation of image objects with a convolutional neural network (CNN) can overcome the limitations of the traditional building detection models using middle/low-level features extracted from a complex background. Aiming at the drawbacks of manual village location, high cost, and the limited accuracy of building detection in the existing rural building detection models, a two-stage CNN model is proposed in this letter to detect rural buildings in high-resolution imagery. Simulating the hierarchical processing mechanism of human vision, the proposed model is constructed with two CNNs, whose architectures can automatically locate villages and efficiently detect buildings, respectively. This two-stage CNN model effectively reduces the complexity of the background and improves the efficiency of rural building detection. The experiments showed that the proposed model could automatically locate all the villages in the two study areas, achieving a building detection accuracy of 88%. Compared with the existing models, the proposed model was proved to be effective in detecting buildings in rural areas with a complex background.

In the field of multispectral (MS) and panchromatic image fusion (pansharpening), the impressive effectiveness of deep neural networks has recently been employed to overcome the drawbacks of the traditional linear models and boost the fusion accuracy. However, the existing methods are mainly based on simple and flat networks with relatively shallow architectures, which severely limits their performance. In this letter, the concept of residual learning is introduced to form a very deep convolutional neural network to make the full use of the high nonlinearity of the deep learning models. Through both quantitative and visual assessments on a large number of high-quality MS images from various sources, it is confirmed that the proposed model is superior to all the mainstream algorithms included in the comparison, and achieves the highest spatial-spectral unified accuracy.

Robust techniques such as sparse subspace clustering (SSC) have been recently developed for hyperspectral images (HSIs) based on the assumption that pixels belonging to the same land-cover class approximately lie in the same subspace. In order to account for the spatial information contained in HSIs, SSC models incorporating spatial information have become very popular. However, such models are often based on a local averaging constraint, which does not allow for a detailed exploration of the spatial information, thus limiting their discriminative capability and preventing the spatial homogeneity of the clustering results. To address these relevant issues, in this letter, we develop a new and effective ℓ2-norm regularized SSC algorithm which adds a four-neighborhood ℓ2-norm regularizer into the classical SSC model, thus taking full advantage of the spatial-spectral information contained in HSIs. The experimental results confirm the potential of including the spatial information (through the newly added ℓ2-norm regularization term) in the SSC framework, which leads to a significant improvement in the clustering accuracy of SSC when applied to HSIs.

As sparse representation gradually obtains better and better results in the analysis of hyperspectral imagery and sparsity-based algorithms are becoming more and more popular, especially in target detection. However, these methods mostly assume an absolute equal contribution by all neighboring pixels while detecting the central pixel. There is no doubt that this approach is unsuitable for pixels located in heterogeneous areas. In this letter, to address this problem, spatially adaptive sparse representation for target detection in hyperspectral images (HSIs) is proposed. Neighboring spatial information is utilized by considering the different contributions of the distinct neighborhood pixels. The different weights are determined according to the similarity between the neighboring pixels and the central test pixel. The proposed algorithm was tested on two HSIs and demonstrated outstanding detection performance when compared with other commonly used detectors.

Filling missing information or removing special objects is often required in the applications of high spatial resolution images. A novel single-image reconstruction method is presented in this letter to solve this task, without the use of any complementary data. First, the spatial pattern of the image is obtained by the statistics of similar patch offsets in the known regions, which provide reliable information for reconstructing the image. The missing regions are then filled by combining a series of shifted pixels via global optimization. The proposed method was tested on a cloudy image for cloud removal and on a public image for military object concealment. The experimental results show that the proposed method can produce visually convincing and coherent reconstructed images, and the accuracy of the reconstruction is better than the existing noncomplementation methods.

How can we find a general way to choose the most suitable samples for training a classifier? Even with very limited prior information? Active learning, which can be regarded as an iterative optimization procedure, plays a key role to construct a refined training set to improve the classification performance in a variety of applications, such as text analysis, image recognition, social network modeling, etc. Although combining representativeness and informativeness of samples has been proven promising for active sampling, state-of-the-art methods perform well under certain data structures. Then can we find a way to fuse the two active sampling criteria without any assumption on data? This paper proposes a general active learning framework that effectively fuses the two criteria. Inspired by a two-sample discrepancy problem, triple measures are elaborately designed to guarantee that the query samples not only possess the representativeness of the unlabeled data but also reveal the diversity of the labeled data. Any appropriate similarity measure can be employed to construct the triple measures. Meanwhile, an uncertain measure is leveraged to generate the informativeness criterion, which can be carried out in different ways. Rooted in this framework, a practical active learning algorithm is proposed, which exploits a radial basis function together with the estimated probabilities to construct the triple measures and a modified best-versus-second-best strategy to construct the uncertain measure, respectively. Experimental results on benchmark datasets demonstrate that our algorithm consistently achieves superior performance over the state-of-the-art active learning algorithms.

Deep networks have achieved excellent performance in learning representation from visual data. However, the supervised deep models like convolutional neural network require large quantities of labeled data, which are very expensive to obtain. To solve this problem, this paper proposes an unsupervised deep network, called the stacked convolutional denoising auto-encoders, which can map images to hierarchical representations without any label information. The network, optimized by layer-wise training, is constructed by stacking layers of denoising auto-encoders in a convolutional way. In each layer, high dimensional feature maps are generated by convolving features of the lower layer with kernels learned by a denoising auto-encoder. The auto-encoder is trained on patches extracted from feature maps in the lower layer to learn robust feature detectors. To better train the large network, a layer-wise whitening technique is introduced into the model. Before each convolutional layer, a whitening layer is embedded to sphere the input data. By layers of mapping, raw images are transformed into high-level feature representations which would boost the performance of the subsequent support vector machine classifier. The proposed algorithm is evaluated by extensive experimentations and demonstrates superior classification performance to state-of-the-art unsupervised networks.

The tradeoff in remote sensing instruments that balances the spatial resolution and temporal frequency limits our capacity to monitor spatial and temporal dynamics effectively. The spatiotemporal data fusion technique is considered as a cost-effective way to obtain remote sensing data with both high spatial resolution and high temporal frequency, by blending observations from multiple sensors with different advantages or characteristics. In this paper, we develop the spatial and temporal nonlocal filter-based fusion model (STNLFFM) to enhance the prediction capacity and accuracy, especially for complex changed landscapes. The STNLFFM method provides a new transformation relationship between the fine-resolution reflectance images acquired from the same sensor at different dates with the help of coarse-resolution reflectance data, and makes full use of the high degree of spatiotemporal redundancy in the remote sensing image sequence to produce the final prediction. The proposed method was tested over both the Coleambally Irrigation Area study site and the Lower Gwydir Catchment study site. The results show that the proposed method can provide a more accurate and robust prediction, especially for heterogeneous landscapes and temporally dynamic areas.

The high-dimensional data space of hyperspectral images (HSIs) often result in ill-conditioned formulations, which finally leads to many of the high-dimensional feature spaces being empty and the useful data existing primarily in a subspace. To avoid these problems, we use distance metric learning for dimensionality reduction. The goal of distance metric learning is to incorporate abundant discriminative information by reducing the dimensionality of the data. Considering that global metric learning is not appropriate for all training samples, this paper proposes an ensemble discriminative local metric learning (EDLML) algorithm for HSI analysis. The EDLML algorithm learns robust local metrics from both the training samples and the relative neighborhood of them and considers the different local discriminative distance metrics by dealing with the data region by region. It aims to learn a subspace to keep all the samples in the same class are as near as possible, while those from different classes are separated. The learned local metrics are then used to build an ensemble metric. Experiments on a number of different hyperspectral data sets confirm the effectiveness of the proposed EDLML algorithm compared with that of the other dimension reduction methods.

Blind hyperspectral unmixing (HU), which includes the estimation of endmembers and their corresponding fractional abundances, is an important task for hyperspectral analysis. Recently, nonnegative matrix factorization (NMF) and its extensions have been widely used in HU. Unfortunately, most of the NMF-based methods can easily lead to an unsuitable solution, due to the nonconvexity of the NMF model and the influence of noise. To overcome this limitation, we make the best use of the structure of the abundance maps, and propose a new blind HU method named total variation regularized reweighted sparse NMF (TV-RSNMF). First, the abundance matrix is assumed to be sparse, and a weighted sparse regularizer is incorporated into the NMF model. The weights of the weighted sparse regularizer are adaptively updated related to the abundance matrix. Second, the abundance map corresponding to a single fixed endmember should be piecewise smooth. Therefore, the TV regularizer is adopted to capture the piecewise smooth structure of each abundance map. In our multiplicative iterative solution to the proposed TV-RSNMF model, the TV regularizer can be regarded as an abundance map denoising procedure, which improves the robustness of TV-RSNMF to noise. A number of experiments were conducted in both simulated and real-data conditions to illustrate the advantage of the proposed TV-RSNMF method for blind HU.

A sparse and low-rank near-isometric linear embedding (SLRNILE) method has been proposed to make dimensionality reduction and extract proper features for hyperspectral imagery (HSI) classification. The SLRNILE stands on the theory of the John-Lindenstrauss lemma, and tries to estimate a sparse and low-rank projection matrix that satisfies the restricted isometric property (RIP) condition on all secants of the HSI data. The RIP condition guarantees that the desired linear mapping near-isometrically preserves nearest neighbor points of all HSI pixels. Seeking the desired mapping is then modeled into minimizing a Lagrange multipliers formulation. The alternating direction method of multipliers framework is utilized to solve the above convex program, and column generation techniques are adopted to alleviate the computation memory burden during the optimization procedure. Five experiments on three widely used HSI data sets are designed to completely test the performance of SLRNILE, and experimental results are compared against those of six state-of-the-art feature extraction methods, including principal component analysis, Laplacian eigenmaps, locality preserving projections, neighborhood preserving embedding, sparse nonnegative matrix underapproximation, and random projections. The results show that SLRNILE performs best among all the seven methods, and its computational time is longest of all but still bearable for regular users. Therefore, the SLRNILE can be a good choice for feature extraction in HSI classification.

As the acquisition of very high resolution (VHR) satellite images becomes easier owing to technological advancements, ever more stringent requirements are being imposed on automatic image interpretation. Moreover, per-pixel classification has become the focus of research interests in this regard. However, the efficient and effective processing and the interpretation of VHR satellite images remain a critical task. Convolutional neural networks (CNNs) have recently been applied to VHR satellite images with considerable success. However, the prevalent CNN models accept input data of fixed sizes and train the classifier using features extracted directly from the convolutional stages or the fully connected layers, which cannot yield pixel-to-pixel classifications. Moreover, training a CNN model requires large amounts of labeled reference data. These are challenging to obtain because per-pixel labeled VHR satellite images are not open access. In this paper, we propose a framework called the unsupervised-restricted deconvolutional neural network (URDNN). It can solve these problems by learning an end-to-end and pixel-to-pixel classification and handling a VHR classification using a fully convolutional network and a small number of labeled pixels. In URDNN, supervised learning is always under the restriction of unsupervised learning, which serves to constrain and aid supervised training in learning more generalized and abstract feature. To some degree, it will try to reduce the problems of overfitting and undertraining, which arise from the scarcity of labeled training data, and to gain better classification results using fewer training samples. It improves the generality of the classification model. We tested the proposed URDNN on images from the Geoeye and Quickbird sensors and obtained satisfactory results with the highest overall accuracy (OA) achieved as 0.977 and 0.989, respectively. Experiments showed that the combined effects of additional kernels and stages may have produced better results, and two-stage URDNN consistently produced a more stable result. We compared URDNN with four methods and found that with a small ratio of selected labeled data items, it yielded the highest and most stable results, whereas the accuracy values of the other methods quickly decreased. For some categories with fewer training pixels, accuracy for categories from other methods was considerably worse than that in URDNN, with the largest difference reaching almost 10%. Hence, the proposed URDNN can successfully handle the VHR image classification using a small number of labeled pixels. Furthermore, it is more effective than state-of-the-art methods.

Less training samples are a challenging problem in hyperspectral image classification. Active learning and semisupervised learning are two promising techniques to address the problem. Active learning solves the problem by improving the quality of the training samples, while semisupervised learning solves the problem by increasing the quantity of the training samples. However, they pay too much attention to the discriminative information in the unlabeled data, leading to information bias to train supervised models, and much more effort to label samples. Therefore, a method to discover representativeness and discriminativeness by semisupervised active learning is proposed. It takes advantages of both active learning and semisupervised learning. The representativeness and discriminativeness are discovered with a labeling process based on a supervised clustering technique and classification results. Specifically, the supervised clustering results can discover important structural information in the unlabeled data, and the classification results are also highly confidential in the active-learning process. With these clustering results and classification results, we can assign pseudolabels to the unlabeled data. Meanwhile, the unlabeled samples that cannot be assigned with pseudolabels with high confidence at each iteration are regarded as candidates in active learning. The methodology is validated on four hyperspectral data sets. Significant improvements in classification accuracy are achieved by the proposed method with respect to the state-of-the-art methods.

In recent years, blind source separation (BSS) has received much attention in the hyperspectral unmixing field due to the fact that it allows the simultaneous estimation of both endmembers and fractional abundances. Although great performances can be obtained by the BSS-based unmixing methods, the decomposition results are still unstable and sensitive to noise. Motivated by the first law of geography, some recent studies have revealed that spatial information can lead to an improvement in the decomposition stability. In this paper, the group-structured prior information of hyperspectral images is incorporated into the nonnegative matrix factorization optimization, where the data are organized into spatial groups. Pixels within a local spatial group are expected to share the same sparse structure in the low-rank matrix (abundance). To fully exploit the group structure, image segmentation is introduced to generate the spatial groups. Instead of a predefined group with a regular shape (e.g., a cross or a square window), the spatial groups are adaptively represented by superpixels. Moreover, the spatial group structure and sparsity of the abundance are integrated as a modified mixed-norm regularization to exploit the shared sparse pattern, and to avoid the loss of spatial details within a spatial group. The experimental results obtained with both simulated and real hyperspectral data confirm the high efficiency and precision of the proposed algorithm.

Scene change detection between multitemporal image scenes can be used to interpret the variation of regional land use, and has significant potential in the application of urban development monitoring at the semantic level. The traditional methods directly comparing the independent semantic classes neglect the temporal correlation, and thus suffer from accumulated classification errors. In this paper, we propose a novel scene change detection method via kernel slow feature analysis (KSFA) and postclassification fusion, which integrates independent scene classification with scene change detection to accurately determine scene changes and identify the “from-to” transition type. After representation with the bag-of-visual-words model, KSFA is proposed to extract the nonlinear temporally invariant features, to better measure the change probability between corresponding multitemporal image scenes. Two postclassification fusion methods, which are based on Bayesian theory and predefined rules, respectively, are then employed to identify the optimal coupled class combinations of multitemporal scene pairs. Furthermore, in addition to identifying semantic changes, the proposed method can also improve the performance of scene classification, since the unchanged scenes are more likely to belong to the same class. Two experiments with high-resolution remote sensing image scene data sets confirm that the proposed method can increase the accuracy of scene change detection, scene transition identification, and scene classification.

Aerial scene classification, which aims to automatically label an aerial image with a specific semantic category, is a fundamental problem for understanding high-resolution remote sensing imagery. In recent years, it has become an active task in the remote sensing area, and numerous algorithms have been proposed for this task, including many machine learning and data-driven approaches. However, the existing data sets for aerial scene classification, such as UC-Merced data set and WHU-RS19, contain relatively small sizes, and the results on them are already saturated. This largely limits the development of scene classification algorithms. This paper describes the Aerial Image data set (AID): a large-scale data set for aerial scene classification. The goal of AID is to advance the state of the arts in scene classification of remote sensing images. For creating AID, we collect and annotate more than 10000 aerial scene images. In addition, a comprehensive review of the existing aerial scene classification techniques as well as recent widely used deep learning methods is given. Finally, we provide a performance analysis of typical aerial scene classification and deep learning approaches on AID, which can be served as the baseline results on this benchmark.

With the high spectral resolution, hyperspectral images (HSIs) provide great potential for target detection, which is playing an increasingly important role in HSI processing. Many target detection methods uniformly utilize all the spectral information or employ reduced spectral information to distinguish the targets and background. Simultaneously reducing spectral redundancy and preserving the discriminative information is a challenging problem in hyperspectral target detection. The multitask learning (MTL) technique may have the potential to solve the above problem, since it can explore the redundancy knowledge to construct multiple sub-HSIs and integrate them without any information loss. This paper proposes the joint sparse representation and MTL (JSR-MTL) method for hyperspectral target detection. This approach: 1) explores the HSIs similarity by a band cross-grouping strategy to construct multiple sub-HSIs; 2) takes full advantage of the MTL technique to integrate the sparse representation models for the multiple related sub-HSIs; and 3) applies the total reconstruction error difference accumulated over all the tasks to detect the targets. Extensive experiments were carried out on three HSIs, and it was founded that JSR-MTL generally shows a better detection performance than the other target detection methods.

Anomaly detection has become an important topic in hyperspectral imagery (HSI) analysis over the last 20 years. HSIs usually possess complexly cluttered spectral signals due to the complicated conditions of the land-cover distribution. This in turn makes it difficult to obtain an accurate background estimation to distinguish the anomaly targets. The sparse learning technique provides a way to obtain an implicit background representation with the learned dictionary and corresponding sparse codes. In this paper, we explore the background/anomaly information content for each atom of the learned dictionary, from an analysis based on the frequency of the dictionary atoms for HSI reconstruction. From this perspective, we propose a novel sparsity score estimation framework for hyperspectral anomaly detection. First, an overcomplete dictionary and the corresponding sparse code matrix are obtained from the HSI. The frequency of each dictionary atom for reconstruction, which is also called the atom usage probability, is then estimated from the sparse code matrix. Finally, the estimated frequencies are transformed to the sparsity score for each pixel, which can be seen as the degree of “anomalousness.” In the proposed detection framework, two strategies are proposed to enhance the diversity between the background and anomaly information in the learned dictionary: 1) dictionary-based background feature transformation and 2) dictionary iterative reweighting. A series of real-world HSI data sets is utilized to evaluate the performance of the proposed framework. The experimental results show that the proposed framework achieves a superior performance compared to some of the state-of-the-art anomaly detection methods.

In high spatial resolution (HSR) imagery scene classification, it is a challenging task to recognize the high-level semantics from a large volume of complex HSR images. The probabilistic topic model (PTM), which focuses on modeling topics, has been proposed to bridge the so-called semantic gap. Conventional PTMs usually model the images with a dense semantic representation and, in general, one topic space is generated for all the different features. However, this approach fails to consider the sparsity of the semantic representation, the classification quality, as well as the time consumption. In this paper, to solve the above problems, a fully sparse semantic topic model (FSSTM) framework is proposed for HSR imagery scene classification. FSSTM, with an elaborately designed modeling procedure, is able to represent the image with sparse but representative semantics. Based on this framework, the topic weights of multiple features are exploited by solving a concave maximization problem, which improves the fusion of the discriminative semantic information at the topic level. Meanwhile, the sparsity and representativeness of the topics generated by FSSTM guarantee that the image is adaptive to the change of a topic number. FSSTM can consistently achieve a good performance with a limited number of training samples, and is robust for HSR image scene classification. The experimental results obtained with three different types of HSR image data sets confirm that the proposed algorithm is effective in improving the performance of scene classification, and is highly efficient in discovering the semantics of HSR images when compared with the state-of-the-art PTM methods.

Multi-label learning draws great interests in many real world applications. It is a highly costly task to assign many labels by the oracle for one instance. Meanwhile, it is also hard to build a good model without diagnosing discriminative labels. Can we reduce the label costs and improve the ability to train a good model for multi-label learning simultaneously? Active learning addresses the less training samples problem by querying the most valuable samples to achieve a better performance with little costs. In multi-label active learning, some researches have been done for querying the relevant labels with less training samples or querying all labels without diagnosing the discriminative information. They all cannot effectively handle the outlier labels for the measurement of uncertainty. Since maximum correntropy criterion (MCC) provides a robust analysis for outliers in many machine learning and data mining algorithms, in this paper, we derive a robust multi-label active learning algorithm based on an MCC by merging uncertainty and representativeness, and propose an efficient alternating optimization method to solve it. With MCC, our method can eliminate the influence of outlier labels that are not discriminative to measure the uncertainty. To make further improvement on the ability of information measurement, we merge uncertainty and representativeness with the prediction labels of unknown data. It cannot only enhance the uncertainty but also improve the similarity measurement of multi-label data with labels information. Experiments on benchmark multi-label data sets have shown a superior performance than the state-of-the-art methods.

Texture characterization is a key problem in image understanding and pattern recognition. In this paper, we present a flexible shape-based texture representation using shape co-occurrence patterns. More precisely, texture images are first represented by a tree of shapes, each of which is associated with several geometrical and radiometric attributes. Then, four typical kinds of shape co-occurrence patterns based on the hierarchical relationships among the shapes in the tree are learned as codewords. Three different coding methods are investigated for learning the codewords, which can be used to encode any given texture image into a descriptive vector. In contrast with existing works, the proposed approach not only inherits the shape-based method's strong ability to capture geometrical aspects of textures and high robustness to variations in imaging conditions but also provides a flexible way to consider shape relationships and to compute high-order statistics on the tree. To the best of our knowledge, this is the first time that co-occurrence patterns of explicit shapes have been used as a tool for texture analysis. Experiments on various texture and scene data sets demonstrate the efficiency of the proposed approach.

Warping-based image stitching methods often suffer from perspective variations among multiple images and lead to shape and perspective distortions in stitching results. Moreover, they also quickly lose their efficiency in low-textured images, due to the lack of reliable point correspondences. To solve these problems, this paper presents a locally warping-based image stitching by imposing line constraints. First, a two-stage alignment scheme with line constraints is introduced to achieve accurate alignment. More precisely, line features are adopted as alignment constraints to jointly estimate local homographies with point correspondences, which provides strong correspondences especially in low-textured cases. Then line constraints are also imposed to the content-preserving warping framework to further reduce alignment errors and preserve image structures. Second, in order to preserve shape and perspective information, a global similarity transform is introduced to mitigate projective distortions. Experimental results demonstrate the efficiency of our method, which yields more encouraging image stitching results in contrast with state-of-the-art methods.

In this paper, a unified superresolution framework, which simultaneously integrates image reconstruction and image registration, is proposed for multiangle-multispectral remote sensing images. With the strong correlation of different angle images, the low-rank structure of multiangle image is explored by lexicographically ordering the 3-D data cube into a 2-D matrix. Based on this property, the observed images is decomposed to a background image and a foreground image. Considering the high relationship between each band of the multispectral image, a multichannel total variation (MTV) constraint was applied on the target high resolution background image. Therefore, a robust superresolution model of multiangle-multispectral remote sensing image based on rank minimization (RSRRM) is proposed to register the low resolution multi-angle images and reconstruct high-resolution image. This method can achieve image registration and superresolution at the same time. Several experiments are conducted to illustrate the performance of the method.

Sparse representation theory has become a powerful tool since it can obtain the sparsest or the unique solution for the underdetermined problem with the development of linear algebra, optimization, scientific computing and more. As subpixel information extraction encountered in hyperspectral remote sensing, which contains many mixed pixels, are famous under-determined ill-posed problem. In addition, there is no unified model to conquer the problems with the subpixel analysis techniques, i.e., spectral unmixing and subpixel mapping. To cope with this under-determined problem, a unified sparse subpixel information extraction framework was proposed in this paper, which connects sparse unmixing and sparse subpixel mapping methods in a unified theoretical system as a serious of sparse regression problem. The experimental results with hyperspectral images indicate that the proposed sparse representation framework outperforms the previous subpixel analysis approaches, hence, provides an effective option for subpixel information extraction idea for hyperspectral remote sensing imagery.

Sparse unmixing has been successfully applied to hyperspectral remote sensing imagery, based on an available standard spectral library. However, as the number of hyperspectral remote sensors increases, more and more hyperspectral remote sensing images are requiring analysis without the use of a corresponding standard spectral library. To address this problem, sparse unmixing with a complete dictionary online self-learning technique is proposed in this paper. This paper focuses on complete dictionary, which can tackle the unmixing problem with exactly atoms needed in the dataset and online learning means to process the specific data, or the current single hyperspectral remote sensing imagery, at real time. The proposed method addresses the sparse unmixing problem by considering the physical meaning of atoms in the complete dictionary, as well as a non-negative constraint for the abundance. Compared with the classical dictionary learning approaches in sparse representation theory, the experiments with two simulated hyperspectral datasets and a real dataset confirmed the effectiveness of the proposed method.

Recently, non-negative matrix factorization (NMF) model has been widely used in hyperspectral unmixing (HU). In this paper, based on NMF, we explore the properties of abundance maps, and propose a new blind HU algorithm named total variation regularized reweighted sparse NMF (TV-RSNMF). Typically, only a subset of endmembers are assumed to generate the fixed pixel. As a result, the abundance maps are assumed to be sparse. So we introduce a weighted sparse regularization to explore the sparsity of abundance maps in the NMF model. In addition, the abundance maps related to fixed material are assumed to be piecewise smooth and we adopt a total variation (TV) regularizer to promote the piecewise smooth property. TV regularizer can be regarded as an abundance maps denoising procedure, which significantly improves the robustness of the proposed method to noise. Several experiments were conducted to illustrate the performance of the proposed algorithm.

In the past decades, tremendous investigations have been made to classify high-spatial-resolution remote sensing (HSR-RS) images at scene level. Among them, Bag-of-Visual-Words (BoVW) model has been widely used thanks to its robustness and efficiency. However, such a representation leaves out the spatial information of the image which is very important for distinguishing various scenes. In this paper, we aim to mine the spatial distribution of the visual words in the BoVW methods so as to incorporate the spatial information of the HSR-RS image and improve the classification accuracy. More precisely, we start from a BoVW representation of each scene image, and then compute local spatial features from this representation, i.e. the encoded image with BoVW model1. The marginal distributions of these local spatial features are finally used to describe HSR-RS scene images. In particular, the local spatial features we used in this paper include the local binary pattern (LBP) and re-learned BoVW dictionaries. The method has been evaluated on a large-scale HSR-RS image dataset, i.e. WHU20, that consists of 5000 HSR-RS images with 20 semantic classes for scene classification. The experimental results show that our method can improve the classification accuracy a lot compared with the standard BoVW method.

In this article, we proposed a novel semi-supervised sparse representation classification for high resolution remote sensing image. First, collaborative representation mechanism that exploits the help from whole training information rather than from only the potential associated class can enhance the class recognition ability. Second, by taking advantage of spatial occurrence and alignment of class label, the adoption of the relearning can gradually learn the flexible class-oriented spatial pattern from the label space with alleviated computational complexity to enhance the original spectral characteristics. Third, inspired by the spatial smoothing phenomenon when spatial feature stacked, a novel stable self-learning method can be designed to automatically select informative unlabeled sample to help the limited supervised set. Experiments on two hyperspectral and high-spatial resolution images validated the effectiveness and robustness of the proposed algorithm.

The cloud and cloud shadow are difficult to capture accurately in optical imagery because of insufficient spectral information. In this paper, an automatic multiple features combined (MFC) method is proposed for cloud and cloud shadow detection in GF-1 WFV imagery which includes three visible and one near-infrared bands. The local optimization strategy with guided filtering, and the proposed object-based filter combining geometry and texture features are used in the proposed method to refine cloud detection results and exclude non-cloud bright objects. The experimental results indicate that MFC performs well under different conditions.

China is currently suffering from a heavy PM2.5pollution. To estimate ground-level PM2.5from satellite-observed aerosol optical depth (AOD), many regional studies have been undertaken, but a few at national scale in China. Moreover, due to the wide spatial range and complex meteorological fields, the previous models' estimation accuracy of PM2.5still has space to improve. In this paper, using the newly available national PM2.5measurements, we develop a generalized regression neural network (GRNN) model to better describe the PM2.5-AOD relationship in China. Besides, a direct average of satellite-derived PM2.5can only reflect the level of PM2.5pollution on some certain days when AOD data is available. To address this issue, a pixel-based merging scheme is proposed. The results suggest that the cross validation R and RMSE are 0.811 and 20.11 μg / m3, respectively. The results also show that our study can provide useful information for global monitoring of PM2.5pollution in China.

Urban villages (UVs) are a special type of urban settlements in China. Their spatial evolution in recent years has a close relationship with urban planning and economic development. However, the remote sensing community pays little attention to UVs. This paper presents a new semantic scene model based on primitive indexes for detecting UVs using high resolution remotely sensed imagery. The model represents scenes as characteristics histograms of primitive objects. In the experiments, UVs in the main urban areas of Shenzhen over 2003–2012 were mapped. The proposed model outperformed conventional scene models quantitatively and visually, and showed good transferability across multitemporal images.

Convolutional neural network (CNN) has outstanding performance on nature image classification, such as facial recognition, ImageNet Large Scale Visual Recognition Challenge. However, due to scale variation of the same object in scene, it's difficult to directly utilize CNN for remote sensing image classification. In order to solve this problem, scene classification based on a random-scale stretched convolutional neural network (SRSCNN) for HSR remote sensing imagery is proposed in this paper. In the proposed method, the patches with random scale is cropped from image and stretched to the specified scale as input to train CNN, and in order to further improve the performance of CNN, the proposed method classifies an image multiple times to decide its label by voting. Experimental results using two datasets, i.e. the UC Merced dataset, Google Dataset of SIRI-WHU, show better performance than the traditional scene classification methods.

In this paper, an unsupervised change detection model based on hybrid conditional random field model (HCRF) is proposed for high spatial resolution (HSR) remote sensing imagery. Traditional random field based algorithms are mainly based on the analysis of the difference image which ignores the spatial-temporal change information of ground objects which is important in dealing with HSR imagery. Thus in HCRF, a new graph structure is designed to explore the correlation of corresponding ground objects from different times to get a better result. The unary potential is selected as the probabilistic result of change vector analysis (CVA), the pairwise potential is modeled to consider the contextual information of difference image and the similarity between objects from bi-temporal original images is considered using an object term. The proposed method is tested on two HSR data sets (IKONOS and QuickBird) and out performs some state-of-art algorithms.

Remote sensing image clustering is a challenging task. Recently, by combining the spectral and spatial information of remote sensing data, the clustering accuracy can be dramatically enhanced. However, it has always been difficult to determine the weight parameter for balancing the spectral and spatial terms of the clustering objective function. In this paper, spectral-spatial clustering with a local weight parameter determination method for remote sensing imagery is proposed, i.e. LSSC. In LSSC, considering the large scale of remote sensing images, the weight parameter is determined locally in a patch image instead of the whole image. The local weight parameter is then used in constructing the objective function of LSSC. Thus, the remote sensing image clustering problem is transformed into an optimization problem. Finally, in order to achieve a better optimization performance, a variant of differential evolution (i.e. jDE) is used as the optimizer due to its powerful optimization capability. Experimental results confirm that the proposed LSSC can acquire a higher clustering accuracy than other spectral-spatial clustering methods.

In the last few years, integrating spatial information into active learning framework has been gaining growing interest in the remote sensing community to optimize the collection of training sample set for supervised image classification. We address this problem from two directions. One of the directions focus on improving the classifier's performance to reduce the need for training samples. For this purpose, relearning model is introduced to combine the active learning framework to form mutually reinforcing process. In the meantime, another direction focus on the way to select most informative samples. For this purpose, new uncertainty criterion is proposed to favor the selection of samples not only with most spectral uncertainty, but also located in most uncertain spatial regions. Experiments on hyperspectral image show the effectiveness of proposed active learning framework.

This manuscript proposes a symmetric sparse representation (SSR) method to extract pure endmembers from Hyperspectral imagery (HSI). The SSR assumes that the desired endmembers and all the HSI pixels can be sparsely represented by each other and it formulates the endmember extraction problem into finding archetypes in the minimal convex hull of the HSI data. The optimization program of SSR is solved by a simple projected gradient algorithm and the endmembers are initialized with the vector quantization scheme. Preliminary results on the popular Urban HSI data infer that the SSR performs better than several state-of-the-art methods (VCA, NFINDER, AVMAX, SVMAX, XRAY, OSP and H2NMF).

In this paper, we introduced a deep learning model: Convolutional neural network(CNN) from the field of natural image classification and restoration, to solve general quality improving tasks for remote sensing images, including super-resolution, denoising and haze removal. To take advantage of the content similarity among aerial images and the learning ability of deep learning models, we proposed the idea of training CNN on datasets collected from aerial images with specific degenerating factors, then apply the model to matched tasks. Experiments showed that our network achieved superior performance in quantified results, and visually reconstructed a satisfying majority of missing details from low-quality observations.

Most of the existed scene classification methods classify the scenes ignoring the semantic ground objects for high spatial resolution imagery. Though lots of methods are proposed to recognize the objects, there are lack of methods modeling the spatial relationship between objects for scene classification. Besides, the frequency vector of the semantic objects in the scenes is inadequate to model the spatial relationship. Therefore, to acquire the semantic relationship among multiple objects or between the objects and the scene, this paper developed a multi-object force histogram (MOFH) to model the topology of multiple objects, and proposed a multi-object spatial relationship model (MOSRM) by combining the frequency vector of the ground objects and MOFH for the high spatial resolution scene classification. The experiments show that the proposed method can outperform the scene classification based on the frequency vector of the semantic objects.

Hyperspectral image super resolution (SR) reconstruction has been studied widely and many algorithms have been proposed. In this paper, a novel super resolution reconstruction method was designed by employing a joint spectral-spatial sub-pixel mapping model which aims to obtain the probabilities of sub-pixels to belong to different land cover classes by dividing mixed pixels into several sub-pixels. Given these sub-pixel probabilities, the resolution enhanced image can be further generated. The proposed approach has been evaluated using both synthetic and real hyperspectral images and compared with other well-known methods. The visual and quantitative comparisons confirm the effectiveness of the proposed method.

The leaked heat pipeline can be detected as temperature anomalies from the air-borne thermal image. Existing methods of thermal anomaly detection are prone to generate a large quantity of false alarms. Although supervised classification can reduce the false positive rate, it requires years of accumulated training data. In this paper, we use human visual system to improve the detection capabilities of thermal anomaly in district heating system. Leakage candidates are selected from the saliency map created by the thermal image, then buffer analysis with pipeline GIS layer is used to reject false detections. Experimental results show that the proposed method has better performance in detection rate when prior knowledge is scarce, and it is more adaptable to actual circumstances.

In this paper, endmember extraction algorithm is described as a combinatorial optimization problem. A novel quantum-behaved particle swarm optimization (QPSO) approach which employs quantum-behaved particle swarm optimization to find endmembers with good performance is proposed. As far as our knowledge, it is the first time that quantum-behaved particle swarm optimization is introduced into hyperspectral endmember extraction. In order to follow the law of particle movement, a high dimensional particles definition is proposed. The proposed algorithm was tested and evaluated by both synthetic and real hyperspectral data sets. Experimental results indicate that the proposed method get a better result compared to the algorithms of vertex component analysis (VCA), N-FINDR and discrete particle swarm optimization (D-PSO).

Band selection is an effective approach to mitigate the “Hughes phenomenon” of hyperspectral image (HSI) classification. In this paper, a novel squaring weighted low-rank subspace clustering band selection (SWLRSC) algorithm is proposed for hyperspectral imagery. The SWLRSC method can effectively capture the global structure information of the HSI band set by constructing a strongly connected adjacency matrix with accurate representation coefficients, and can adaptively determine an appropriate size for the selected band subset. The experimental results indicate that the proposed SWLRSC algorithm outperforms the state-of-the-art band selection algorithms.

Complementary information from multi-sensor can be integrated to effectively solve many problems in remote sensing application. Synthetic Aperture Radar (SAR) imaging can be a feasible alternative to traditional optical remote sensing techniques because it is independent of solar illumination and weather conditions. This paper proposes a novel fusion framework combining IHS transform with sparse representation theory to fuse multispectral and SAR images. In addition, the simultaneous orthogonal matching pursuit (SOMP) technique is introduced to guarantee the efficiency. Experiments on various datasets have verified the effectiveness of proposed method.

Robust background representation is a key issue for detecting anomaly targets in hyperspectral imagery. Meanwhile, the inherent nonlinearity of hyperspectral images may cover up the intrinsic data structure in the anomaly detection process. This paper for the first time aims to implement robust background representation, as well as to explore the intrinsic data structure of the hyperspectral imagery simultaneously. Robust background dataset and potential anomaly dataset are firstly assigned by a robust background regression (RBR) procedure. In the RBR, a minimum squared loss relationship is constructed between the nonlinear data density feature and the corresponding assigned labels, to formulate the foundation of the regression. Furthermore, a manifold regularization term and a robust background average margin maximization term are jointly appended in the RBR procedure. After this, a paired-dataset based k-nn score estimation is undertaken to implement the detection output. Experimental results prove that the proposed method outperforms some other state-of-the-art anomaly detectors.

In this paper, a novel feature extraction framework is proposed for hyperspectral image classification. Inspired by the role of discriminant function in classifier, which intends to learn a mapping from the input features to label information in class space, we develop a feature extraction framework to learn the new feature representation of original input features in class space, by establishing the relevance between feature extraction and discriminative classifier. The new learning features integrate the input features and the discrimination information of used classifier with available training samples, which reveal the cues of class in class space. Therefore, the new features are called as the features of class-in-class. Several experiments were conducted to illustrate the availability of the proposed features.

Deep networks like the convolutional neural network and its variants usually learn hierarchical features from labeled images, which is very expensive to obtain. How can we find an unsupervised way to effectively extract deep and abstract features from images without annotations? Even from large qualities of images with noise? In this paper, we propose a robust deep neural network, named as stacked convolutional denoising auto-encoders (SCDAE), which can map raw images to hierarchical representations in an unsupervised manner. Our network is elaborately designed to fit for the visual recognition tasks. It is established by stacking the denoising auto-encoders. Unlike the prior works, in the training phase, the auto-encoders are trained patch-wisely so that the latent features can be applied to powerful regularizers for better representation; in the inference phase, the denoising auto-encoders are stacked convolutionally, hence the generated feature maps in the higher layers can preserve the coherent structures within the features in the lower layers. To achieve better performance, we apply whitening to each layer to sphere the input features. Our network is evaluated on the challenging image datasets MNIST, CIFAR-10 and STL-10 and demonstrates superior performance to the state-of-the-art unsupervised networks.

Object tracking usually suffers from the geometrical deformations and occlusions of objects. This paper presents a new method for accurate object tracking by combining the multi-angle discriminative correlation filters and key-points under the framework of Discriminative Scale Space Tracker (DSST) tracker. Experimental results demonstrate that the proposed method can produce promising tracking results and outperform the-state-of-the-art methods using correlation filters.

Gait recognition is a rising biometric technology which aims to distinguish people purely through the analysis of the way they walk, while the problem is that the dimensionality of the gait data is too high, so it is necessary to carry on dimensionality reduction task. Up to date, in the area of computer vision and pattern recognition, various dimensionality reduction algorithms have been employed for gait data, including the conventional vector representation based methods principal components analysis (PCA) and, locality preserving projection (LPP), and the recently proposed multi-linear subspace learning based approaches such as multilinear principal component analysis (MPCA). In this paper, inspired by the advantages of the tensor representation and manifold learning, we propose a novel sparse tensor discriminative locality alignment for human gait feature representation and dimensionality reduction algorithm, and subsequently apply the refined feature for gait recognition by a lazy classifier of the KNN. The proposed method adopts sparse multi-way projection based on the high-order version of discriminative locality alignment, by which the class separability is enhanced and the potential model overfitting is simultaneously avoided. Extensive experiments on the University of South Florida (USF) HumanID Gait Database show that the proposed method achieves better recognition rate compared with some existing classical dimensionality reduction algorithms.

Dimensionality reduction (DR) is an important and helpful preprocessing step for hyperspectral image (HSI) classification. Recently, sparse graph embedding (SGE) has been widely used in the DR of HSIs. SGE explores the sparsity of the HSI data and can achieve good results. However, in most cases, locality is more important than sparsity when learning the features of the data. In this letter, we propose an extended SGE method: the weighted sparse graph based DR (WSGDR) method for HSIs. WSGDR explicitly encourages the sparse coding to be local and pays more attention to those training pixels that are more similar to the test pixel in representing the test pixel. Furthermore, WSGDR can offer data-adaptive neighborhoods, which results in the proposed method being more robust to noise. The proposed method was tested on two widely used HSI data sets, and the results suggest that WSGDR obtains sparser representation results. Furthermore, the experimental results also confirm the superiority of the proposed WSGDR method over the other state-of-the-art DR methods.

Due to the limitation of hyperspectral sensors and optical imaging systems, there are several irreconcilable conflicts between high spatial resolution and high spectral resolution of hyperspectral images (HSIs). Therefore, HSI super-resolution (SR) is regarded as an important preprocessing task for subsequent applications. In this letter, we use sparse representation to analyze the spectral and spatial feature of HSIs. Considering the sparse characteristic of spectral unmixing and high pattern repeatability of spatial-spectral blocks, we proposed a novel HSI SR framework utilizing spectral mixture analysis and spatial-spectral group sparsity. By simultaneously combining the sparsity and the nonlocal self-similarity of the images in the spatial and spectral domains, the method not only maintains the spectral consistency but also produces plenty of image details. Experiments on three hyperspectral data sets confirm that the proposed method is robust to noise and achieves better results than traditional methods.

The spatial information has been proved to be effective in improving the performance of spectral-based classification. However, it is difficult to describe different image scenes by using monofeature owing to complexity of the geospatial scenes. In this letter, a novel framework is developed to combine the multiple spectral and spatial features based on the Markov random field (MRF). Specifically, the pixels in an image are separated into reliable and unreliable ones according to the decision of multifeature classifications. The labels of the reliable pixels can be conveniently determined, but the unreliable pixels are then classified by fusing the multifeature classification results and reducing the classification uncertainties based on the MRF optimization. Experiments are conducted on three multispectral high-resolution images to verify the effectiveness of the proposed method. Several state-of-the-art multifeature classification methods are also achieved for the purpose of comparison. Moreover, three classifiers (i.e., multinomial logistic regression, support vector machines, and random forest) are used to test the performance of the proposed framework. It is shown that the proposed method can effectively integrate multiple features, yield promising results, and outperform other approaches compared.

In this letter, a multifeature probabilistic ensemble conditional random field (MFPECRF) model is proposed to perform the task of change detection for high spatial resolution (HSR) remote sensing imagery. MFPECRF not only considers the spectral feature of single pixels but also the interaction between neighborhood pixels and the structural property of the ground objects in HSR imagery to give a higher detection accuracy than the traditional random field methods, which only utilize spectral and label information. In the unary potential, the spectral and morphological features of the difference image are combined using a probabilistic ensemble strategy, and the pairwise potential considers the contextual information of the observed field. The parameters of MFPECRF are estimated using a piecewise strategy, and the final result is obtained by the use of the loopy belief propagation algorithm. The experimental results of two groups of HSR multispectral images confirm the potential of the proposed method in improving the detection accuracy for HSR imagery.

It is well known that polarimetric synthetic aperture radar (PolSAR) backscattering features are highly influenced by the variation of incidence angle (VIA), which usually hampers the classification of most grazing-angle-sensitive targets, such as land and ocean targets. To relieve this issue, various feature extraction approaches have been suggested to enhance the class discriminability while reducing the observed feature dimensionality. The Laplacian eigenmap-based dimension reduction (DR) has been proven to be an effective way to deal with VIA problems, provided that the manifold parameters [e.g., the heat kernel (HK)] have been optimally sought, which is often difficult in practice. In this letter, an adaptive Laplacian eigenmap-based DR method is presented to find a learned subspace where the local geometry with discriminative prior knowledge is preserved as much as possible while near optimal HK and scale factor parameters are automatically identified. The learned feature representation is then employed for the subsequent classification. The improved Laplacian eigenmap algorithm was validated by three uninhabited-aerial-vehicle-synthetic-aperture-radar L-band PolSAR images from the Gulf Deepwater Horizon oil spill, which were clearly impacted by the VIA phenomenon. The experimental results showed that the proposed algorithm works well in ocean target discrimination compared with the current common methods.

Scene classification has been studied to allow us to semantically interpret high spatial resolution (HSR) remote sensing imagery. The bag-of-visual-words (BOVW) model is an effective method for HSR image scene classification. However, the traditional BOVW model only captures the local patterns of images by utilizing local features. In this letter, a local-global feature bag-of-visual-words scene classifier (LGFBOVW) is proposed for HSR imagery. In LGFBOVW, the shape-based invariant texture index is designed as the global texture feature, the mean and standard deviation values are employed as the local spectral feature, and the dense scale-invariant feature transform (SIFT) feature is employed as the structural feature. The LGFBOVW can effectively combine the local and global features by an appropriate feature fusion strategy at histogram level. Experimental results on UC Merced and Google data sets of SIRI-WHU demonstrate that the proposed method outperforms the state-of-the-art scene classification methods for HSR imagery.

In the commonly employed regularization models of image restoration and super-resolution (SR), the norm determination is often challenging. This paper proposes a method to adaptively determine the optimal norms for both fidelity term and regularization term in the (SR) restoration model. Inspired by a generalized likelihood ratio test, a piecewise function is proposed to solve the norm of the fidelity term. This function can find the stable norm value in a certain number of iterations, regardless of whether the noise type is Gaussian, impulse, or mixed. For the regularization norm, the main advantage of the proposed method is that it is locally adaptive. Specifically, it assigns different norms for different pixel locations, according to the local activity measured by a structure tensor metric. The proposed method was tested using different types of images. The experimental results and error analyses verify the efficacy of the method.

Subpixel mapping, which is a promising technique based on the assumption of spatial dependence, enhances the spatial resolution of images by dividing a mixed pixel into several subpixels and assigning each subpixel to a single land-cover class. The traditional subpixel mapping methods usually utilize the fractional abundance images obtained by a spectral unmixing technique as input and consider the spatial correlation information among pixels and subpixels. However, most of these algorithms treat subpixels separately and locally while ignoring the rationality of global patterns. In this paper, a novel subpixel mapping model based on sparse representation theory, namely, adaptive sparse subpixel mapping with a total variation model (ASSM-TV), is proposed to explore the possible spatial distribution patterns of subpixels by considering these subpixels as an integral patch. In this way, the proposed method can obtain the optimal subpixel mapping result by determining the most appropriate subpixel spatial pattern. However, the number of possible spatial configurations of subpixels can increase sharply with large-scale factors, and therefore, in ASSM-TV, the subpixel mapping is considered as a sparse representation problem. A preconstructed discrete cosine transform dictionary, which consists of piecewise smooth subpixel patches and textured patches, is utilized to express the original subpixel mapping observation in a sparse representation pattern. The total variation prior model is designed as a spatial regularization constraint to characterize the relationship between a subpixel and its neighboring subpixels. In addition, a joint maximum a posteriori model is proposed to adaptively select the regularization parameters. Compared with the other traditional and state-of-the-art subpixel mapping approaches, the experimental results using a simulated image, three synthetic hyperspectral remote sensing images, and two real remote sensing images demonstrate that the proposed algorithm can obtain better results, in both visual and quantitative evaluations.

In recent years, the support vector machines (SVMs) have been very successful in remote sensing image classification, particularly when dealing with high-dimensional data and limited training samples. Nevertheless, the vector-based feature alignment of the SVM can lead to an information loss in representation of hyperspectral images, which intrinsically have a tensor-based data structure. In this paper, a new multiclass support tensor machine (STM) is specifically developed for hyperspectral image classification. Our newly proposed STM processes the hyperspectral image as a data cube and then identifies the information classes in tensor space. The multiclass STM is developed from a set of binary STM classifiers using the one-against-one parallel strategy. As a part of our tensor-based processing chain, a multilinear principal component analysis (MPCA) is used for preprocessing, in order to reduce the tensorial data redundancy and, at the same time, preserve the tensorial structure information in sparse and high-order subspaces. As a result, the contributions of this work are twofold: a new multiclass STM model for hyperspectral image classification is developed, and a tensorial image interpretation framework is constructed, which provides a system consisting of tensor-based feature representation, feature extraction, and classification. Experiments with four hyperspectral data sets, covering agricultural and urban areas, are conducted to validate the effectiveness of the proposed framework. Our experimental results show that the proposed STM and MPCA-STM can achieve better results than traditional SVM-based classifiers.

In this paper, we present a spatial spectral hyperspectral image (HSI) mixed-noise removal method named total variation (TV)-regularized low-rank matrix factorization (LRTV). In general, HSIs are not only assumed to lie in a low-rank subspace from the spectral perspective but also assumed to be piecewise smooth in the spatial dimension. The proposed method integrates the nuclear norm, TV regularization, and L1-norm together in a unified framework. The nuclear norm is used to exploit the spectral low-rank property, and the TV regularization is adopted to explore the spatial piecewise smooth structure of the HSI. At the same time, the sparse noise, which includes stripes, impulse noise, and dead pixels, is detected by the L1-norm regularization. To tradeoff the nuclear norm and TV regularization and to further remove the Gaussian noise of the HSI, we also restrict the rank of the clean image to be no larger than the number of endmembers. A number of experiments were conducted in both simulated and real data conditions to illustrate the performance of the proposed LRTV method for HSI restoration.

Hyperspectral image (HSI) denoising is a crucial preprocessing task that is used to improve the quality of images for object detection, classification, and other subsequent applications. It has been reported that noise can be effectively removed using the sparsity in the nonnoise part of the image. With the appreciable redundancy and correlation in HSIs, the denoising performance can be greatly improved if this redundancy and correlation is utilized efficiently in the denoising process. Inspired by this observation, a noise reduction method based on joint spectral-spatial distributed sparse representation is proposed for HSIs, which exploits the intraband structure and the interband correlation in the process of joint sparse representation and joint dictionary learning. In joint spectral-spatial sparse coding, the interband correlation is exploited to capture the similar structure and maintain the spectral continuity. The intraband structure is utilized to adaptively code the spatial structure differences of the different bands. Furthermore, using a joint dictionary learning algorithm, we obtain a dictionary that simultaneously describes the content of the different bands. Experiments on both synthetic and real hyperspectral data show that the proposed method can obtain better results than the other classic methods.

Remote sensing images are often contaminated by varying degrees of stripes, which severely affects the visual quality and subsequent application of the data. Unlike with conventional methods, we achieve the destriping by separating the stripe component based on a full analysis of the various stripe properties. Under an optimization framework, an ℓ0-norm-based regularization is used to characterize the global sparse distribution of the stripes. In addition, difference-based constraints are adopted to describe the local smoothness and discontinuity in the along-stripe and across-stripe directions, respectively. The alternating direction method of multipliers is applied to solve and accelerate the model optimization. Experiments with both simulated and real data demonstrate the effectiveness of the proposed model, in terms of both qualitative and quantitative perspectives.

In this paper, we focus on the despeckling of synthetic aperture radar (SAR) images by variational methods which introduce nonlocal regularization functionals. To achieve this goal, two models are investigated from different aspects. The first model is derived for the logarithmically transformed (homomorphic) domain of the SAR data, and the other is derived for the original (nonhomomorphic) domain. The statistical properties of the speckle and the log-transformed speckle are analyzed, and the similarity measurements between pixels in the homomorphic domain and nonhomomorphic domain are then derived for constructing the corresponding nonlocal regularization functionals. Meanwhile, in the proposed models, we develop a strategy to adaptively choose the regularization parameters based on both the local heterogeneity information and the noise level of the images, aiming at getting a better balance between the goodness of fit of the original data and the amount of smoothing. A quasi-Newton iteration method is employed to quickly minimize the proposed adaptive nonlocal functionals. Experiments conducted on both simulated images and real SAR images confirm the good performances of the proposed methods, both in reducing speckle and preserving image quality.

Hyperspectral remote sensing images, which are characterized by their high dimensionality, provide us with the capability to accurately identify objects on the ground. They can also be used to identify subclasses of objects. However, these subclasses are usually embedded in different subspaces due to the complex distribution of pixels in the feature space. In the literature, few hyperspectral image classification methods can take both the subclass and subspace into consideration at the same time. Motivated by the fact that natural DNA can distinguish biological subspecies (subclasses in hyperspectral images) using critical DNA fragments (subspaces in hyperspectral images), a semisupervised subspace-based DNA encoding and matching classifier for hyperspectral remote sensing imagery (SSDNA) is proposed in this paper. First, in the process of DNA encoding, the hyperspectral remote sensing image is transformed into a DNA cube, in which the first-order spectral curve of the hyperspectral remote sensing image is utilized in order to take the gradient information of the spectral curve into consideration. Second, in the process of DNA optimization, evolutionary algorithms are used to obtain the best DNA library of the typical objects, which includes the following: 1) A multicenter individual representation is designed in order to consider the existence of subclasses in the hyperspectral remote sensing image; 2) the unlabeled samples are utilized in the process of population initialization and fitness calculation to enhance the diversity of the population and the generalization of the classification performance; and 3) the different classes are embedded in different subspaces. A semisupervised technique is used to extract the subspaces, including the global subspace for all the classes and the local subspace for each class. Three hyperspectral data sets were tested and confirm that SSDNA performs better than the other supervised or semisupervised classifiers.

Remote sensing satellite sensors feature a tradeoff between the spatial, temporal, and spectral resolutions. In this paper, we propose an integrated framework for the spatio-temporal-spectral fusion of remote sensing images. There are two main advantages of the proposed integrated fusion framework: it can accomplish different kinds of fusion tasks, such as multiview spatial fusion, spatio-spectral fusion, and spatio-temporal fusion, based on a single unified model, and it can achieve the integrated fusion of multisource observations to obtain high spatio-temporal-spectral resolution images, without limitations on the number of remote sensing sensors. The proposed integrated fusion framework was comprehensively tested and verified in a variety of image fusion experiments. In the experiments, a number of different remote sensing satellites were utilized, including IKONOS, the Enhanced Thematic Mapper Plus (ETM+), the Moderate Resolution Imaging Spectroradiometer (MODIS), the Hyperspectral Digital Imagery Collection Experiment (HYDICE), and Système Pour l' Observation de la Terre-5 (SPOT-5). The experimental results confirm the effectiveness of the proposed method.

The new generation of Earth observation sensors with high spatial resolution can provide detailed information for change detection. The widely used methods for high-resolution image change detection rely on textural/structural features. However, these spatial features always produce high-dimensional data space since they are related to a series of parameters, e.g., window sizes and directions. Machine learning methods are also commonly employed, but their performances are subject to the quantity and quality of the training samples, and hence, much effort should be made to collect the high-quality samples. To address these problems, in this study, a novel multiindex automatic change detection method is proposed for the high-resolution imagery. The notable advantages of the proposed model include the following: 1) Complicated urban scenes are represented by a set of low dimensional but semantic information indexes, replacing the high-dimensional but low-level features (e.g., textural and structural features), and 2) the change detection model is carried out automatically without using training samples since the information indexes can directly indicate the primitive urban classes. The multiindex representation refers to the enhanced vegetation index, the water index, and the recently developed morphological building index. Experiments were conducted on the multitemporal WorldView-2 images over Shenzhen City (south of China) and Kuala Lumpur (the capital of Malaysia), where promising results were achieved by the proposed method. Moreover, the traditional methods based on the state-of-the-art textural/morphological features were also implemented for the purpose of comparison, which further validates the advantages of our proposed model.

The segmentation of synthetic aperture radar (SAR) images is a long-standing yet challenging task, not only because of the presence of speckle but also due to the variations of surface backscattering properties in the images. Tremendous investigations have been made to suppress the speckle effects for the segmentation of SAR images, whereas few works are devoted to dealing with the variations of backscattering intensities in the images. To overcome the two difficulties, this paper presents a novel SAR image segmentation method by exploiting a multiscale active contour model based on the nonlocal processing principle. More precisely, we first formulize the SAR segmentation problem with an active contour model by integrating the nonlocal interactions between pairs of patches inside and outside the segmented regions. Second, a multiscale strategy is proposed to speed up the nonlocal active contour segmentation procedure and to avoid falling into a local minimum for achieving more accurate segmentation results. Experimental results on simulated and real SAR images demonstrate the efficiency and feasibility of the proposed method: It can not only achieve precise segmentations for images with heavy speckle and nonlocal intensity variations but also be used for SAR images from different types of sensors.

Due to the recent advances in satellite sensors, a large amount of high-resolution remote sensing images is now being obtained each day. How to automatically recognize and analyze scenes from these satellite images effectively and efficiently has become a big challenge in the remote sensing field. Recently, a lot of work in scene classification has been proposed, focusing on deep neural networks, which learn hierarchical internal feature representations from image data sets and produce state-of-the-art performance. However, most methods, including the traditional shallow methods and deep neural networks, only concentrate on training a single model. Meanwhile, neural network ensembles have proved to be a powerful and practical tool for a number of different predictive tasks. Can we find a way to combine different deep neural networks effectively and efficiently for scene classification? In this paper, we propose a gradient boosting random convolutional network (GBRCN) framework for scene classification, which can effectively combine many deep neural networks. As far as we know, this is the first time that a deep ensemble framework has been proposed for scene classification. Moreover, in the experiments, the proposed method was applied to two challenging high-resolution data sets: 1) the UC Merced data set containing 21 different aerial scene categories with a submeter resolution and 2) a Sydney data set containing eight land-use categories with a 1.0-m spatial resolution. The proposed GBRCN framework outperformed the state-of-the-art methods with the UC Merced data set, including the traditional single convolutional network approach. For the Sydney data set, the proposed method again obtained the best accuracy, demonstrating that the proposed framework can provide more accurate classification results than the state-of-the-art methods.

Anomaly detection is playing an increasingly important role in hyperspectral image (HSI) processing. The traditional anomaly detection methods mainly extract knowledge from the background and use the difference between the anomalies and the background to distinguish them. Anomaly contamination and the inverse covariance matrix problem are the main difficulties with these methods. The low-rank and sparse matrix decomposition (LRaSMD) technique may have the potential to solve the aforementioned hyperspectral anomaly detection problem since it can extract knowledge from both the background and the anomalies. This paper proposes an LRaSMD-based Mahalanobis distance method for hyperspectral anomaly detection (LSMAD). This approach has the following capabilities: 1) takes full advantage of the LRaSMD technique to set the background apart from the anomalies; 2) explores the low-rank prior knowledge of the background to compute the background statistics; and 3) applies the Mahalanobis distance differences to detect the probable anomalies. Extensive experiments were carried out on four HSIs, and it was found that LSMAD shows a better detection performance than the current state-of-the-art hyperspectral anomaly detection methods.

Aircraft detection from very high resolution (VHR) remote sensing images has been drawing increasing interest in recent years due to the successful civil and military applications. However, several challenges still exist: 1) extracting the high-level features and the hierarchical feature representations of the objects is difficult; 2) manual annotation of the objects in large image sets is generally expensive and sometimes unreliable; and 3) locating objects within such a large image is difficult and time consuming. In this paper, we propose a weakly supervised learning framework based on coupled convolutional neural networks (CNNs) for aircraft detection, which can simultaneously solve these problems. We first develop a CNN-based method to extract the high-level features and the hierarchical feature representations of the objects. We then employ an iterative weakly supervised learning framework to automatically mine and augment the training data set from the original image. We propose a coupled CNN method, which combines a candidate region proposal network and a localization network to extract the proposals and simultaneously locate the aircraft, which is more efficient and accurate, even in large-scale VHR images. In the experiments, the proposed method was applied to three challenging high-resolution data sets: the Sydney International Airport data set, the Tokyo Haneda Airport data set, and the Berlin Tegel Airport data set. The extensive experimental results confirm that the proposed method can achieve a higher detection accuracy than the other methods.

Hyperspectral imaging offers new opportunities for pattern recognition tasks in the remote sensing community through its improved discrimination in the spectral domain. However, such advanced image processing also brings new challenges due to the high data dimensionality in both the spatial and spectral domains. To relieve this issue, in this paper, we present a novel multidomain subspace (MDS) feature representation and classification method for hyperspectral images. The proposed method is based on a patch alignment framework. In order to optimally combine the feature representations from the various domains and simultaneously enhance the subspace discriminability, we incorporate the supervised label information into each domain and further generalize the framework to a multidomain version. Furthermore, we develop an iterative approach to alternately optimize the MDS objective function by considering it as two subconvex optimizations. The classification performance on three standard hyperspectral remote sensing images confirms the superiority of the proposed MDS algorithm over the state-of-the-art subspace learning methods.

Clustering for hyperspectral images (HSIs) is a very challenging task due to its inherent complexity. In this paper, we propose a novel spectral-spatial sparse subspace clustering S4C algorithm for hyperspectral remote sensing images. First, by treating each kind of land-cover class as a subspace, we introduce the sparse subspace clustering (SSC) algorithm to HSIs. Then, considering the spectral and spatial properties of HSIs, the high spectral correlation and rich spatial information of the HSIs are taken into consideration in the SSC model to obtain a more accurate coefficient matrix, which is used to build the adjacent matrix. Finally, spectral clustering is applied to the adjacent matrix to obtain the final clustering result. Several experiments were conducted to illustrate the performance of the proposed S4C algorithm.

Background feature extraction is an important step in hyperspectral anomaly detection. However, the lack of prior information about anomaly targets and the complex spectral mixture result in a challenge for robust background feature extraction. Can we solve the anomaly detection problem other than with background feature extraction? Relative to anomalies, the background spectral signal is usually stable and slowly varying. In view of this point, slowly varying background analysis is introduced into anomaly detection in this paper. The desired background signals are obtained through a generalized eigenvalue decomposition problem based on the original data and the differential image. The extracted signals are then combined with a Mahalanobis distance metric to construct the detection estimation. Different data processing procedures and signal extraction patterns are respectively formulated to construct different versions of the slowly varying background-signal-based detector. The performances of the proposed methods were validated on both synthetic and real hyperspectral data. The experimental results reveal that the proposed methods outperform the state-of-the-art anomaly detectors, with superior receiver operating characteristic (ROC) curves, area-under-ROC values, and background-target separation. The sensitivity of the relevant parameters was also analyzed in an experimental analysis.

Due to the complex arrangements of the ground objects in high spatial resolution (HSR) imagery scenes, HSR imagery scene classification is a challenging task, which is aimed at bridging the semantic gap between the low-level features and the high-level semantic concepts. A combination of multiple complementary features for HSR imagery scene classification is considered a potential way to improve the performance. However, the different types of features have different characteristics, and how to fuse the different types of features is a classic problem. In this paper, a Dirichlet-derived multiple topic model (DMTM) is proposed to fuse heterogeneous features at a topic level for HSR imagery scene classification. An efficient algorithm based on a variational expectation-maximization framework is developed to infer the DMTM and estimate the parameters of the DMTM. The proposed DMTM scene classification method is able to incorporate different types of features with different characteristics, no matter whether these features are local or global, discrete or continuous. Meanwhile, the proposed DMTM can also reduce the dimension of the features representing the HSR images. In our experiments, three types of heterogeneous features, i.e., the local spectral feature, the local structural feature, and the global textural feature, were employed. The experimental results with three different HSR imagery data sets show that the three types of features are complementary. In addition, the proposed DMTM is able to reduce the dimension of the features representing the HSR images, to fuse the different types of features efficiently, and to improve the performance of the scene classification over that of other scene classification algorithms based on spatial pyramid matching, probabilistic latent semantic analysis, and latent Dirichlet allocation.

In this paper, a framework for multiscale and multifeature normalized cut (MMNCut) segmentation is proposed for high spatial resolution (HSR) remote sensing images. Normalized cuts (NCuts), as a widely used segmentation method for natural images, can obtain a globally optimized segmentation result corresponding to the optimized partitions of a graph. However, it is difficult to apply the traditional NCuts directly to HSR images because of the huge computational complexity and the diversity of the characteristics of the land covers. In order to solve these problems, the proposed MMNCuts builds a multiscale graph based on superpixels, which can provide powerful grouping cues to guide the segmentation. Generated by different algorithms with varying parameters, superpixels can capture diverse and multiscale visual patterns of HSR images. In addition, the newly constructed graph integrates the multiscale information by considering various connection relationships. Meanwhile, the successful integration of the multifeature cues, including the spectral information, texture information, and structure information, from a large number of superpixels, helps to enhance the expression ability of the graph. Computationally, this leads to a much more efficient algorithm than the traditional NCuts, and in effect, the proposed method achieves a significantly better performance than the traditional approaches. The experimental results with three HSR image data sets demonstrate that the proposed MMNCut algorithm shows a competitive performance in both qualitative and quantitative evaluations when compared with the other state-of-the-art segmentation algorithms for HSR images.

Hyperspectral images provide great potential for target detection, however, new challenges are also introduced for hyperspectral target detection, resulting that hyperspectral target detection should be treated as a new problem and modeled differently. Many classical detectors are proposed based on the linear mixing model and the sparsity model. However, the former type of model cannot deal well with spectral variability in limited endmembers, and the latter type of model usually treats the target detection as a simple classification problem and pays less attention to the low target probability. In this case, can we find an efficient way to utilize both the high-dimension features behind hyperspectral images and the limited target information to extract small targets? This paper proposes a novel sparsity-based detector named the hybrid sparsity and statistics detector (HSSD) for target detection in hyperspectral imagery, which can effectively deal with the above two problems. The proposed algorithm designs a hypothesis-specific dictionary based on the prior hypotheses for the test pixel, which can avoid the imbalanced number of training samples for a class-specific dictionary. Then, a purification process is employed for the background training samples in order to construct an effective competition between the two hypotheses. Next, a sparse representation-based binary hypothesis model merged with additive Gaussian noise is proposed to represent the image. Finally, a generalized likelihood ratio test is performed to obtain a more robust detection decision than the reconstruction residual-based detection methods. Extensive experimental results with three hyperspectral data sets confirm that the proposed HSSD algorithm clearly outperforms the state-of-the-art target detectors.

With the increase in the availability of high-resolution remote sensing imagery, classification is becoming an increasingly useful technique for providing a large area of detailed land-cover information by the use of these high-resolution images. High-resolution images have the characteristics of abundant geometric and detail information, which are beneficial to detailed classification. In order to make full use of these characteristics, a classification algorithm based on conditional random fields (CRFs) is presented in this paper. The proposed algorithm integrates spectral, spatial contextual, and spatial location cues by modeling the probabilistic potentials. The spectral cues modeled by the unary potentials can provide basic information for discriminating the various land-cover classes. The pairwise potentials consider the spatial contextual information by establishing the neighboring interactions between pixels to favor spatial smoothing. The spatial location cues are explicitly encoded in the higher order potentials. The higher order potentials consider the nonlocal range of the spatial location interactions between the target pixel and its nearest training samples. This can provide useful information for the classes that are easily confused with other land-cover types in the spectral appearance. The proposed algorithm integrates spectral, spatial contextual, and spatial location cues within a CRF framework to provide complementary information from varying perspectives, so that it can address the common problem of spectral variability in remote sensing images, which is directly reflected in the accuracy of each class and the average accuracy. The experimental results with three high-resolution images show the validity of the algorithm, compared with the other state-of-the-art classification algorithms.

Detecting certain targets from hyperspectral images (HSIs) is of great interest for both civilian and military applications, with the aim being to detect and identify target pixels based on specific spectral signatures. However, the classical algorithms are generally dependent on the specific statistical hypothesis test, and the algorithms may only perform well with certain assumptions. Therefore, in this paper, a novel metric-learning-based target detection framework, named local decision maximum margin metric learning (LDM3L), is proposed for HSI target detection. The proposed method can better separate the target samples from background ones, without the need for certain assumptions. The experimental results demonstrate that the proposed method outperforms both the state-of-the-art target detection algorithms and the other classical metric learning methods.

Scene classification for high-resolution remotely sensed imagery have been widely investigated in recent years. However, there is few public, widely accepted and large scale dataset for benchmarking different methods. This paper presents a new and large dataset consisting of 5000 high-resolution remote sensing images which is manually labeled in 20 semantic classes for scene classification. Each class includes more than 200 image samples with different appearances. Some classic classification algorithms are compared on this dataset. To our knowledge, this work is the first time to give a public benchmark dataset at this size on the problem of scene classification in high-resolution remote sensing imagery, and give comparative results and analysis of various classic classification algorithms.

Feature extraction is at the core of satellite scene classification task. In this paper, we propose a fast binary coding (FBC) method to effectively generate the global discriminative feature representation of image scenes. Equipped with unsupervised feature learning technique, we first learn a set of optimal “filters” from large quantities of randomly sampled image patches, and then we obtain feature maps by convolving image scene with the learned filter bank. After binarizing the feature maps, a simple skillful conversion of binary-valued feature map to integer-valued feature map is performed. The final statistical histograms, which are considered as the global feature representations of scenes, are computed on the integer-valued feature map similar to the conventional BOW model. Experiments on two datasets demonstrate that the proposed FBC achieve satisfying classification performance as well as has much faster computational speed compared with traditional scene classification methods.

Scene classification is a key problem in the interpretation of high-resolution remote sensing imagery. The state-of-the-art methods, e.g. bag-of-visual-words model and its various extensions as well as the topic models, share similar procedures: patch sampling, feature description/learning and classification. Patch sampling is the first and the key procedure which has a great influence on the results. In this paper, we focus on the effects of different sampling strategies used in the literature sa as to find a suitable sampling strategy for the scene classification of high-resolution remote sensing images. We divide the existing sampling methods into two types: random sampling and saliency-based sampling, and embed them in the bag-of-visual-words framework for comparison owing to its simplicity, robustness and efficiency. Moreover, we compare it using another framework - Fisher kernel, to validate our conclusions. The experimental results on two commonly used datasets using two different frameworks both show that random sampling can give better or comparable results than other sampling methods.

Hyperspectral Unmixing aims at getting the endmember signature and their corresponding abundance maps from highly mixed Hyperspectral image. Nonnegative Matrix Factorization (NMF) is a widely used method for spectral unmixing because it can obtain better performance while there is no pure pixels in the hyperspectral image. However, many methods based on nonnegative matrix factorization seldom consider the spatial information both on local and nonlocal. To combine the spatial and spectral information together to improve the unmixing accuracy, an adaptive endmember extraction based sparse nonnegative matrix factorization with spatial local information (ASNMF) is proposed in this paper. A superpixel segmentation is to obtain many meaningful regions which are spectral similar and spatial adjacent. Endmember is adaptively extracted on each superpixel to generate endmember set. Initialing the endmember set, ASNMF could adaptively obtain the final endmembers with the sparse nonnegative matrix factorization. Both the experiments on synthetic and real scene images show the effectiveness of the proposed method for hyperspectral unmixing.

With regard to the specific role of each pixel within a spatial parcel of a hyperspectral image (HSI), we propose a novel superpixel-oriented sparse representation classification method with a multi-task learning approach. The proposed algorithm exploits the class-level sparsity prior for multiple-feature fusion, and also the correlation and distinctiveness of pixels in a spatial local region. Compared with the state-of-the-art hyperspectral classifiers, the superiority of the spatial prior utilization, the multiple-feature fusion, and the computational efficiency are maintained at the same time in the proposed method. The proposed classification framework was tested on two HSIs. The experimental results suggest that the proposed algorithm performs better than the other representation-based classification algorithms and some popular hyperspectral multiple-feature classifiers.

Hyperspectral remote sensing image classification is one of the most challenging tasks. In our previous work, motivated by the similarity between the structures of DNA and hyperspectral remote sensing images, a DNA matching mechanism was used to transform the hyperspectral remote sensing image into a DNA cube for classification. However, the above DNA encoding strategy lacks the process of encoding accurate spectral and spatial feature into the DNA cube, resulting in unsatisfying classification performance. In this paper, a spectral-spatial DNA encoding strategy for encoding accurate spectral and spatial feature of hyperspectral remote sensing image is proposed. In the spectral dimension, the first-order spectral curve is encoded into the DNA cube, while in the spatial dimension, the principal components or their corresponding texture feature (GLCM) are encoded into the DNA cube. Finally, different with the previous DNA encoding classifier using genetic algorithm (GA), the paper combines the discriminative classifier (i.e. SVM) with spectral-spatial DNA encoding to improve classification performance for hyperspectral remote sensing imagery. The experimental results confirmed the effectiveness of the newly devised DNA encoding strategy and the discriminative classifier in classifying the DNA cube.

In this paper, a unified framework for the spatio-temporal-spectral fusion of remote sensing images is proposed. The relationships between the observed images and the desired image are first established based on general image observation models. Maximum a posteriori (MAP) theory is then employed to formulate the unified fusion framework. The proposed method is able to fuse images from an arbitrary number of optical sensors with different spatial, temporal, and spectral resolutions. The experimental results verify the effectiveness of the proposed method.

This paper addresses the problem of finding edges of buildings in high-resolution remotely sensed images, which is of great help for subsequent analysis of built-up areas in urban remote sensing. More precisely, we propose a novel algorithm to extract meaningful edges associated to buildings with their saliency, by integrating an edge detection procedure with a junction process. This is inspired by the observation that meaningful junctions mainly emerge around buildings rather than on non-building objects in high-resolution remote sensing images. Thus, given a high-resolution remotely sensed image, we first use an edge detection algorithm, e.g. canny edge extractor, to compute all possible candidate edges of buildings, and then refine those candidates by meaningful junctions around buildings with their significance. The meaningful junctions are provided by an a-contrario junction detector, whose significance is related to the structural saliency in the images. For the evaluation of the proposed method, we test it on a small set of remote sensing images of half-meter resolution from WorldView-2, IKONOS and QuicBird. It demonstrates that our approach can find and locate edges of buildings with high precision and efficiency.

As a significant digital representation of terrain surface, varieties of DEM products have been available to the public. The most widely used global DEM products are SRTM and ASTER GDEM. Given the comparable horizontal resolution and vertical error, accuracy validation and comparison have been of interest since the release, however, usually on a wide range. In this paper, we presented the results of accuracy assessment for ASTER GDEM v2 and SRTM v4.1 in Yulong Mountain, Yunnan province, China. Topographic map was chosen as the benchmark. The results and discussions were centered on the relationship between error distribution in elevation and mountainous hypsography based on data causes. The results revealed their levels of reliability for applied glaciology and hydrology in the typical snow mountain area.

Sparse representation has achieved great success in the hyperspectral image processing field. However, with regard to target detection, the state-of-the-art sparsity-based algorithms are ad hoc and no different to a classifier. In this paper, a novel target detection algorithm is proposed, combining an elaborately designed sparsity model and the binary hypothesis statistics. With the strong similarity of the material spectra from the same class, sparse representation theory is explored by constructing hypothesis-designed dictionaries. Based on the local smooth property, locally optimized selection methods are employed for the background samples. For hyperspectral images, the pixels are usually assumed to obey a Gaussian normal distribution. Therefore, in this paper, a statistics-inspired sparsity model is established. The generalized likelihood ratio test is utilized to solve the model and build a statistics-inspired sparsity target detector (SISTOR). A number of experiments were conducted to illustrate the performance of the proposed algorithm.

In this paper, we propose a novel spectral-spatial conditional random field classification algorithm with location cues (CRFSS) for high spatial resolution remote sensing imagery. In the CRFSS algorithm, the spectral and spatial location cues are integrated to provide the complementary information from spectral and spatial location perspectives. The spectral cues of different land-cover types are mainly provided by support vector machine (SVM), because of its excellent spectral classification performance. However, it is difficult to deal with the common spectral variability problem in remote sensing images. To alleviate this dilemma, considering the spectral similarity of the same land-cover in a local region, a point-to-point (P2P) classifier is designed to emphasize the spatial location cues. The P2P classifier considers the nonlocal range of the spatial location interactions between the target pixel and its nearest training samples for all the classes. In addition, the pairwise potential of CRFSS also considers the spatial contextual information to favor spatial smoothing. The experimental results showed that the algorithm has a competitive classification performance, in both the quantitative and qualitative evaluation.

Temporal tie points have been manually selected for sea ice concentration retrieval based on the linear combination relationship between the open water and the complete ice coverage pixel. In this paper, the multichannel information has been exploited using the constrained least-squares linear unmixing algorithm from AMSR-E multi-channels brightness temperature. Snow depth can be obtained from the innovative algorithm without considering the weather effect.

The digital elevation model (DEM) is a significant digital representation of a terrain surface. Although a variety of DEM products are available, they often suffer from problems varying in spatial coverage, data resolution, and accuracy. However, the multi-source DEMs often contain supplementary information, which makes it possible to produce a higher-quality DEM through blending the multi-scale data. Inspired by super-resolution (SR) methods, we propose a regularized framework for the production of high-resolution (HR) DEM data with extended coverage. To deal with the registration error and the horizontal displacement among multi-scale measurements, robust data fidelity with weighted norm is employed to measure the conformance of the reconstructed HR data to the observed data. Furthermore, a slope-based Markov random field (MRF) regularization is used as the spatial regularization. The proposed method can simultaneously handle complex terrain features, noises, and data voids. Using the proposed method, we can reconstruct a seamless DEM data with the highest resolution among the input data, and an extensive spatial coverage. The experiments confirmed the effectiveness of the proposed method under different cases.

As a result of the spatial consideration of the imagery, spatial sparse unmixing (SU) can improve the unmixing accuracy for hyperspectral imagery, based on the application of a spectral library and sparse representation. To better utilize the spatial information, spatial SU methods such as SU via variable splitting augmented Lagrangian and total variation (SUnSAL-TV) and nonlocal SU (NLSU) have been proposed. However, the spatial information considered in these algorithms comes from the estimated abundance maps, which will change along with the iterations. As the spatial correlations of the imagery are fixed and certain, the spatial relationships obtained from the variable abundances are not reliable during the process of optimization. To obtain more precise and fixed spatial relationships, an improved weight calculation NLSU (I-NLSU) algorithm is proposed in this letter by changing the spatial information acquisition source from the variable estimated abundances to the original hyperspectral imagery. A noise-adjusted principal component analysis strategy is also applied for the feature extraction in the proposed algorithm, and the obtained principal components are the foundation of the spatial relationships. The experimental results of both simulated and real hyperspectral data sets indicate that the proposed I-NLSU algorithm outperforms the previous spatial SU methods.

The Moderate Resolution Imaging Spectroradiometer (MODIS) land surface temperature (LST) product can offer accurate LST with high temporal and spatial resolution, but the quality is often degraded by cloud. To improve the usability of the MODIS LST, this letter proposes a reconstruction method based on multitemporal data. First, a multitemporal classification is employed to distinguish the different land surface types. The invalid LST values can then be predicted using a robust regression with the multitemporal information from the other LSTs. Finally, postprocessing is proposed to eliminate outliers. Simulated and actual experiments show that the method can accurately reconstruct the missing values.

In this letter, a change detection algorithm based on pulse-coupled neural networks (PCNN) and the normalized moment of inertia (NMI) feature is proposed for high spatial resolution (HSR) remote sensing imagery. To better analyze a large remote sensing image, the whole image is divided into blocks by the use of a deblocking mechanism. The PCNN model is utilized to obtain the initial binary image, and the NMI feature is calculated based on the binary image to detect the hot spot changed areas. Finally, the changed areas are processed by expectation–maximization to obtain the final change map. The experimental results using QuickBird and IKONOS images demonstrate that the proposed algorithm has the ability to provide better change detection results for HSR images than the traditional PCNN change detection algorithms.

Different Earth-observing (EO) sensors have various capabilities for diverse observing tasks. Sensor planning services make the choice of web-ready sensors for specific observing tasks with regard to observing requests and sensor capabilities. Sensor capabilities rely on various parameters; thus, choosing EO sensors for specific observing tasks relying directly on these parameters is a multicriteria decision process. A sensor's capability can be drawn from these parameters with the help of an algorithm. Furthermore, if divided into different clusters based on capabilities, applicable sensors can be more easily chosen for a category of observing tasks. In this paper, a spaceborne EO optical sensor static capability index (SSCI) mechanism is drawn from an evaluation-and-clustering algorithm, which is composed of a self-organizing neural map in combination with weighted principal component analysis. The scheme of SSCI relies on no expert analysis system and thus is more flexible and efficient. EO scenarios of disaster reactions are among the application of this algorithm. In particular, scenarios of flooding disaster forecasting, relief aiding, and postdisaster loss assessment within the framework of International Charter on Space and Major Disasters have been utilized for experiments. They have shown that the SSCI assessing algorithm is feasible and stable, and the EO optical sensor clustering algorithm based on SSCI can offer reasonable clustering accuracies of EO optical sensors. In our experiments, the EO optical sensor SSCI computation and clustering algorithm had a time consumption within 2 s and 2 min, respectively, and memory consumption within 200 MB on a normal personal computer.

Due to the rapid urbanization of China, many villages in the urban fringe are enveloped by ever-expanding cities and become so-called urban villages (UVs) with substandard living conditions. Despite physical similarities to informal settlements in other countries (e.g., slums in India), UVs have access to basic public services, and more importantly, villagers own the land legitimately. The resulting socio-economic impact on urban development attracts increasing interest. However, the identification of UVs in previous studies relies on fieldwork, leading to late and incomplete analyses. In this paper, we present three scene-based methods for detecting UVs using high-resolution remotely sensed imagery based on a novel multi-index scene model and two popular scene models, i.e., bag-of-visual-words and supervised latent Dirichlet allocation. In the experiments, our index-based approach produced Kappa values around 0.82 and outperformed conventional models both quantitatively and visually. Moreover, we performed multitemporal classification to evaluate the transferability of training samples across multitemporal images with respect to three methods, and the index-based approach yielded best results again. Finally, using the detection results, we conducted a systematic spatiotemporal analysis of UVs in Shenzhen and Wuhan, two mega cities of China. At the city level, we observe the decline of UVs in urban areas over the recent years. At the block level, we characterize UVs quantitatively from physical and geometrical perspectives and investigate the relationships between UVs and other geographic features. In both levels, the comparison between UVs in Shenzhen and Wuhan is made, and the variations within and across cities are revealed.

Abstract—Hyperspectral image classification has been an active topic of research in recent years. In the past, many different types of features have been extracted (using both linear and nonlinear strategies) for classification problems. On the one hand, some approaches have exploited the original spectral information or other features linearly derived from such information in order to have classes which are linearly separable. On the other hand, other techniques have exploited features obtained through nonlinear transformations intended to reduce data dimensionality, to better model the inherent nonlinearity of the original data (e.g., kernels) or to adequately exploit the spatial information contained in the scene (e.g., using morphological analysis). Special attention has been given to techniques able to exploit a single kind of features, such as composite kernel learning or multiple kernel learning, developed in order to deal with multiple kernels. However, few

In this paper, we propose a superpixel-level sparse representation classification framework with multitask learning for hyperspectral imagery. The proposed algorithm exploits the class-level sparsity prior for multiple-feature fusion, and the correlation and distinctiveness of pixels in a spatial local region. Compared with some of the state-of-the-art hyperspectral classifiers, the superiority of the multiple-feature combination, the spatial prior utilization, and the computational complexity are maintained at the same time in the proposed method. The proposed classification algorithm was tested on three hyperspectral images. The experimental results suggest that the proposed algorithm performs better than the other sparse (collaborative) representation-based algorithms and some popular hyperspectral multiple-feature classifiers.